Blackroot/Ammeg-26B-A4

Ammeg-26B-A4: Experimental Finetune for Creative Story Generation

Blackroot/Ammeg-26B-A4 is an experimental 26 billion parameter model, fine-tuned from gemma-26b-4A. It maintains the original Gemma instruct prompting methods and features a 32768 token context length. This model was developed using a unique two-part process:

Key Characteristics & Tuning Method

• Novel Obliteration: Employs a quality-preserving obliteration method, loosely based on 'heretic', to modify the base model.
• Evolutionary Strategy Retraining: The ablated model is then retrained using an evolutionary strategy, drawing inspiration from recent research (e.g., arXiv:2511.16652).
• Data & Objective: Training data primarily consists of books, chunked and captioned by the ablated model to generate prompts like "Generate a story with a protagonist named Alice...". The finetuning uses a non-differentiable objective that exploits the Zipf structure of language as a proxy for long-term dependencies in stories.
• Precision & LORA: Trained in full BF16 precision, utilizing Rank 1 LORA on a per-sample basis into a full-precision buffer with stochastic rounding, which eventually leads to noise cancellation and full-rank tuning.

Important Considerations

• Safeguards Removed: The tuning process has likely removed most of the model's original safeguards, including its tendency to soft-refuse or redirect. Users should be aware that typical expected safeguards may not be present.
• Hardware & Time: The training was conducted on powerful CPUs over approximately 280 hours, highlighting the time-intensive nature of this experimental method.

Good For

• Creative Story Generation: Excels at generating stories based on provided prompts, leveraging its focus on long-term linguistic dependencies.
• Experimental Use Cases: Ideal for developers interested in exploring models with novel training methodologies and unique characteristics.