Tlacuilo-12B: A Creative Writing Model
Tlacuilo-12B is a 12 billion parameter language model developed by allura-org, building upon the Muse-12B base model. This iteration focuses on enhancing creative writing capabilities, particularly in areas like roleplay (RP) and adventure narratives, addressing common limitations found in previous creative writing models.
Key Capabilities
- Enhanced Creative Prose: The model is designed to generate more varied and engaging prose, suitable for diverse creative applications.
- Improved Roleplay and Adventure: Specifically fine-tuned to perform well in interactive storytelling and roleplaying scenarios.
- ChatML Compatibility: Utilizes the ChatML chat template, consistent with its Muse-12B foundation.
- Flexible Temperature Settings: Recommended usage includes Temperature 1 / min-p 0.05 for balanced output, with options up to Temperature 1.3 / min-p 0.02 for more adventurous generation.
Training Methodology
The model underwent a three-stage QLoRA training process:
- Stage 1 (Prose Style): Fine-tuned on a diverse collection of books (28M tokens/epoch) using r32/a32 QLoRA at 32k context for 2 epochs, targeting QKV tensors.
- Stage 2 (Roleplay Data): Further trained on 4M tokens of RP data using r32/a32 QLoRA at 16k context for 1 epoch, applied to
o_proj and down_proj. - Stage 3 (Instruction Following): Final stage involved training on 1.2M tokens from
koto-instruct-sft using r32/a32 QLoRA at 4k context for 1 epoch, applied to all linear modules.
Good For
- Developers and writers seeking a model optimized for generating creative text, including stories, character dialogues, and adventure prompts.
- Applications requiring robust roleplaying capabilities and varied narrative styles.