Muse-12B: Narrative-Focused Language Model
Muse-12B, developed by LatitudeGames, is a 12 billion parameter model based on the Mistral Nemo architecture, specifically engineered for rich, coherent narrative generation. It was trained through a multi-stage process involving Supervised Fine-Tuning (SFT) and two distinct Direct Preference Optimization (DPO) phases.
Key Capabilities & Training:
- Narrative Coherence: Excels at maintaining consistent character relationships and emotional depth over long contexts (up to 32768 tokens).
- Fine-tuning: Initial SFT used diverse multi-turn datasets, including text adventures, long emotional narratives, and general roleplay, carefully balanced and rewritten to avoid common AI clichés.
- DPO Stages:
- Gutenberg DPO: Introduced human writing techniques, enhancing stylistic quality.
- Reward Model User Preference Data DPO: Refined the model, restoring intelligence while preserving enhanced writing quality.
- Versatile Writing: Produces natural, emotionally resonant text across various genres, particularly strong in character-driven scenarios.
Good For:
- Creative Writing: Generating detailed stories, roleplay scenarios, and text adventures.
- Emotional Depth: Crafting narratives where character relationships and emotions are central.
- Long-form Content: Producing verbose responses (often 1000+ tokens) with sustained coherence.
Limitations:
- Primarily trained on second-person present tense data, which may lead to suboptimal results in other narrative styles.
- Average response lengths tend to be verbose, though this can be managed with system prompts.
LatitudeGames plans to continue improving and open-sourcing similar models, encouraging user feedback for further enhancements. Quantized GGUF weights are available for download.