Marcjoni/QuasiStarSynth-12B
Marcjoni/QuasiStarSynth-12B is a 12 billion parameter merged language model, created by Marcjoni, combining several 12B models including EtherealAurora-12B-v2, Irix-12B-Model_Stock, Violet-Lyra-Gutenberg-v2, and patricide-12B-Unslop-Mell-v2. This model is configured with specific layer-wise weighting for attention and MLP blocks, and supports a stable context length of 12k tokens. It is designed for general text generation tasks, utilizing a ChatML prompt format.
Loading preview...
QuasiStarSynth-12B Overview
QuasiStarSynth-12B is a 12 billion parameter language model developed by Marcjoni. It is a sophisticated merge of four distinct 12B models: yamatazen/EtherealAurora-12B-v2, DreadPoor/Irix-12B-Model_Stock, ohyeah1/Violet-Lyra-Gutenberg-v2, and redrix/patricide-12B-Unslop-Mell-v2. This merging process, utilizing LazyMergekit with a ties method, involves specific layer-wise weighting for attention and MLP blocks, aiming to combine the strengths of its constituent models.
Key Capabilities
- Merged Architecture: Leverages a unique
tiesmerge method with custom layer-specific weights for attention and MLP layers, potentially enhancing performance across various tasks. - Context Length: Stable operation at 12,000 tokens, with potential for extended context support.
- Prompt Format: Supports the widely used ChatML style for conversational interactions.
- Sampling Flexibility: Recommended sampling settings include a temperature range of 0.75 to 1.25 and a minimum P of 0.035, allowing for diverse output generation.
Good For
- General Text Generation: Suitable for a broad range of text generation tasks due to its merged architecture.
- Exploration of Merged Models: Ideal for developers interested in experimenting with models created through advanced merging techniques and custom layer configurations.
- Chat-based Applications: Its ChatML support makes it well-suited for integration into conversational AI systems.