Marcjoni/SingularitySynth-12B
SingularitySynth-12B is a 12-billion parameter language model developed by Marcjoni, created by merging DreadPoor/Irix-12B-Model_Stock and yamatazen/EtherealAurora-12B-v2 using the TIES merging method. This model is designed for general text generation tasks, supporting a stable context length of 12,000 tokens. It utilizes a bfloat16 dtype and is configured with specific layer-wise weighting for attention, MLP, and embedding layers to optimize its performance.
Loading preview...
SingularitySynth-12B: A Merged Language Model
SingularitySynth-12B is a 12-billion parameter language model developed by Marcjoni, constructed through a merge of existing models. It combines DreadPoor/Irix-12B-Model_Stock and yamatazen/EtherealAurora-12B-v2 using the TIES merging method via LazyMergekit.
Key Characteristics:
- Architecture: A merged model leveraging the strengths of its constituent base models.
- Parameter Count: 12 billion parameters.
- Context Length: Stable performance up to 12,000 tokens, with potential for extended contexts.
- Data Type: Utilizes
bfloat16for efficient computation. - Prompt Format: Supports ChatML style messages for structured input.
- Merge Configuration: Specific weighting applied to different layers (attention, MLP, embed_tokens) from the source models to fine-tune its behavior.
Recommended Usage:
This model is suitable for a variety of text generation tasks. For optimal results, the recommended sampling settings include a temperature range of 0.75 to 1.25 and a Min P of 0.035. Its stable 12k token context window makes it versatile for applications requiring longer conversational histories or document processing.