SingularitySynth-12B: A Merged Language Model

SingularitySynth-12B is a 12-billion parameter language model developed by Marcjoni, constructed through a merge of existing models. It combines DreadPoor/Irix-12B-Model_Stock and yamatazen/EtherealAurora-12B-v2 using the TIES merging method via LazyMergekit.

Key Characteristics:

• Architecture: A merged model leveraging the strengths of its constituent base models.
• Parameter Count: 12 billion parameters.
• Context Length: Stable performance up to 12,000 tokens, with potential for extended contexts.
• Data Type: Utilizes bfloat16 for efficient computation.
• Prompt Format: Supports ChatML style messages for structured input.
• Merge Configuration: Specific weighting applied to different layers (attention, MLP, embed_tokens) from the source models to fine-tune its behavior.

Recommended Usage:

This model is suitable for a variety of text generation tasks. For optimal results, the recommended sampling settings include a temperature range of 0.75 to 1.25 and a Min P of 0.035. Its stable 12k token context window makes it versatile for applications requiring longer conversational histories or document processing.