CorticalStack/neurotic-crown-clown-7b-ties
CorticalStack's neurotic-crown-clown-7b-ties is a 7 billion parameter language model created using the TRIM, ELECT SIGN & MERGE (TIES) method. This merge combines mlabonne/NeuralMonarch-7B, mlabonne/AlphaMonarch-7B, and bardsai/jaskier-7b-dpo-v5.6, leveraging their strengths. It is designed for general language tasks, offering a balanced performance profile derived from its constituent models.
Loading preview...
Overview
CorticalStack/neurotic-crown-clown-7b-ties is a 7 billion parameter language model developed using the TIES-Merging method. This technique, detailed in the paper "TIES-Merging: Resolving Interference When Merging Models," combines multiple pre-trained models to create a new model that aims to leverage the strengths of its components while mitigating interference.
Key Components
This model is a merge of three distinct 7B parameter models:
- mlabonne/NeuralMonarch-7B
- mlabonne/AlphaMonarch-7B
- bardsai/jaskier-7b-dpo-v5.6
Merge Configuration
The TIES merge was performed with specific density and weight parameters for each contributing model, using mlabonne/NeuralMonarch-7B as the base model. The configuration ensures normalization and uses float16 for efficiency. This approach allows for a nuanced combination of the underlying models' capabilities, potentially leading to improved performance across various language understanding and generation tasks.