CorticalStack/pikus-pikantny-7B-dare
CorticalStack/pikus-pikantny-7B-dare is a 7 billion parameter language model created by CorticalStack, formed by a DARE merge of several existing models including bardsai/jaskier-7b-dpo-v5.6 and mlabonne/NeuralDaredevil-7B. This model leverages the DARE (DARE: Drop and Restore) merging technique, as described in the "Language Models are Super Mario" paper, to combine abilities from its constituent models. It is designed to integrate diverse capabilities from its base models, offering a consolidated solution for various language generation tasks.
Loading preview...
Overview
pikus-pikantny-7B-dare is a 7 billion parameter language model developed by CorticalStack. It is constructed using a DARE (DARE: Drop and Restore) merge method, combining the strengths of multiple pre-existing models. This approach, detailed in the paper "Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch", allows the model to inherit and integrate diverse capabilities from its components.
Key Components
The model is a merge of the following base models:
- bardsai/jaskier-7b-dpo-v5.6
- mlabonne/NeuralDaredevil-7B
- Gille/StrangeMerges_21-7B-slerp
- CultriX/NeuralTrix-7B-dpo
Merge Configuration
The merge was performed using mergekit with a dare_ties method. The configuration specifies int8_mask: true and dtype: bfloat16, indicating optimizations for efficiency and numerical precision. The base model for the merge was bardsai/jaskier-7b-dpo-v5.6.