Nitral-Archive/Eris-Daturamix-7b
Eris-Daturamix-7b is a 7 billion parameter language model created by Nitral-Archive, resulting from a merge of Test157t/Eris-Floramix-7b and ResplendentAI/Datura_7B. This model leverages a slerp merge method with specific parameter weighting for self_attn and mlp layers. It is designed for general language tasks, combining the strengths of its constituent models.
Loading preview...
Eris-Daturamix-7b: Merged Language Model
Eris-Daturamix-7b is a 7 billion parameter language model developed by Nitral-Archive, created through a strategic merge of two distinct base models: Test157t/Eris-Floramix-7b and ResplendentAI/Datura_7B.
Merge Configuration
The model was constructed using a slerp (spherical linear interpolation) merge method. This process involved combining the full 32 layers of both source models. A key aspect of its configuration is the differentiated weighting applied to specific architectural components:
- Self-attention layers (
self_attn): These layers were merged with varyingtvalues, specifically[0, 0.5, 0.3, 0.7, 1]across different filters. - MLP layers (
mlp): These layers also received distincttvalues, specifically[1, 0.5, 0.7, 0.3, 0]across different filters. - General parameters: A default
tvalue of0.5was applied to other parameters.
This precise merging strategy aims to integrate the unique characteristics and capabilities of both Eris-Floramix-7b and Datura_7B into a single, cohesive model. The model was produced with bfloat16 dtype.
Good for
- General language generation tasks
- Exploring merged model performance