nebchi/solar-merge-slerp
nebchi/solar-merge-slerp is a 10.7 billion parameter language model created by nebchi through a SLERP merge of davidkim205/nox-solar-10.7b-v2 and chihoonlee10/T3Q-ko-solar-dpo-v6.0. This model leverages the strengths of its constituent models, combining their learned representations to potentially enhance performance across various tasks. Its primary use case is for applications requiring a robust, merged model derived from established SOLAR-based architectures, offering a 4096 token context length.
Loading preview...
Overview
nebchi/solar-merge-slerp is a 10.7 billion parameter language model resulting from a merge of two pre-trained models: davidkim205/nox-solar-10.7b-v2 and chihoonlee10/T3Q-ko-solar-dpo-v6.0. This model was created using the mergekit tool.
Merge Method
The model was constructed using the SLERP (Spherical Linear Interpolation) merge method. This technique combines the weights of the base models in a specific, non-linear way, often leading to a balanced integration of their respective capabilities.
Configuration Details
The merge process involved a detailed configuration, applying varying interpolation values (t) across different layers and components of the neural network. Specifically, the self_attn (self-attention) layers and mlp (multi-layer perceptron) layers were weighted differently, indicating a fine-tuned approach to blending the characteristics of the source models. The base model for the merge was chihoonlee10/T3Q-ko-solar-dpo-v6.0.
Good for
- Experimenting with merged models based on the SOLAR architecture.
- Leveraging combined strengths of
nox-solar-10.7b-v2andT3Q-ko-solar-dpo-v6.0. - Applications where a 10.7B parameter model with a 4096 token context window is suitable.