shidowake/test-240114-mergekit-neural-japanese-stablelm-gamma-7b
The shidowake/test-240114-mergekit-neural-japanese-stablelm-gamma-7b is a 7 billion parameter language model created by shidowake, resulting from a SLERP merge of stabilityai/japanese-stablelm-instruct-gamma-7b, Intel/neural-chat-7b-v3-3, and mistralai/Mistral-7B-v0.1. This model leverages the strengths of its constituent models, combining Japanese language instruction following with general chat capabilities and Mistral's base architecture. It is designed for tasks requiring a blend of multilingual understanding and robust conversational interaction, particularly in Japanese contexts.
Loading preview...
Model Overview
This model, shidowake/test-240114-mergekit-neural-japanese-stablelm-gamma-7b, is a 7 billion parameter language model created through a SLERP merge using mergekit. It combines the capabilities of three distinct base models to offer a unique blend of features.
Key Components and Merge Strategy
The merge incorporates:
- stabilityai/japanese-stablelm-instruct-gamma-7b: Providing strong Japanese language instruction-following abilities.
- Intel/neural-chat-7b-v3-3: Contributing general conversational and chat-optimized performance.
- mistralai/Mistral-7B-v0.1: Serving as the foundational base model, known for its efficient architecture.
The SLERP (Spherical Linear Interpolation) merge method was applied, with specific weighting parameters for different tensor types (self_attn, mlp) to balance the contributions of the source models. The merge was configured to use bfloat16 for efficiency.
Potential Use Cases
This merged model is particularly well-suited for applications that require:
- Japanese language processing: Leveraging the
japanese-stablelm-instruct-gamma-7bcomponent for tasks in Japanese. - Instruction following: Benefiting from the instruction-tuned nature of its merged parts.
- General conversational AI: Utilizing the
neural-chat-7b-v3-3for robust dialogue systems. - Multilingual contexts: Where a combination of strong Japanese understanding and general language capabilities is advantageous.