shivash/moetest1
shivash/moetest1 is an 8 billion parameter language model created by shivash, derived from a merge of meta-llama/Llama-3.1-8B-Instruct using the SLERP method. This model leverages a layered task arithmetic merge strategy to combine different weightings of the base Llama 3.1 Instruct model across its layers. It is designed to inherit the general capabilities of the Llama 3.1-8B-Instruct architecture, with potential optimizations from the specific merging configuration.
Loading preview...
Overview
shivash/moetest1 is an 8 billion parameter language model developed by shivash. It is a merged model, specifically created using the MergeKit tool and the SLERP (Spherical Linear Interpolation) merge method.
Merge Details
This model is a composite of a single base model, meta-llama/Llama-3.1-8B-Instruct, which has been merged with itself using a layered approach. The merging process involved applying different task_arithmetic weights across three distinct layer ranges (0-8, 9-20, and 21-31) of the Llama 3.1-8B-Instruct model. The overall merge used a t parameter of 0.5 for the SLERP method, and the model's weights are in bfloat16 format.
Key Characteristics
- Base Architecture: Inherits from the Llama 3.1-8B-Instruct family.
- Parameter Count: 8 billion parameters.
- Context Length: Supports a context length of 32768 tokens.
- Merge Method: Utilizes SLERP for the primary merge, with layered task arithmetic for specific weight adjustments.
Potential Use Cases
Given its foundation in Llama 3.1-8B-Instruct and the merging technique, shivash/moetest1 is likely suitable for general-purpose instruction following tasks, including:
- Text generation and completion.
- Question answering.
- Summarization.
- Chatbot applications.