trinhkhng/linear_Merged_Qwen2-0.5B_0.1
trinhkhng/linear_Merged_Qwen2-0.5B_0.1 is a 0.5 billion parameter language model with a 32768 token context length, created by trinhkhng using a linear merge of two Qwen2-0.5B base models. This model is specifically designed through a weighted linear merge, combining a standard Qwen2-0.5B with a debiased version. It is intended for applications requiring a compact yet capable model with potentially reduced bias characteristics.
Loading preview...
Overview
trinhkhng/linear_Merged_Qwen2-0.5B_0.1 is a compact 0.5 billion parameter language model with a substantial 32768 token context length. It was developed by trinhkhng using the MergeKit tool, specifically employing the Linear merge method.
Merge Details
This model is a composite of two distinct Qwen2-0.5B base models:
- A standard
/kaggle/working/Qwen2-0.5Bmodel, contributing 90% of the weight. - A debiased
/kaggle/working/debias_Qwen2-0.5Bmodel, contributing 10% of the weight.
The linear merge strategy, as described in the Linear merge paper, was applied with normalization to combine these components. The tokenizer from the base Qwen2-0.5B model was retained.
Key Characteristics
- Parameter Count: 0.5 billion parameters, making it suitable for resource-constrained environments.
- Context Length: Supports a long context window of 32768 tokens.
- Merge Method: Utilizes a weighted linear merge to combine model strengths.
- Bias Mitigation: Incorporates a debiased version of the base model, suggesting an intent to reduce unwanted biases.
Potential Use Cases
This model is well-suited for applications where a small footprint and efficient inference are critical, while also benefiting from the characteristics introduced by the debiased component. It could be particularly useful for:
- Edge device deployment: Its small size makes it viable for on-device AI.
- Research into model merging: Provides a practical example of linear merging with specific weighting.
- Applications requiring reduced bias: The inclusion of a debiased model suggests suitability for tasks where fairness and neutrality are important considerations.