wvnvwn/llama-2-13b-chat-hf-lr5e-5-resta-0.3
The wvnvwn/llama-2-13b-chat-hf-lr5e-5-resta-0.3 is a 13 billion parameter Llama-2-based language model, created by wvnvwn, that was produced by merging three distinct Llama-2-13b-chat-hf variants using a linear merge method. This model integrates the base Llama-2-13b-chat-hf with versions fine-tuned for specific tasks, including one for GSM8K (mathematical reasoning) and another for SSFT. It is designed to combine the strengths of its constituent models, potentially enhancing performance in areas like chat and mathematical problem-solving.
Loading preview...
Model Overview
The wvnvwn/llama-2-13b-chat-hf-lr5e-5-resta-0.3 is a 13 billion parameter language model derived from the Llama-2 architecture. It was created by wvnvwn through a linear merge of three distinct Llama-2-13b-chat-hf variants using the MergeKit tool.
Key Merge Details
This model is a composite of:
- The foundational
meta-llama/Llama-2-13b-chat-hfmodel. wvnvwn/llama-2-13b-chat-hf-SSFT-lr5e-5, a version likely fine-tuned for specific supervised instruction following.wvnvwn/llama-2-13b-chat-hf-lr5e-5-gsm8k-lr5e-5, a variant specifically fine-tuned for the GSM8K dataset, indicating an emphasis on mathematical reasoning capabilities.
Configuration
The merge utilized a float16 dtype and applied specific weights to each component model across all 40 layers. Notably, the base Llama-2-13b-chat-hf model was included with a negative weight, suggesting an attempt to subtract or de-emphasize certain characteristics of the base model while integrating the fine-tuned components.
Potential Use Cases
This merged model is potentially well-suited for applications requiring:
- Enhanced chat capabilities due to its Llama-2-chat base.
- Improved mathematical reasoning and problem-solving, benefiting from the GSM8K fine-tuning.
- General-purpose instruction following, leveraging the SSFT-tuned component.