iRASC/Llama-Ko-8B
iRASC/Llama-Ko-8B is an 8 billion parameter language model developed by iRASC, created by merging a top-performing multilingual LLM with a specialized Korean language model using the DARE technique. This model is optimized for enhancing reasoning capabilities, particularly in complex tasks like mathematical problem-solving, demonstrating a 1.69% average performance improvement across six benchmarks and over 20% higher performance on GSM8K. It leverages the inherent complexity of the Korean language to boost LLM reasoning abilities, making it suitable for applications requiring advanced analytical skills.
Loading preview...
iRASC/Llama-Ko-8B: Enhanced Reasoning Through Korean Language Model Merging
iRASC/Llama-Ko-8B is an 8 billion parameter language model developed through a novel merging approach that integrates a top-ranking multilingual LLM with a specialized Korean language model. This process utilizes the DARE (Drop and REscale) technique to efficiently combine models by minimizing delta parameter redundancy. The core innovation lies in demonstrating that incorporating the Korean language model significantly enhances the reasoning capabilities of the merged LLM.
Key Capabilities & Performance
- Improved Reasoning: The model shows a 1.69% average performance improvement across six benchmark tasks and a notable 20% higher performance on GSM8K, which demands complex reasoning skills. This suggests that the linguistic features of Korean contribute to stronger analytical abilities.
- DARE Merging: The model was created using the DARE-TIES merge method, specifically combining
swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITAwithbeomi/Llama-3-Open-Ko-8Bat an optimal density value of 0.25. - Benchmark Scores: Achieves an average score of 76.23 on the Open LLM Leaderboard, including 75.17 on AI2 Reasoning Challenge, 91.78 on HellaSwag, 66.84 on MMLU, 71.95 on TruthfulQA, 82.24 on Winogrande, and 69.37 on GSM8k.
Use Cases
This model is particularly well-suited for applications requiring advanced reasoning and problem-solving, especially in contexts where the nuanced linguistic features of Korean can be leveraged to improve analytical performance. Its enhanced GSM8K scores indicate strong potential for mathematical and logical reasoning tasks.