Model Overview
orai-nlp/Gemma-Kimu-9b-it is a 9 billion parameter instruction-tuned large language model (LLM) specifically designed for the Basque language. It is built upon Google's Gemma-2-9b foundational and instruct models. The development approach involves continually pre-training the foundational LLM on Basque data while maintaining English proficiency through replay, followed by injecting instruction-following capabilities via delta-based weight merging from the instructed base LLM.
Key Capabilities & Features
- Basque Language Specialization: Optimized for instruction following, safety, and linguistic correctness in Basque.
- Efficient Adaptation: Utilizes a novel method that decouples language adaptation from post-training alignment, merging post-training deltas into the language-adapted model.
- Bilingual Training: Continually pre-trained on a combination of Basque (ZelaiHandi dataset, 1.5 billion tokens) and English (FineWeb dataset, 300 million tokens) data to improve cross-lingual transfer.
Performance
Evaluations using the NoRobotsEU benchmark show significant improvements in Basque instruction following:
- Gemma-Kimu-9b-it: Achieves 71% on Instruct follow. EU, compared to 57% for Gemma-2-9b-it.
Good for
- Applications requiring high-quality instruction following and linguistic accuracy in the Basque language.
- Developers looking for a specialized LLM for Basque-centric tasks, leveraging the strengths of the Gemma-2 architecture.