orai-nlp/Gemma-Kimu-9b-it

Model Overview

orai-nlp/Gemma-Kimu-9b-it is a 9 billion parameter instruction-tuned large language model (LLM) specifically designed for the Basque language. It is built upon Google's Gemma-2-9b foundational and instruct models. The development approach involves continually pre-training the foundational LLM on Basque data while maintaining English proficiency through replay, followed by injecting instruction-following capabilities via delta-based weight merging from the instructed base LLM.

Key Capabilities & Features

• Basque Language Specialization: Optimized for instruction following, safety, and linguistic correctness in Basque.
• Efficient Adaptation: Utilizes a novel method that decouples language adaptation from post-training alignment, merging post-training deltas into the language-adapted model.
• Bilingual Training: Continually pre-trained on a combination of Basque (ZelaiHandi dataset, 1.5 billion tokens) and English (FineWeb dataset, 300 million tokens) data to improve cross-lingual transfer.

Performance

Evaluations using the NoRobotsEU benchmark show significant improvements in Basque instruction following:

• Gemma-Kimu-9b-it: Achieves 71% on Instruct follow. EU, compared to 57% for Gemma-2-9b-it.

Good for

• Applications requiring high-quality instruction following and linguistic accuracy in the Basque language.
• Developers looking for a specialized LLM for Basque-centric tasks, leveraging the strengths of the Gemma-2 architecture.