Hebrew_Nemo: State-of-the-Art Hebrew Language Model
Hebrew_Nemo is a 12 billion parameter causal language model developed by SicariusSicariiStuff, specifically optimized for Hebrew language understanding and generation. Built on the Mistral Nemo architecture, it combines robust multilingual foundations with extensive Hebrew-specific fine-tuning. The model is released under an Apache 2.0 license, aiming to democratize advanced Hebrew NLP capabilities.
Key Capabilities & Performance
Hebrew_Nemo demonstrates state-of-the-art performance for its size, notably achieving a BLEU score of 30.83 in Hebrew translation, outperforming significantly larger models like DeepSeek-14B and AI21 Jamba-Mini (52B). It maintains high competence in reasoning and QA, with an SNLI accuracy of 79.76 and a HeQ score of 70.51. The model also shows exceptional knowledge in Israeli Trivia (50.83), closely matching models over four times its size. Compared to the base Mistral-Nemo-Base, Hebrew_Nemo shows substantial improvements, including a +13.2% increase in average score and a +22.4% increase in Israeli Trivia.
Training and Features
Training involved continued pre-training on a diverse, Hebrew-rich corpus including Wikipedia, literature, code-mix data, and synthetic data, followed by instruction fine-tuning and alignment. Key features include:
- Native Hebrew Understanding: Trained on millions of high-quality Hebrew documents.
- Contextual Mastery: Handles complex anaphora, idiomatic expressions, and mixed Hebrew-English text.
- Instruction-Tuned: Aligned for chat, Q&A, summarization, and reasoning.
- Cultural Awareness: Sensitive to Hebrew cultural, religious, and social nuances.
Primary Use Cases
- High-quality Hebrew text generation
- Bidirectional Hebrew translation
- Question answering in Hebrew contexts
- Conversational AI for Hebrew speakers
- Text classification and named entity recognition for Hebrew content