NorMistral-7b-warm: A Specialized Norwegian Language Model

NorMistral-7b-warm is a 7 billion parameter language model developed by the Language Technology Group at the University of Oslo, in collaboration with the HPLT project, the National Library of Norway, and the University of Turku. It is built upon the Mistral-7b-v0.1 architecture and has undergone extensive continual pretraining on a massive dataset of 260 billion subword tokens, primarily consisting of open Norwegian texts, augmented with 20% code data from Starcoder. This makes it a highly specialized model for Norwegian language processing.

Key Capabilities

• Strong Norwegian Language Proficiency: Continuously pretrained on 260 billion Norwegian subword tokens, making it highly adept at understanding and generating Norwegian text.
• Mistral-based Architecture: Leverages the optimized transformer architecture of Mistral/Llama models for efficient performance.
• Multilingual Translation: Demonstrates strong performance in machine translation tasks involving Norwegian (Bokmål and Nynorsk) to and from English, as well as between Norwegian dialects.
• Research-Oriented: Primarily intended for research, with instruction-finetuned versions planned for future release.

Good for

• Norwegian NLP Applications: Ideal for tasks requiring deep understanding and generation of Norwegian, such as sentiment analysis, reading comprehension, and grammatical error correction in Norwegian.
• Machine Translation (Norwegian-centric): Particularly effective for translation between English and Norwegian, and for dialectal translation within Norwegian.
• Academic and Research Projects: A valuable base model for researchers exploring Norwegian language models and their applications.