OpenLLM-Ro/RoLlama2-7b-Base: A Foundational Romanian LLM

OpenLLM-Ro/RoLlama2-7b-Base is a 7 billion parameter foundational generative text model developed by OpenLLM-Ro, marking the first open-source initiative to create a large language model specialized for Romanian. This model is continually pretrained from Meta's Llama-2-7b, leveraging the extensive CulturaX dataset to enhance its proficiency in Romanian.

Key Capabilities and Features

• Romanian Language Specialization: Designed specifically for Romanian, offering improved performance on Romanian natural language tasks compared to its English-centric predecessor.
• Foundational Model: Serves as a base model, suitable for adaptation to a wide array of natural language processing tasks.
• Academic Benchmarks: Outperforms Llama-2-7b in several academic benchmarks, including ARC (37.95 vs 36.05) and Hellaswag (57.22 vs 48.00), and shows stronger performance in specific Romanian downstream tasks like LaRoSeDa Multiclass (61.04 vs 54.11 few-shot, 87.72 vs 87.22 finetuned) and WMT EN-RO (27.85 vs 24.95 finetuned).
• Open-Source Development: Part of a broader family of OpenLLM-Ro models, including instruct and chat variants, all publicly released under the Llama2 Community License Agreement.

Intended Use Cases

• Research in Romanian NLP: Ideal for academic and research purposes focused on the Romanian language.
• Adaptation for NLP Tasks: Can be fine-tuned or adapted for various natural language tasks in Romanian.
• Foundation for Specialized Models: Serves as a strong base for developing more specialized Romanian LLMs, such as instruction-tuned or chat models.