OpenLLM-Ro/RoLlama2-7b-Base
OpenLLM-Ro/RoLlama2-7b-Base is a 7 billion parameter foundational generative text model developed by OpenLLM-Ro, specifically designed and continually pretrained for the Romanian language. Based on the Llama-2-7b architecture, it represents the first open-source effort to build a large language model specialized for Romanian. This model excels in various Romanian natural language tasks, demonstrating strong performance in areas like multiclass classification, machine translation (RO-EN), and question answering (XQuAD finetuned).
Loading preview...
RoLlama2-7b-Base: A Foundational Romanian LLM
OpenLLM-Ro/RoLlama2-7b-Base is a 7 billion parameter foundational model developed by OpenLLM-Ro, marking the first open-source initiative to create a large language model specialized for Romanian. This model is continually pretrained from Llama-2-7b using the CulturaX dataset, focusing exclusively on the Romanian language.
Key Capabilities and Performance
- Romanian Language Specialization: Designed from the ground up for Romanian, offering a strong base for various NLP tasks in this language.
- Competitive Benchmarks: Outperforms the original Llama-2-7b in several Romanian-specific benchmarks and downstream tasks:
- Achieves an average score of 38.03 in academic benchmarks, surpassing Llama-2-7b's 37.04.
- Demonstrates superior performance in multiclass classification (61.04 Macro F1) and XQuAD finetuned F1 (79.96).
- Shows improved machine translation from Romanian to English (RO-EN Bleu) in finetuned settings.
- Research-Oriented: Intended primarily for research use, providing a robust base model that can be adapted for diverse natural language applications.
Intended Use Cases
- Research and Development: Ideal for researchers and developers working on Romanian NLP projects.
- Adaptation for Specific Tasks: Can be fine-tuned for a wide array of Romanian natural language tasks, including text generation, classification, and question answering.
- Foundation for Instruct/Chat Models: Serves as the base for instruction-tuned and chat variants within the RoLlama2 family, enabling assistant-like applications.