OpenLLM-Ro/RoLlama2-7b-Base-2024-05-14
OpenLLM-Ro/RoLlama2-7b-Base-2024-05-14 is a 7 billion parameter foundational generative text model developed by OpenLLM-Ro, specifically designed for the Romanian language. Continually pretrained from Llama-2-7b using the CulturaX dataset, this model represents the first open-source effort to build a large language model specialized for Romanian. It is intended for research use in Romanian natural language tasks, serving as a base model adaptable for various applications.
Loading preview...
Overview
OpenLLM-Ro/RoLlama2-7b-Base-2024-05-14 is a 7 billion parameter foundational generative text model developed by OpenLLM-Ro. It is the first open-source large language model specialized for the Romanian language, continually pretrained from Meta's Llama-2-7b using the CulturaX dataset. This model is part of the broader RoLlama2 family, which includes instruct and chat variants.
Key Capabilities and Features
- Romanian Language Specialization: Developed specifically for Romanian, addressing a gap in open-source LLMs.
- Foundational Model: Serves as a base model adaptable for a variety of natural language tasks in Romanian.
- Research-Oriented: Primarily intended for research use, with potential for adaptation to specific applications.
- Performance: While Llama-2-7b generally outperforms it on MMLU and GSM8k, RoLlama2-7b-Base-2024-05-14 shows stronger performance on ARC, Winogrande, Hellaswag, and TruthfulQA benchmarks, indicating improved general reasoning and common sense for Romanian contexts. It also demonstrates superior performance in finetuned downstream tasks like LaRoSeDa (multiclass) and WMT (RO-EN Bleu).
Intended Use Cases
- Research: Ideal for academic and experimental work involving Romanian natural language processing.
- Adaptation: Can be fine-tuned or adapted for specific Romanian NLP tasks.
- Base for Instruction/Chat Models: Serves as the foundation for instruct and chat-tuned variants within the RoLlama2 family.