OpenLLM-Ro/RoLlama2-7b-Base

Hugging Face
TEXT GENERATIONConcurrency Cost:1Model Size:7BQuant:FP8Ctx Length:4kPublished:Oct 9, 2024License:llama2Architecture:Transformer0.0K Open Weights Warm

OpenLLM-Ro/RoLlama2-7b-Base is a 7 billion parameter foundational generative text model developed by OpenLLM-Ro, specifically designed for the Romanian language. Continually pretrained from Llama-2-7b using the CulturaX dataset, it represents the first open-source effort to build a large language model specialized for Romanian. This model excels in Romanian-specific natural language tasks, demonstrating improved performance over Llama-2-7b in several Romanian downstream tasks and academic benchmarks like ARC and Hellaswag. It is intended for research use in Romanian, serving as a base model adaptable for various NLP applications.

Loading preview...

OpenLLM-Ro/RoLlama2-7b-Base: A Foundational Romanian LLM

OpenLLM-Ro/RoLlama2-7b-Base is a 7 billion parameter foundational generative text model developed by OpenLLM-Ro, marking the first open-source initiative to create a large language model specialized for Romanian. This model is continually pretrained from Meta's Llama-2-7b, leveraging the extensive CulturaX dataset to enhance its proficiency in Romanian.

Key Capabilities and Features

  • Romanian Language Specialization: Designed specifically for Romanian, offering improved performance on Romanian natural language tasks compared to its English-centric predecessor.
  • Foundational Model: Serves as a base model, suitable for adaptation to a wide array of natural language processing tasks.
  • Academic Benchmarks: Outperforms Llama-2-7b in several academic benchmarks, including ARC (37.95 vs 36.05) and Hellaswag (57.22 vs 48.00), and shows stronger performance in specific Romanian downstream tasks like LaRoSeDa Multiclass (61.04 vs 54.11 few-shot, 87.72 vs 87.22 finetuned) and WMT EN-RO (27.85 vs 24.95 finetuned).
  • Open-Source Development: Part of a broader family of OpenLLM-Ro models, including instruct and chat variants, all publicly released under the Llama2 Community License Agreement.

Intended Use Cases

  • Research in Romanian NLP: Ideal for academic and research purposes focused on the Romanian language.
  • Adaptation for NLP Tasks: Can be fine-tuned or adapted for various natural language tasks in Romanian.
  • Foundation for Specialized Models: Serves as a strong base for developing more specialized Romanian LLMs, such as instruction-tuned or chat models.

Finetunes
3 models