Hugging Face Overview
Overview
Llammas-base is a 7 billion parameter language model built upon the Llama-2 architecture. Developed by tartuNLP, this model has undergone significant continued pre-training using 5 billion tokens from the CulturaX dataset. The dataset composition is notable, consisting of 75% Estonian and 25% English documents, which facilitates cross-lingual knowledge transfer.
Key Capabilities
- Bilingual Proficiency: Enhanced performance in Estonian due to extensive pre-training on Estonian text, while maintaining English language understanding.
- Foundation Model: Serves as the base for the instruction-tuned Llammas model.
- Research-Backed: The model's development and methodology are detailed in a dedicated research paper.
Good For
- Applications requiring strong Estonian language processing.
- Research into cross-lingual knowledge transfer and adaptation of large language models.
- As a foundational model for further fine-tuning on specific Estonian or bilingual tasks.