Nos-PT/Llama-Carvalho-GL
Nos-PT/Llama-Carvalho-GL is an 8-billion parameter transformer-based causal language model developed by Nos-PT, built upon Meta's Llama-3.1-8B. It is continually pretrained with a multilingual corpus of 340 million tokens, specifically emphasizing Galician, making it specialized for Galician, Portuguese, Spanish, and English. This model is designed for causal language modeling and text generation tasks, particularly excelling in scenarios requiring proficiency in these Iberian languages.
Loading preview...
Nos-PT/Llama-Carvalho-GL: Multilingual LLM for Iberian Languages
Nos-PT/Llama-Carvalho-GL is an 8-billion parameter causal language model, part of the Carvalho family of LLMs. It is a continually pretrained version of Meta's Llama-3.1-8B, specifically enhanced for Galician, Portuguese, Spanish, and English.
Key Capabilities & Features
- Multilingual Proficiency: Specialized in Galician, Portuguese (PT), Spanish, and English, with a training corpus heavily weighted towards Galician (74% of the base corpus).
- Causal Language Modeling: Ready-to-use for text generation tasks and adaptable for fine-tuning in specific applications.
- Foundation Model: Built on the robust Llama-3.1-8B architecture.
- Training Details: Trained using HuggingFace Transformers and PyTorch with DeepSpeed, on a corpus of 340 million tokens.
Intended Use Cases
- Text Generation: Ideal for generating content in Galician, Portuguese, Spanish, and English.
- Language-Specific Applications: Suitable for tasks requiring deep understanding and generation in the specified languages, especially Galician.
- Further Fine-tuning: Can be fine-tuned for specialized downstream tasks or domain-specific applications within its supported languages.
Limitations
- Currently evaluated as "In process," so specific performance benchmarks are not yet available.
This model was developed within the Nós Project, funded by the Ministerio para la Transformación Digital y de la Función Pública – EU NextGenerationEU.