PORTULAN/gervasio-8b-portuguese-ptpt-decoder
PORTULAN/gervasio-8b-portuguese-ptpt-decoder is an 8 billion parameter decoder model developed by the NLX-Natural Language and Speech Group at the University of Lisbon, Portugal. Based on the LLaMA 3.1 8B Instruct architecture, it is specifically fine-tuned for the Portuguese language using datasets like extraGLUE-Instruct and MMLU PT. This model excels in Portuguese language generation and can be run on consumer-grade hardware, making it suitable for various applications requiring Portuguese text processing.
Loading preview...
Gervásio 8B PTPT: A Portuguese Language Decoder
Gervásio 8B PTPT is an open decoder model developed by the NLX-Natural Language and Speech Group at the University of Lisbon, Portugal. It is built upon the LLaMA 3.1 8B Instruct architecture, featuring 8 billion parameters, 32 layers, and a 32768-token context length. The model is specifically enhanced for the Portuguese language through additional training on curated datasets.
Key Capabilities and Training
- Portuguese Language Specialization: Fine-tuned extensively on Portuguese language resources, including extraGLUE-Instruct, MMLU PT, Natural Instructions PT, and a Wikipedia subset.
- Architecture: A Transformer-based decoder from the LLaMA family, utilizing a RoPE tokenizer with a vocabulary size of 128256.
- Training Method: Supervised fine-tuning with a causal language modeling objective, employing a zero-out technique where only response tokens were back-propagated. Training was accelerated using FSDP across 10 L40S GPUs.
Performance Highlights
Gervásio 8B PTPT demonstrates strong performance on Portuguese benchmarks, outperforming its base model, LLaMA 3.1 8B Instruct, in several key areas:
| Model | GPQA Diamond PT | MMLU PT | MMLU Pro PT | CoPA | MRPC | RTE | Average |
|---|---|---|---|---|---|---|---|
| Gervásio 8B PTPT | 34.85 | 62.15 | 36.79 | 87.00 | 77.45 | 77.62 | 62.64 |
| LLaMA 3.1 8B Instruct | 32.32 | 61.49 | 36.10 | 83.00 | 75.25 | 79.42 | 61.26 |
Good For
- Portuguese Text Generation: Excels at generating coherent and contextually relevant text in Portuguese.
- Research and Commercial Applications: Distributed under an open license, suitable for both academic research and commercial deployment.
- Consumer-Grade Hardware: Its 8B parameter size allows it to be run on readily available consumer hardware.
- Chatbot Integration: Integrated into the Evaristo.ai chatbot for interactive experimentation.