Hugging Face Amadeus-Verbo-BI-Qwen2.5-32B-PT-BR-Instruct Overview
This model, developed by amadeusai, is a 32.5 billion parameter Brazilian Portuguese (PT-BR) language model. It is built upon the Qwen2.5-32B base architecture, fine-tuned for two epochs using a 600,000-instruction dataset. The model utilizes a Transformer-based architecture incorporating RoPE, SwiGLU, RMSNorm, and Attention QKV bias.
Key Technical Specifications
- Architecture: Transformer-based with RoPE, SwiGLU, RMSNorm, Attention QKV bias.
- Parameters: 32.5 billion (31.0 billion non-embedding).
- Context Length: 131,072 tokens, enabling extensive contextual understanding.
- Language: Specifically optimized for Brazilian Portuguese.
- Training: Fine-tuned for 2 epochs on 600k instructions.
Usage and Integration
The model is designed for use with the HuggingFace Transformers library. Users are advised to use the latest version of Transformers (avoiding versions older than 4.37.0 to prevent KeyError: 'qwen2'). Example code snippets are provided for loading the model and generating text using pipeline, AutoTokenizer, and AutoModelForCausalLM.
Research and Citation
Further details on the model's development and performance can be found in the associated research article: Amadeus-Verbo Technical Report: The powerful Qwen2.5 family models trained in Portuguese.