cemig-temp/qwen3.5-4b-regulacao-normativa

Model Overview

This model, cemig-temp/qwen3.5-4b-regulacao-normativa, is a 4.5 billion parameter language model built upon the Qwen3.5 architecture. It has been specifically fine-tuned for specialized tasks within the Brazilian energy sector, focusing on regulatory and distribution normative content. The training involved a two-stage process, with this model representing the second stage of refinement on domain-specific data.

Key Capabilities

• Domain-Specific Understanding: Optimized for processing and generating text related to Brazilian energy regulation and distribution norms.
• Context Length: Supports a substantial context window of 32768 tokens, enabling the processing of longer documents and complex queries.
• Training Data: Fine-tuned on proprietary datasets including TokenLab/CemigConvoV1.1 (regulacao split) and /workspace/pipeline/data/distribuicao/norm_tecnicas_dist.jsonl, ensuring high relevance to its intended application.
• Performance Metrics: Achieved a validation loss of 0.6464 and a perplexity (PPL) of 1.9087 on its evaluation set, indicating strong performance on its specialized domain.

Training Details

The model was trained using axolotl with a learning rate of 5e-06, a micro batch size of 8, and a total training batch size of 32 over 1 epoch. It utilized adamw_torch_fused optimizer and a cosine learning rate scheduler with 20 warmup steps. Flash Attention 2 was implemented for efficient attention computation. The training process involved freezing the vision encoder, focusing refinement on the language model components.