formalmathatepfl/qwen3-8b-feedback-non-base-with-repair
The formalmathatepfl/qwen3-8b-feedback-non-base-with-repair is an 8 billion parameter Qwen3 model, fine-tuned from formalmathatepfl/qwen3-8b-post-trained-cpt. This model is specifically fine-tuned on an sft dataset, indicating an optimization for supervised fine-tuning tasks. It is designed for applications requiring a robust language model with a 32K context window, leveraging its fine-tuned nature for specific use cases.
Loading preview...
Model Overview
This model, formalmathatepfl/qwen3-8b-feedback-non-base-with-repair, is an 8 billion parameter Qwen3-based language model. It is a fine-tuned variant of the formalmathatepfl/qwen3-8b-post-trained-cpt model, specifically optimized through supervised fine-tuning (SFT) on a dedicated dataset. The model was trained with a learning rate of 2e-05 over 1 epoch, utilizing a cosine learning rate scheduler with a 0.05 warmup ratio.
Key Characteristics
- Base Model: Qwen3-8b architecture.
- Fine-tuning: Supervised fine-tuning (SFT) from a post-trained checkpoint.
- Parameters: 8 billion.
- Context Length: Supports a 32,768 token context window.
Training Details
The training process involved a batch size of 1 per device across 8 GPUs, resulting in a total training batch size of 8. An AdamW optimizer with specific betas and epsilon values was used. The model leverages Transformers 4.57.3, Pytorch 2.9.0+cu128, Datasets 4.0.0, and Tokenizers 0.22.2.