formalmathatepfl/qwen3-cpt
The formalmathatepfl/qwen3-cpt is an 8 billion parameter language model, fine-tuned from Qwen/Qwen3-8B-Base on the cpt_data dataset. This model is designed for general language tasks, leveraging its Qwen3 architecture and a 32768 token context length. Its fine-tuning process aimed to optimize performance, achieving a final validation loss of 0.7642.
Loading preview...
Model Overview
The formalmathatepfl/qwen3-cpt is an 8 billion parameter language model, fine-tuned from the Qwen/Qwen3-8B-Base architecture. It was trained on the cpt_data dataset, indicating a specialization or optimization for tasks related to this specific data.
Key Training Details
This model underwent a fine-tuning process with the following notable hyperparameters:
- Learning Rate: 3e-06
- Optimizer: ADAMW_TORCH_FUSED with betas=(0.9, 0.999) and epsilon=1e-08
- Batch Size: A total training batch size of 16 (2 per device across 8 GPUs)
- Epochs: 2.0
- LR Scheduler: Cosine type with a warmup ratio of 0.03
Performance
During its training, the model achieved a final validation loss of 0.7642. The training loss progressively decreased, demonstrating effective learning over 2 epochs and 3300 steps. The evaluation loss also showed a consistent downward trend, stabilizing around the final reported value.