etiennebamas/qwen3-lr-2-e-5
etiennebamas/qwen3-lr-2-e-5 is an 8 billion parameter language model, fine-tuned from formalmathatepfl/qwen3-cpt. This model is trained with a learning rate of 2e-05 and a context length of 32768 tokens. It is optimized for tasks related to its sft dataset, making it suitable for applications requiring specialized language understanding based on its fine-tuning data.
Loading preview...
Model Overview
etiennebamas/qwen3-lr-2-e-5 is an 8 billion parameter language model, fine-tuned from the formalmathatepfl/qwen3-cpt base model. This fine-tuning process utilized a specific sft dataset, indicating a specialization for tasks aligned with that data.
Training Details
The model was trained with the following key hyperparameters:
- Learning Rate: 2e-05
- Batch Size: 1 (train), 8 (eval)
- Optimizer: AdamW with betas=(0.9, 0.999) and epsilon=1e-08
- LR Scheduler: Cosine type with a warmup ratio of 0.05
- Epochs: 1.0
- Distributed Training: Multi-GPU setup with 8 devices, resulting in a total effective batch size of 8 for training and 64 for evaluation.
Framework Versions
The training was conducted using:
- Transformers 4.57.3
- Pytorch 2.9.0+cu128
- Datasets 3.6.0
- Tokenizers 0.22.2
Intended Use
Given its fine-tuning on an sft dataset, this model is likely best suited for tasks that align with the characteristics and content of that specific dataset. Users should consider the nature of the sft data when evaluating its applicability for their use cases.