etiennebamas/qwen3-sft-classic-small-data-equal-compute
The etiennebamas/qwen3-sft-classic-small-data-equal-compute model is an 8 billion parameter language model, fine-tuned from formalmathatepfl/qwen3-cpt. It was trained using a cosine learning rate scheduler and AdamW optimizer over 2.15 epochs. This model is designed for general language understanding and generation tasks, leveraging its Qwen3 architecture and 32768 token context length.
Loading preview...
Model Overview
This model, etiennebamas/qwen3-sft-classic-small-data-equal-compute, is an 8 billion parameter language model. It is a fine-tuned variant of the formalmathatepfl/qwen3-cpt base model, specifically adapted using an sft dataset. The training process involved 2.15 epochs, utilizing a learning rate of 2e-05, a batch size of 1 per device across 8 GPUs, and an AdamW optimizer with betas=(0.9, 0.999) and epsilon=1e-08. A cosine learning rate scheduler with a 0.05 warmup ratio was employed.
Key Training Details
- Base Model:
formalmathatepfl/qwen3-cpt - Parameters: 8 billion
- Learning Rate: 2e-05
- Optimizer: AdamW (betas=(0.9, 0.999), epsilon=1e-08)
- LR Scheduler: Cosine with 0.05 warmup ratio
- Epochs: 2.15
- Frameworks: Transformers 4.57.3, Pytorch 2.9.0+cu128, Datasets 4.0.0, Tokenizers 0.22.2
Intended Use Cases
While specific intended uses and limitations are not detailed in the provided information, as a fine-tuned language model, it is generally suitable for a range of natural language processing tasks. Developers should consider its 8B parameter count and 32768 token context length for applications requiring substantial contextual understanding and generation capabilities.