etiennebamas/qwen3-lr-2-e-5-neat-packing
The etiennebamas/qwen3-lr-2-e-5-neat-packing model is an 8-billion parameter language model, fine-tuned from formalmathatepfl/qwen3-cpt. This model was trained with a learning rate of 2e-05 and a cosine learning rate scheduler over one epoch. It is based on the Qwen3 architecture and has a context length of 32768 tokens.
Loading preview...
Model Overview
This model, etiennebamas/qwen3-lr-2-e-5-neat-packing, is an 8-billion parameter language model. It is a fine-tuned version of the formalmathatepfl/qwen3-cpt model, indicating its foundation in the Qwen3 architecture.
Training Details
The fine-tuning process involved specific hyperparameters:
- Learning Rate: 2e-05
- Optimizer: ADAMW_TORCH with betas=(0.9, 0.999) and epsilon=1e-08
- LR Scheduler: Cosine type with a warmup ratio of 0.05
- Epochs: 1.0
- Batch Size: A
train_batch_sizeof 1 andeval_batch_sizeof 8 were used, leading to atotal_train_batch_sizeof 8 andtotal_eval_batch_sizeof 64 across 8 devices.
The model was trained using Transformers 4.57.3, Pytorch 2.9.0+cu128, Datasets 3.6.0, and Tokenizers 0.22.2.
Current Status
As per the provided information, further details regarding the model's description, intended uses, limitations, and training/evaluation data are needed to fully understand its specific capabilities and optimal applications.