RefalMachine/ruadapt_qwen2.5_3B_bpe_32000_full_lr2e4_bs256
RefalMachine/ruadapt_qwen2.5_3B_bpe_32000_full_lr2e4_bs256 is a 3.1 billion parameter language model, fine-tuned from RefalMachine's ruadapt_qwen2.5_3B_bpe_32000_mean_init. This model utilizes a BPE tokenizer with a vocabulary of 32000 and supports a context length of 32768 tokens. It was trained with a focus on achieving a low loss and improved accuracy, demonstrating a final validation loss of 2.4027 and an accuracy of 0.5129 on its evaluation set. The model's specific applications and primary differentiators beyond its fine-tuning process are not detailed in the provided information.
Loading preview...
Model Overview
This model, RefalMachine/ruadapt_qwen2.5_3B_bpe_32000_full_lr2e4_bs256, is a fine-tuned variant of the RefalMachine/ruadapt_qwen2.5_3B_bpe_32000_mean_init base model. It is a 3.1 billion parameter language model designed with a BPE tokenizer supporting a vocabulary size of 32,000 and a substantial context length of 32,768 tokens.
Training Details
The model underwent a single epoch of training using specific hyperparameters:
- Learning Rate: 0.0002
- Optimizer: Adam with betas=(0.9, 0.95) and epsilon=1e-05
- Batch Size: A total training batch size of 256 (with
train_batch_size: 2andgradient_accumulation_steps: 2across 64 devices). - Scheduler: Cosine learning rate scheduler with 100 warmup steps.
Performance Metrics
During training, the model achieved the following results on its evaluation set:
- Final Validation Loss: 2.4027
- Final Accuracy: 0.5129
Framework Versions
The training was conducted using:
- Transformers 4.37.2
- Pytorch 2.3.0a0+6ddf5cf85e.nv24.04
- Datasets 2.18.0
- Tokenizers 0.15.2
Further details regarding the specific dataset used for fine-tuning, intended uses, and limitations are not provided in the available documentation.