RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_part1_lr1e4_bs256
RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_part1_lr1e4_bs256 is a 7 billion parameter language model, fine-tuned from RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_mean_init_03_07_24. This model was trained with a 4096 token context length and achieved a validation loss of 2.0608 and an accuracy of 0.5629 on its evaluation set. Its specific primary use case and training dataset are not detailed in the available information.
Loading preview...
Model Overview
This model, RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_part1_lr1e4_bs256, is a 7 billion parameter language model. It is a fine-tuned version of RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_mean_init_03_07_24.
Training Details
The model was trained using the following key hyperparameters:
- Learning Rate: 0.0001
- Batch Size: 4 (train and eval), with a total effective batch size of 128 across 32 devices
- Optimizer: Adam with betas=(0.9, 0.95) and epsilon=1e-05
- Scheduler: Cosine learning rate scheduler with 100 warmup steps
- Epochs: 1.0
Performance
On the evaluation set, the model achieved a final validation loss of 2.0608 and an accuracy of 0.5629. During training, the validation loss steadily decreased from 2.1902 at 2000 steps to its final value, while accuracy improved from 0.5446 to 0.5629.
Limitations
The specific dataset used for fine-tuning, the intended uses, and further limitations are not detailed in the provided model information.