RefalMachine/llm_test_unigram
RefalMachine/llm_test_unigram is a 7 billion parameter language model, fine-tuned from a llama7b_rulm_spm_unigram base model. It was trained for 1 epoch, achieving a final validation loss of 2.9526 and an accuracy of 0.4388. This model is a fine-tuned variant, with its specific primary differentiator and intended use cases requiring further information from the developer.
Loading preview...
Model Overview
RefalMachine/llm_test_unigram is a 7 billion parameter language model, fine-tuned from an existing llama7b_rulm_spm_unigram base model. The fine-tuning process involved 1 epoch of training, resulting in a final validation loss of 2.9526 and an accuracy of 0.4388.
Training Details
The model was trained using the following key hyperparameters:
- Learning Rate: 0.0003
- Batch Size: 12 (train and eval)
- Optimizer: Adam with betas=(0.9, 0.95) and epsilon=1e-05
- LR Scheduler: Linear with 200 warmup steps
- Epochs: 1.0
Training was conducted across 10 devices with a gradient accumulation of 2 steps, leading to a total effective batch size of 240. The training results show a gradual decrease in validation loss and an increase in accuracy over 121,000 steps.
Current Status
Further information regarding the model's specific description, intended uses, limitations, and the dataset used for training and evaluation is currently marked as "More information needed" in the original model card. Users should consult the developer for these details to understand its primary capabilities and suitable applications.