RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_test_pipeline_1k_steps
RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_test_pipeline_1k_steps is a 7 billion parameter language model, fine-tuned from RefalMachine's mistral_darulm_20_05_24_part1-2_32000_bpe_mean_init_03_07_24. This model was trained for 1000 steps, achieving a validation loss of 2.2672 and an accuracy of 0.5391. It is a specialized iteration focusing on specific data characteristics, though the exact dataset remains unspecified.
Loading preview...
Model Overview
This model, RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_test_pipeline_1k_steps, is a 7 billion parameter language model. It is a fine-tuned variant of RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_mean_init_03_07_24, developed by RefalMachine.
Training Details
The model underwent a focused training regimen of 1000 steps. Key hyperparameters included a learning rate of 0.0001, a train_batch_size of 4, and an Adam optimizer. Training was distributed across 16 devices, resulting in a total effective batch size of 64. The learning rate scheduler used a cosine decay with 100 warmup steps.
Performance Metrics
During evaluation, the model achieved a validation loss of 2.2672 and an accuracy of 0.5391. These metrics reflect its performance on the specific, though currently unspecified, evaluation dataset.
Current Status
As indicated in the original model card, further information regarding the model's description, intended uses, limitations, and the specific training and evaluation data is needed to fully understand its capabilities and optimal applications. This model represents an intermediate step in a pipeline, likely for testing or specific research purposes.