RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_part1_lr5e5_bs256
RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_part1_lr5e5_bs256 is a 7 billion parameter language model, fine-tuned from a Mistral base model by RefalMachine. This model was fine-tuned on an unspecified dataset, achieving a validation loss of 2.0859 and an accuracy of 0.5605. It is suitable for tasks requiring a moderately sized language model with a 4096-token context length, though its specific primary use case is not detailed.
Loading preview...
Model Overview
RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_part1_lr5e5_bs256 is a 7 billion parameter language model developed by RefalMachine. It is a fine-tuned variant of the RefalMachine/mistral_darulm_20_05_24_part1-2_32000_bpe_mean_init_03_07_24 base model. The fine-tuning process involved 1.0 epoch with a learning rate of 5e-05 and a total batch size of 128 across 32 devices.
Key Characteristics
- Base Architecture: Fine-tuned from a Mistral-based model.
- Parameter Count: 7 billion parameters.
- Context Length: Supports a context window of 4096 tokens.
- Performance Metrics: Achieved a validation loss of 2.0859 and an accuracy of 0.5605 on its evaluation set.
- Training Details: Utilized Adam optimizer with betas=(0.9, 0.95) and epsilon=1e-05, and a cosine learning rate scheduler with 100 warmup steps.
Use Cases and Limitations
While the specific dataset used for fine-tuning is not disclosed, and detailed intended uses are not provided, this model can be considered for general language understanding and generation tasks where a 7B parameter model is appropriate. Users should be aware that the model's specific strengths and limitations beyond its reported loss and accuracy are not explicitly defined in the available documentation.