eganwo/mistral7b-distilled-from-deepseek-r1-qwen32b
The eganwo/mistral7b-distilled-from-deepseek-r1-qwen32b model is a 7 billion parameter language model fine-tuned from Mistral-7B-Instruct-v0.3. This model was distilled from a larger DeepSeek-R1-Qwen32B model, aiming to retain performance in a smaller footprint. It is designed for general language generation tasks, leveraging the Mistral architecture for efficient inference.
Loading preview...
Model Overview
This model, eganwo/mistral7b-distilled-from-deepseek-r1-qwen32b, is a 7 billion parameter language model. It is a fine-tuned version of the mistralai/Mistral-7B-Instruct-v0.3 base model. A key characteristic of this model is its distillation process, where it was trained to mimic the behavior of a larger DeepSeek-R1-Qwen32B model.
Training Details
The model was fine-tuned using the following hyperparameters:
- Learning Rate: 5e-06
- Batch Size: 1 (train), 8 (eval)
- Gradient Accumulation Steps: 4
- Optimizer: AdamW with default betas and epsilon
- LR Scheduler: Linear
- Epochs: 1
Framework Versions
The training utilized:
- Transformers 4.50.0
- Pytorch 2.6.0
- Datasets 3.4.1
- Tokenizers 0.21.1
Intended Use
While specific intended uses and limitations are not detailed in the provided information, as a distilled Mistral-7B variant, it is generally suitable for a range of natural language processing tasks where a balance between performance and computational efficiency is desired. Its distillation from a larger model suggests an aim to achieve strong performance relative to its size.