The daze-unlv/medmcqa-alignment-lora-7b-2-epoch model is a fine-tuned version of Mistral-7B-v0.1, developed by daze-unlv. This model was trained for two epochs using a LoRA adaptation on a generator dataset, with a learning rate of 0.0002 and a total batch size of 8 across 4 GPUs. Its primary use case is specialized alignment, though specific details on its intended applications and performance metrics beyond training loss are not provided.
Loading preview...
medmcqa-alignment-lora-7b-2-epoch
This model is a fine-tuned variant of the Mistral-7B-v0.1 base model, developed by daze-unlv. It underwent a LoRA (Low-Rank Adaptation) training process for two epochs, specifically on a generator dataset. While the training achieved a reported loss of 0.0, further details regarding its specific performance metrics or evaluation results are not available.
Key Capabilities
- Fine-tuned Mistral-7B-v0.1: Leverages the architecture and pre-training of the Mistral-7B-v0.1 model.
- LoRA Adaptation: Utilizes LoRA for efficient fine-tuning, suggesting a focus on adapting the base model to specific tasks without full retraining.
- Two-Epoch Training: Indicates a focused training duration on the provided generator dataset.
Good For
- Specialized Alignment Tasks: Potentially suitable for tasks requiring alignment to the characteristics of the generator dataset it was trained on.
- Research and Experimentation: Useful for researchers exploring LoRA fine-tuning techniques on the Mistral-7B architecture for specific data distributions.
Training Details
The model was trained with a learning rate of 0.0002, a batch size of 1 per device (totaling 8 across 4 GPUs with gradient accumulation), and a cosine learning rate scheduler with a 0.1 warmup ratio. The training utilized PEFT 0.7.1, Transformers 4.36.2, Pytorch 2.1.2, Datasets 2.14.6, and Tokenizers 0.15.2.