Model Overview
The openaccess-ai-collective/neft-exp3 model is a fine-tuned iteration of the mistralai/Mistral-7B-v0.1 base model, developed by OpenAccess-AI-Collective. It was trained using the Axolotl framework, indicating a focus on efficient and accessible fine-tuning practices. The model achieved a final validation loss of 1.3754 over 4 epochs.
Training Details
The fine-tuning process utilized a learning rate of 6e-06, with a train_batch_size and eval_batch_size of 2 across 8 GPUs, resulting in a total batch size of 16 for both training and evaluation. The optimizer used was Adam with standard betas and epsilon, employing a cosine learning rate scheduler with 10 warmup steps. The training involved 247 steps per epoch, totaling 4 epochs.
Key Characteristics
- Base Model: Mistral-7B-v0.1, a 7 billion parameter model known for its strong performance in its size class.
- Training Framework: Built with Axolotl, a popular tool for fine-tuning large language models.
- Performance Metric: Achieved a validation loss of 1.3754, indicating its performance on the evaluation dataset.
Limitations
The model card explicitly states that more information is needed regarding its specific intended uses, limitations, and the training and evaluation data utilized. Users should exercise caution and conduct further evaluation before deploying this model for critical applications, as its specific strengths and weaknesses are not yet fully documented.