Model Overview

The sedrickkeh/mistral_openhermes_v3 is a 7 billion parameter language model, fine-tuned from the base mistralai/Mistral-7B-v0.1 architecture. It was trained with a learning rate of 5e-06 over 3 epochs, utilizing a total batch size of 1024 across 8 GPUs. The model achieved a final validation loss of 0.5579.

Training Details

• Base Model: mistralai/Mistral-7B-v0.1
• Parameters: 7 billion
• Context Length: 4096 tokens (inferred from base model)
• Optimizer: Adam with betas=(0.9, 0.999) and epsilon=1e-08
• Learning Rate Scheduler: Constant with 0.1 warmup ratio
• Epochs: 3.0
• Final Validation Loss: 0.5579

Limitations

Detailed information regarding the specific training dataset, intended uses, and limitations is not provided in the model card. Users should exercise caution and conduct further evaluation to determine its suitability for specific applications.