lewtun/mistral-7b-sft-ultrachat-arithmo-50
lewtun/mistral-7b-sft-ultrachat-arithmo-50 is a 7 billion parameter language model fine-tuned from mistralai/Mistral-7B-v0.1. This model was specifically trained on a combination of the UltraChat and Arithmo datasets (50% Arithmo) to enhance its conversational abilities and mathematical reasoning. It is designed for tasks requiring both general chat interaction and arithmetic problem-solving.
Loading preview...
Model Overview
lewtun/mistral-7b-sft-ultrachat-arithmo-50 is a 7 billion parameter language model built upon the mistralai/Mistral-7B-v0.1 architecture. This model has undergone supervised fine-tuning (SFT) using a specialized dataset blend, incorporating both the UltraChat dataset for general conversational capabilities and the Arithmo dataset, which comprises 50% of the training data, to improve its arithmetic reasoning.
Key Capabilities
- Enhanced Conversational Skills: Benefits from the UltraChat dataset, making it suitable for interactive chat applications.
- Improved Arithmetic Reasoning: The significant inclusion of the Arithmo dataset aims to bolster its performance on mathematical and logical tasks.
- Mistral-7B Foundation: Inherits the strong base performance and efficiency of the Mistral-7B architecture.
Training Details
The model was trained with a learning rate of 2e-05, a batch size of 8, and a single epoch. It utilized Adam optimizer with betas=(0.9, 0.999) and epsilon=1e-08, along with a cosine learning rate scheduler with a 0.1 warmup ratio. During training, a validation loss of 0.8892 was observed.
Good For
- Applications requiring a balance of general conversational interaction and numerical problem-solving.
- Chatbots that need to handle basic arithmetic queries or logical reasoning within conversations.
- Developers looking for a Mistral-7B variant with a specific focus on improving mathematical capabilities through fine-tuning.