Model Overview
uukuguy/speechless-thoughts-mistral-7b-v1.0 is a 7 billion parameter Mistral-based language model, fine-tuned by uukuguy. It features a maximum context length of 8192 tokens and was developed as a foundational model for the larger speechless-sparsetral-16x7b-MoE. The fine-tuning process utilized a diverse set of datasets, including jondurbin/airoboros-2.2 (filtered for coding, reasoning, and planning), Open-Orca/OpenOrca (specifically the 'cot' category), garage-bAInd/Open-Platypus, WizardLM/WizardLM_evol_instruct_V2_196k (coding conversation), TokenBender/python_eval_instruct_51k, Spider, and codefuse-ai/Evol-Instruction-66k.
Key Capabilities
- Reasoning and Planning: Optimized through datasets like Airoboros and OpenOrca's CoT category.
- Code Generation: Enhanced with extensive training on coding-related datasets such as WizardLM, TokenBender/python_eval_instruct_51k, and Spider.
- Instruction Following: Fine-tuned using an Alpaca-style prompt format for clear instruction adherence.
Performance Highlights
Evaluations on the Open LLM Leaderboard show an average score of 59.36. Notable scores include:
- HellaSwag (10-shot): 81.25
- MMLU (5-shot): 54.59
- Winogrande (5-shot): 78.14
Good for
- Developing applications requiring strong reasoning and planning abilities.
- Code generation and understanding tasks, particularly in Python.
- As a baseline for further fine-tuning or research into larger sparse Mixture-of-Experts models.