rishiraj/smol-7b
rishiraj/smol-7b is a 7 billion parameter instruction-tuned causal language model developed by rishiraj. It is a fine-tuned version of openchat/openchat_3.5, trained on the HuggingFaceH4/no_robots dataset. This model achieves a notable 65 on the MMLU benchmark, making it the highest-ranked 7B chat model on MMLU at its release. It is optimized for general chat applications and reasoning tasks.
Loading preview...
Model Overview
rishiraj/smol-7b is a 7 billion parameter instruction-tuned language model, fine-tuned from openchat/openchat_3.5. It was trained by rishiraj between December 1st and 3rd, 2023, utilizing the HuggingFaceH4/no_robots dataset and recipes from The Alignment Handbook.
Key Capabilities & Performance
This model demonstrates strong performance, particularly in reasoning and general language understanding. At the time of its release, smol-7b was the highest-ranked 7B chat model on the MMLU Benchmark, achieving a score of 65. Its overall average score on the Open LLM Leaderboard is 67.11, with notable scores in ARC (63.74), HellaSwag (84.77), and GSM8K (62.32).
Training Details
The model was trained with a learning rate of 2e-05, a batch size of 4 (total effective batch size of 512 with gradient accumulation), and for 1 epoch. The optimizer used was Adam with betas=(0.9, 0.999) and epsilon=1e-08, with a cosine learning rate scheduler.