Model Overview

rishiraj/smol-7b is a 7 billion parameter instruction-tuned language model, fine-tuned from openchat/openchat_3.5. It was trained by rishiraj between December 1st and 3rd, 2023, utilizing the HuggingFaceH4/no_robots dataset and recipes from The Alignment Handbook.

Key Capabilities & Performance

This model demonstrates strong performance, particularly in reasoning and general language understanding. At the time of its release, smol-7b was the highest-ranked 7B chat model on the MMLU Benchmark, achieving a score of 65. Its overall average score on the Open LLM Leaderboard is 67.11, with notable scores in ARC (63.74), HellaSwag (84.77), and GSM8K (62.32).

Training Details

The model was trained with a learning rate of 2e-05, a batch size of 4 (total effective batch size of 512 with gradient accumulation), and for 1 epoch. The optimizer used was Adam with betas=(0.9, 0.999) and epsilon=1e-08, with a cosine learning rate scheduler.