nbeerbower/SmolNemo-12B-FFT-experimental
SmolNemo-12B-FFT-experimental by nbeerbower is a 12.2 billion parameter experimental language model with a 32768 token context length. It is a full finetune of Mahou-1.5-mistral-nemo-12B-lorablated on the HuggingFaceTB/smoltalk dataset. This model is explicitly noted as having erratic behavior and poor performance, serving primarily as a public experiment in training parameters rather than a production-ready solution.
Loading preview...
SmolNemo-12B-FFT-experimental Overview
SmolNemo-12B-FFT-experimental is an experimental 12.2 billion parameter language model developed by nbeerbower. It is a full finetune of the Mahou-1.5-mistral-nemo-12B-lorablated model, trained on the HuggingFaceTB/smoltalk dataset. This model is shared as part of an ongoing exploration into training parameters and ideas, rather than a formal release.
Key Characteristics
- Experimental Nature: This model is explicitly labeled as an experiment, not intended for production use.
- Performance: It exhibits erratic behavior and poor performance, with the developer noting potential conflicts with Mistral Instruct and ChatML as contributing factors.
- Training Method: The model underwent Supervised Fine-Tuning (SFT) for 0.1 epochs using 8x A100 GPUs.
- Context Length: It supports a context length of 32768 tokens.
Evaluation Results
According to the Open LLM Leaderboard, the model shows low performance across various metrics:
- Avg.: 8.32
- IFEval (0-Shot): 33.48
- BBH (3-Shot): 6.54
- MATH Lvl 5 (4-Shot): 0.23
- GPQA (0-shot): 1.34
- MuSR (0-shot): 5.92
- MMLU-PRO (5-shot): 2.41
Good for
- Research and Experimentation: Ideal for developers and researchers interested in observing the effects of specific training parameters and finetuning approaches.
- Learning: Useful for understanding potential challenges and outcomes in model training, particularly regarding conflicts between instruction-tuning formats.
- Transparency: Provides insight into an iterative development process for language models.