Model Overview
nbeerbower/mistral-nemo-gutenberg-12B-v3 is a 12 billion parameter language model built upon the intervitens/mini-magnum-12b-v1.1 base model. It has been further fine-tuned using the jondurbin/gutenberg-dpo-v0.1 dataset, focusing on instruction following and general language generation tasks. The training process involved 3 epochs on an A100 GPU via Google Colab, utilizing methods similar to those for fine-tuning Llama 3 with ORPO.
Key Capabilities & Performance
This model is designed for general-purpose language understanding and generation, with a notable context length of 32768 tokens. Its performance on the Open LLM Leaderboard provides insights into its capabilities across various benchmarks:
- Average Score: 19.06
- IFEval (0-Shot): 21.83
- BBH (3-Shot): 34.96
- MATH Lvl 5 (4-Shot): 4.61
- GPQA (0-shot): 8.61
- MuSR (0-shot): 15.00
- MMLU-PRO (5-shot): 29.38
These metrics suggest its utility in tasks requiring instruction adherence, common-sense reasoning, and basic mathematical understanding. The model's fine-tuning on the Gutenberg DPO dataset likely enhances its ability to generate coherent and contextually relevant text.
When to Use This Model
This model is suitable for applications requiring a balance of instruction following and general text generation. Its performance on benchmarks indicates it can be a viable option for tasks such as:
- Instruction-based text generation
- General question answering
- Content creation where adherence to prompts is important
For detailed evaluation results, refer to the Open LLM Leaderboard details.