nbeerbower/mistral-nemo-gutenberg-12B-v3

Model Overview

nbeerbower/mistral-nemo-gutenberg-12B-v3 is a 12 billion parameter language model built upon the intervitens/mini-magnum-12b-v1.1 base model. It has been further fine-tuned using the jondurbin/gutenberg-dpo-v0.1 dataset, focusing on instruction following and general language generation tasks. The training process involved 3 epochs on an A100 GPU via Google Colab, utilizing methods similar to those for fine-tuning Llama 3 with ORPO.

Key Capabilities & Performance

This model is designed for general-purpose language understanding and generation, with a notable context length of 32768 tokens. Its performance on the Open LLM Leaderboard provides insights into its capabilities across various benchmarks:

• Average Score: 19.06
• IFEval (0-Shot): 21.83
• BBH (3-Shot): 34.96
• MATH Lvl 5 (4-Shot): 4.61
• GPQA (0-shot): 8.61
• MuSR (0-shot): 15.00
• MMLU-PRO (5-shot): 29.38

These metrics suggest its utility in tasks requiring instruction adherence, common-sense reasoning, and basic mathematical understanding. The model's fine-tuning on the Gutenberg DPO dataset likely enhances its ability to generate coherent and contextually relevant text.

When to Use This Model

This model is suitable for applications requiring a balance of instruction following and general text generation. Its performance on benchmarks indicates it can be a viable option for tasks such as:

• Instruction-based text generation
• General question answering
• Content creation where adherence to prompts is important

For detailed evaluation results, refer to the Open LLM Leaderboard details.