Overview

OpenHermes 2.5 Mistral 7B, developed by Teknium, is an advanced fine-tune of the Mistral 7B architecture. It builds upon the OpenHermes 2 model by incorporating additional code datasets, leading to notable improvements across various benchmarks.

Key Capabilities and Performance

• Enhanced Generalist Performance: Training on a specific ratio of code instruction data unexpectedly boosted several non-code benchmarks, including TruthfulQA, AGIEval, and the GPT4All suite.
• Improved Code Generation: The model's HumanEval score for code tasks increased from 43% to 50.7% @ Pass 1, demonstrating significant progress in coding capabilities.
• High-Quality Training Data: Fine-tuned on 1,000,000 entries, primarily GPT-4 generated data, alongside other high-quality open datasets, with extensive filtering and conversion to ShareGPT format.
• ChatML Support: Utilizes the ChatML prompt format, enabling structured multi-turn dialogue and compatibility with OpenAI's API, making system prompts effective for guiding model behavior.

Benchmark Highlights

• GPT4All: Achieved an average score of 73.12.
• AGIEval: Scored an average of 43.07%.
• TruthfulQA: Reached 53.04% on the mc2 metric.
• Overall Improvement: Outperforms previous OpenHermes models (except Hermes 70B) and many current Mistral fine-tunes, with a total average score of 52.38 across key benchmarks.

Use Cases

This model is well-suited for a wide range of applications requiring strong conversational abilities, reasoning, and code generation. Its generalist improvements make it versatile for tasks from programming assistance to creative writing and complex dialogue.