Hermes 2 Pro - Mistral 7B Overview

Hermes 2 Pro - Mistral 7B is a 7 billion parameter language model developed by Nous Research, @interstellarninja, and Fireworks.AI. It represents an upgraded and retrained version of Nous Hermes 2, leveraging an updated OpenHermes 2.5 Dataset and a newly developed Function Calling and JSON Mode dataset. The model maintains strong general task and conversation capabilities while significantly enhancing its performance in specific areas.

Key Capabilities

• Function Calling: Achieves 90% accuracy on function calling evaluations, utilizing a specialized system prompt and multi-turn structure with a new ChatML role for reliable parsing. Detailed guidance and inference code are available on the NousResearch/Hermes-Function-Calling GitHub repository.
• JSON Structured Outputs: Scores 84% accuracy on structured JSON output evaluations, designed to respond with only a JSON object adhering to a given schema.
• General Conversation: Retains excellent performance in general conversational tasks.
• ChatML Format: Uses the ChatML prompt format, enabling structured multi-turn dialogue and OpenAI endpoint compatibility.

Benchmarks

The model demonstrates competitive performance across various benchmarks, including an average of 71.19 on GPT4All, 44.52 on AGIEval, and 41.65 on BigBench. These metrics, alongside its high accuracy in function calling and JSON mode, highlight its specialized strengths.

Good for

• Applications requiring robust function calling capabilities.
• Scenarios demanding structured JSON outputs based on defined schemas.
• General-purpose conversational AI where reliable and parseable interactions are crucial.