Hermes 2 Pro - Llama-3 8B Overview

Hermes 2 Pro - Llama-3 8B is an 8 billion parameter language model developed through a collaboration between Nous Research, @interstellarninja, and Fireworks.AI. This model is an enhanced version of Nous Hermes 2, incorporating an updated and cleaned OpenHermes 2.5 Dataset, alongside a new in-house developed Function Calling and JSON Mode dataset.

Key Capabilities

• Function Calling: Achieves 90% on internal function calling evaluations, utilizing a special system prompt and multi-turn structure with new ChatML roles (<tools>, <tool_call>, <tool_response>).
• JSON Structured Outputs: Scores 84% on structured JSON output evaluations, designed to respond with only a JSON object based on a provided schema.
• General Task & Conversation: Maintains strong performance in general conversational tasks.
• ChatML Format: Uses ChatML for structured multi-turn dialogue, offering OpenAI endpoint compatibility.

Benchmarks

The model demonstrates competitive performance across various benchmarks:

• GPT4All Average: 72.62
• AGIEval Average: 42.44
• BigBench Average: 43.55
• TruthfulQA: mc1 0.410, mc2 0.578

Use Cases

This model is particularly well-suited for applications requiring reliable function calling, structured data extraction in JSON format, and general conversational AI. Its agentic capabilities, supported by specific tokens for parsing while streaming, make it valuable for complex interactive systems. Further details on function calling implementation are available on the NousResearch/Hermes-Function-Calling GitHub repository.