vicgalle/OpenHermes-Qwen1.5-1.8B

Model Overview

The vicgalle/OpenHermes-Qwen1.5-1.8B is a 1.8 billion parameter language model built upon the Qwen1.5 architecture. It is designed for general-purpose language tasks, offering a balance of performance in a relatively small footprint. The model's evaluation on the Open LLM Leaderboard indicates its capabilities across several key metrics.

Key Performance Metrics

Evaluated on the Hugging Face Open LLM Leaderboard, the model achieved an average score of 44.95. Specific benchmark results include:

• AI2 Reasoning Challenge (25-Shot): 37.80
• HellaSwag (10-Shot): 59.73
• MMLU (5-Shot): 45.80
• TruthfulQA (0-shot): 42.28
• Winogrande (5-shot): 60.22
• GSM8k (5-shot): 23.88

These scores highlight its proficiency in common sense reasoning, multiple-choice question answering, and general knowledge tasks, while indicating areas for potential improvement in complex mathematical reasoning (GSM8k).

Use Cases

This model is well-suited for applications where a smaller, efficient language model is preferred without sacrificing too much performance. Potential use cases include:

• Text generation: Creating coherent and contextually relevant text.
• Question Answering: Responding to queries based on general knowledge.
• Reasoning tasks: Handling basic logical and common sense reasoning problems.
• Prototyping and development: A good choice for initial development due to its manageable size and balanced performance.