AGI-Eval/UNO-Scorer-Qwen3-14B
UNO-Scorer, developed by AGI-Eval, is a 14-billion parameter LLM-based evaluation model built on the Qwen3-14B architecture. It is fine-tuned to automate the evaluation of Large Multimodal Models (LMMs) by comparing model responses against reference answers across 6 distinct question types, including Multi-Step Open-Ended Questions. This model provides numerical scores and detailed reasoning, demonstrating superior accuracy compared to models like GPT-4.1 in specific evaluation domains.
Loading preview...
UNO-Scorer: A Unified Evaluation Model
UNO-Scorer is a 14-billion parameter LLM-based evaluation model, developed by AGI-Eval and built upon the Qwen3-14B backbone. It is specifically designed to automate the assessment of Large Multimodal Models (LMMs) with high precision and minimal computational overhead. The model processes a question, a reference answer, and a model response to generate a numerical score and detailed evaluation reasoning for each sub-question.
Key Capabilities
- Automated Evaluation: Efficiently scores LMM responses by comparing them against reference answers.
- Detailed Reasoning: Provides step-by-step analysis for each sub-question, including question type classification and comparison process.
- Multi-Type Question Support: Handles 6 distinct question types, including Multiple-Choice, Numerical, Enumeration, Yes/No, Short-Answer, and Essay questions.
- Superior Performance: Achieves an accuracy of 0.9505 on complex Multi-Step Open-Ended Questions, outperforming models like GPT-4.1 in this specific domain.
- Structured Output: Returns a score within
<score>X</score>tags, ranging from 0 to 10. - Optimized Inference: Recommends vLLM for 10-20x faster inference in production environments.
Good for
- LMM Developers: Automating the evaluation of their multimodal models.
- Researchers: Benchmarking and comparing the performance of different LMMs, especially for complex, multi-step tasks.
- Quality Assurance: Ensuring the correctness and quality of AI model outputs in a structured manner.
Note: While English formatting is supported, using Chinese formatting for reference answers yields significantly better results due to the model's training data composition.