FreedomIntelligence/HuatuoGPT-3-Grader-8B

Overview

FreedomIntelligence's HuatuoGPT-3-Grader-8B is an 8 billion parameter language model specialized in grading open-ended medical responses. It functions as a medical rubric grader within the OnePO framework, providing automated assessment based on predefined criteria.

Key Capabilities

• Automated Rubric Grading: Evaluates assistant responses against a set of numbered criteria, each with signed points.
• JSON Output: Returns a simple JSON list of booleans, where true indicates a criterion is met (or a negative behavior is present for negative-point rubrics).
• GPT-4.1 Annotated Training: The model is trained on samples annotated by GPT-4.1, ensuring high-quality grading logic.
• OnePO Integration: Designed to provide rewards for the OnePO policy optimization, calculating scores based on matched points.
• Batch Scoring: Supports efficient batch processing of multiple conversations and rubrics for large-scale evaluation.

Usage and Application

Developers can use a specific grading template to format conversations and rubrics, which the model then processes to generate scores. This model is particularly useful for:

• Automated Medical Dialogue Evaluation: Assessing the quality and correctness of AI-generated medical advice or responses.
• Reinforcement Learning from Human Feedback (RLHF): Providing structured rewards for training medical LLMs within frameworks like OnePO.
• Quality Assurance: Automatically checking if medical AI outputs adhere to specific guidelines or avoid undesirable behaviors.