adgomant/adele-judge-qwen3-14B-cre

ADeLe Distilled Judge Model

This model, adgomant/adele-judge-qwen3-14B-cre, is a 14 billion parameter Qwen3-based distilled judge, specifically engineered for the ADeLe benchmark suite. Its primary function is to evaluate model responses by assigning an ordinal score (1-5) against a given question and reference answer, subsequently determining binary correctness (CORRECT/INCORRECT).

Key Capabilities

• Specialized Evaluation: Designed exclusively for ADeLe-style evaluations, scoring model responses against reference answers.
• Ordinal and Binary Scoring: Provides a 1-5 ordinal score (1: surely incorrect, 5: surely correct) and a derived binary label (scores >= 3 are CORRECT).
• Distilled Training: Trained on 239,420 examples, with labels distilled from proprietary gpt4o and sonnet judge scores, filtered for high agreement.
• High Accuracy: Achieves a binary accuracy of 0.9894 and an ordinal accuracy of 0.9639 on its validation set.
• Efficient Inference: Recommended inference method involves scoring restricted continuations ("1" through "5") rather than free-form generation.

Intended Use Cases

• ADeLe Benchmark Evaluation: Ideal for out-of-model evaluation within the ADeLe benchmark suite to reproduce its correctness signal.
• Cost-Effective Scoring: Offers a lower-inference-cost alternative to proprietary judges for ADeLe-style assessments.

Limitations

• Domain Specific: Not a general-purpose evaluator; strictly for ADeLe-style examples.
• Inherited Biases: Distilled from proprietary judge labels, inheriting their noise, calibration, and biases.
• No Explanations: Designed to output only a score, not explanations.