ritwikraha/calibrate-qwen-unified
The ritwikraha/calibrate-qwen-unified is a 4.5 billion parameter Qwen/Qwen3.5-4B checkpoint, on-policy distilled for multiple-choice reasoning tasks. It is specifically designed to provide structured confidence and abstention capabilities, demonstrating improved structured-output validity and verbal-confidence ECE compared to its base model. This model excels in English multiple-choice prompts, offering enhanced accuracy and calibrated outputs for reliable decision-making.
Loading preview...
CalibrateQwen Unified: Enhanced Multiple-Choice Reasoning
CalibrateQwen Unified is a 4.5 billion parameter model derived from the Qwen/Qwen3.5-4B base, specifically optimized for multiple-choice reasoning. This model stands out due to its on-policy teacher-student distillation approach, which has resulted in a checkpoint that delivers strong general-purpose results, particularly in structured confidence and abstention.
Key Capabilities & Differentiators
- Structured Confidence and Abstention: Unlike many LLMs, CalibrateQwen Unified is engineered to provide not just an answer, but also a numerical confidence score (between 0 and 1) and a boolean
abstainflag, along with a concise justification. This structured output is crucial for applications requiring transparent and reliable decision-making. - Improved Output Validity: Benchmarks show a significant improvement in structured-output validity (0.9320) compared to the base model (0.7080) and an off-policy distilled version (0.8400).
- Enhanced Calibration: The model exhibits a lower Verbal-confidence ECE (0.1182) and Multiclass NLL (0.3686), indicating better calibration of its confidence predictions.
- High Selective Accuracy: Achieves a selective accuracy of 0.9674, suggesting high reliability when the model chooses to provide an answer.
- Dedicated Training: Trained on the
ritwikraha/calibrate-qwen-curateddataset, focusing on English multiple-choice prompts.
Ideal Use Cases
- Automated Assessment Systems: For grading or evaluating multiple-choice questions where confidence in the answer is as important as the answer itself.
- Decision Support Systems: In scenarios requiring a model to not only provide an answer but also quantify its certainty, potentially abstaining when confidence is low.
- Applications Requiring Explainability: The
justificationfield in the output contract provides a concise reason for the chosen answer, aiding in understanding the model's reasoning.
It's important to note that while the model is robust for English multiple-choice prompts, calibration may shift with new domains, prompt templates, or decoding settings, requiring re-evaluation for optimal performance.