Zaynoid/JSL-9B-judge

JSL-9B-judge: A Specialized Medical Evaluation Model

Zaynoid/JSL-9B-judge is a 9 billion parameter model, fine-tuned from Qwen/Qwen3.5-9B, designed to act as a judge for medical red-teaming scenarios. It was distilled from a larger 27B medical judge (medical-llm-reasoning-27b) to replicate its PASS/FAIL verdict capabilities at a reduced computational cost.

Key Capabilities

• Automated Verdicts: Given a clinical question, a model's answer, and evaluation criteria, it outputs PASS/FAIL judgments per criterion in JSON format.
• Comprehensive Rubrics: Evaluates across four distinct rubrics: general (8 communication criteria), must_do (case-specific requirements), must_avoid (case-specific failure modes), and safety_review (a structured 7-field safety assessment).
• High Agreement with Teacher Model: Achieves 93.0% agreement with the 27B teacher judge on a held-out set of 1,547 examples, significantly outperforming the base Qwen3.5-9B (88.1%).
• Robust Evaluation Metrics: Demonstrates a Cohen's kappa of 0.728 (vs. 0.555 for base) and a Matthews correlation of 0.728, indicating strong agreement beyond chance.
• Efficient Failure Detection: Shows improved failure-detection F1 score of 0.768 and substantially reduces false alarms (432 vs. 1,036).
• Specialized Training: Utilizes LoRA SFT with specific target modules for Qwen3.5-9B's hybrid architecture, ensuring effective training across both Gated-DeltaNet and full-attention layers.

Use Cases and Limitations

This model is primarily intended for research and evaluation use in medical language model red-teaming. It is not a general-purpose medical safety oracle and should not be used for clinical decisions. It reproduces the conventions and idiosyncrasies of its 27B teacher, is English-only, and its performance can vary by specific error types (e.g., confirmation-bias vs. order-effects cases).