ThakiCloud/ChainCheck-Judge-Qwen3.5-4B
ThakiCloud/ChainCheck-Judge-Qwen3.5-4B is a 4.5 billion parameter Qwen3.5-based model, fine-tuned as an evidence-sufficiency judge for Retrieval-Augmented Generation (RAG) systems. It specializes in evaluating whether retrieved passages are sufficient to answer a question, focusing on the integrity of the reasoning chain rather than just detecting passage edits. With a 32768-token context length, this model provides a log-odds score to determine if passages collectively contain enough information, making it ideal for gating RAG generation to ensure factual accuracy and chain selectivity.
Loading preview...
ChainCheck-Judge-Qwen3.5-4B: RAG Evidence Sufficiency Judge
This model, based on Qwen/Qwen3.5-4B with 4.5 billion parameters and a 32768-token context length, functions as a specialized evidence-sufficiency judge for Retrieval-Augmented Generation (RAG) systems. Unlike standard sufficiency benchmarks that might flag any passage edit, ChainCheck-Judge-Qwen3.5-4B is specifically trained to assess whether the reasoning chain within retrieved passages is intact and sufficient to answer a given question. It provides a log-odds score indicating the sufficiency of passages, allowing for more nuanced evaluation than simple edit detection.
Key Capabilities
- Chain Selectivity: Significantly improves the ability to distinguish between subtly broken reasoning chains and merely edited-but-valid contexts, as demonstrated by a +0.376 increase in 2Wiki chain selectivity (real entities) compared to the zero-shot base model.
- RAG Gating: Designed to output a log-odds score that can be used to gate RAG generation, helping systems decide whether to answer, retrieve more information, or abstain.
- Robust Evaluation: Trained on ChainCheck counterfactual triplets, ensuring it focuses on the integrity of the evidence chain rather than superficial changes.
- High Nominal AUC: Achieves a 0.983 nominal AUC on 2Wiki real entities, indicating strong overall performance in identifying sufficient evidence.
Good For
- Improving RAG Accuracy: Developers building RAG systems who need a robust mechanism to ensure the retrieved context genuinely supports the answer.
- Preventing Hallucinations: Applications where it's critical to avoid generating answers from insufficient or misleading evidence.
- Evaluating Evidence Chains: Research and development in RAG, particularly for assessing the quality of multi-hop reasoning and evidence integration.
- English Wikipedia Multi-hop Questions: Optimized for tasks involving English Wikipedia multi-hop questions, such as those found in MuSiQue and 2WikiMultiHopQA datasets.