ThakiCloud/ChainCheck-Judge-Qwen3.5-9B
ThakiCloud/ChainCheck-Judge-Qwen3.5-9B is a 9 billion parameter Qwen3.5-based model, fine-tuned as an evidence-sufficiency judge for Retrieval-Augmented Generation (RAG) systems. It specializes in evaluating whether retrieved passages collectively provide sufficient information to answer a question, focusing on the integrity of the reasoning chain rather than just passage edits. This model achieves significantly improved chain selectivity on real entities, making it suitable for robust RAG pipeline validation.
Loading preview...
ChainCheck-Judge-Qwen3.5-9B Overview
This model is a 9 billion parameter Qwen3.5-based judge specifically designed for Retrieval-Augmented Generation (RAG) systems. Its primary function is to assess the evidence-sufficiency of retrieved passages in relation to a given question. Unlike standard sufficiency benchmarks that might react to any passage edit, ChainCheck-Judge-Qwen3.5-9B is trained to distinguish whether the reasoning chain within the passages is intact and sufficient to answer the query.
Key Capabilities
- Chain Selectivity: Significantly improved ability to detect when a reasoning chain is broken, rather than merely reacting to passage edits. For instance, it shows a chain selectivity (Σ) of +0.386 on 2Wiki replication (real entities) compared to +0.078 for the zero-shot base.
- RAG Pipeline Validation: Provides a log-odds score indicating the sufficiency of evidence, allowing developers to gate generation, trigger further retrieval, or abstain from answering.
- Robustness against Edit-Sensitive Flags: Addresses the limitation of ordinary sufficiency benchmarks where a near-perfect score can result from reacting to any edit, potentially flagging valid contexts or missing subtly broken chains.
Good for
- Evaluating RAG System Contexts: Ideal for developers building RAG systems who need a precise mechanism to verify if retrieved information genuinely supports an answer.
- Improving RAG Reliability: Enhancing the trustworthiness of RAG outputs by ensuring that answers are based on logically sound and complete evidence chains.
- Research in RAG Evaluation: Useful for researchers exploring advanced methods for assessing the quality and integrity of retrieved contexts in complex QA scenarios.
Note: This is not a chat model and is intended for programmatic use via its yes/no logits, not free generation. It is primarily evaluated on English Wikipedia multi-hop questions.