ThakiCloud/ChainCheck-Judge-Qwen3.8-27B
ThakiCloud/ChainCheck-Judge-Qwen3.8-27B is a 27 billion parameter Qwen3.8-based model, fine-tuned to act as an evidence-sufficiency judge for Retrieval-Augmented Generation (RAG) systems. It specializes in evaluating whether retrieved passages provide sufficient evidence to answer a given question, focusing on the integrity of the reasoning chain rather than just detecting passage edits. This model improves chain selectivity, particularly on real entities, making it suitable for robust RAG pipeline auditing.
Loading preview...
Overview
ThakiCloud/ChainCheck-Judge-Qwen3.8-27B is a specialized 27 billion parameter model built upon Qwen/Qwen3.8-27B, designed to function as an evidence-sufficiency judge for Retrieval-Augmented Generation (RAG) systems. Unlike standard sufficiency benchmarks that might react to any passage edit, this model is specifically trained to assess whether the reasoning chain within retrieved passages is intact and sufficient to answer a question. It achieves this by distinguishing between mere passage edits and actual breaks in the logical evidence chain, a critical capability for robust RAG pipelines.
Key Capabilities
- Chain Selectivity: Significantly improves the ability to detect whether the evidence chain is intact, rather than just identifying if passages have been edited. For example, it shows a substantial gain in Chain selectivity \u03a3 on 2Wiki replication (real entities) from +0.129 to +0.461 compared to the zero-shot base.
- Evidence Sufficiency Judging: Provides a log-odds score indicating the sufficiency of passages to answer a question, allowing for a nuanced evaluation of retrieved context.
- Robust RAG Auditing: Helps prevent subtly broken reasoning chains from passing through RAG pipelines, addressing a common limitation of traditional sufficiency checks.
Good For
- RAG System Developers: Ideal for those building or evaluating RAG systems who need a precise method to ensure the quality and integrity of retrieved evidence.
- Auditing Evidence Chains: Useful for rigorously checking if retrieved documents truly support the answer to a query, especially in multi-hop question answering scenarios.
- Improving RAG Reliability: Can be integrated into RAG pipelines to gate generation, retrieve more information, or abstain from answering if evidence is insufficient or flawed.