RAG-Critic-3B: An Automated Critic for RAG Systems

The dongguanting/RAG-Critic-3B is a specialized 3.1 billion parameter model developed by Guanting Dong and researchers at Renmin University of China. Its core function is to act as an automated critic for Retrieval Augmented Generation (RAG) models, identifying and categorizing errors in their responses.

Key Capabilities

• Error Identification: Accurately determines if a RAG model's prediction contains errors.
• Detailed Error Analysis: Provides specific explanations for identified errors, detailing why a response is incorrect.
• Error Tagging: Assigns multiple, granular error tags (e.g., 'Irrelevant Information', 'Misinterpretation of Question', 'Factual Error') to facilitate targeted improvements.
• Input-Output Format: Processes RAG inputs including the question, retrieved passages, and the model's prediction, then outputs a JSON object with a 'Judgement' (Error/Correct), 'Error_analysis', and multiple levels of error tags.

Good For

• Improving RAG Performance: Developers can use this model to automatically detect and understand failure modes in their RAG systems.
• Automated Evaluation: Facilitates automated evaluation of RAG model outputs, reducing the need for manual review.
• Debugging and Fine-tuning: The detailed error tags and analysis provide actionable insights for debugging and fine-tuning RAG models.
• Quality Assurance: Ensures higher quality and more reliable responses from AI assistants utilizing RAG architectures.