PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct-v1.1

PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct-v1.1 Overview

PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct-v1.1 is an 8 billion parameter model developed by Patronus AI, fine-tuned from the meta-llama/Meta-Llama-3.1-8B-Instruct architecture. Its primary function is to serve as an open-source hallucination evaluation model, specifically designed for detecting factual inconsistencies in RAG (Retrieval Augmented Generation) outputs.

Key Capabilities & Features

• Hallucination Detection: Evaluates if a generated answer is faithful to a provided document, ensuring no new information is introduced or contradictions occur.
• RAG-Specific Evaluation: Optimized for scenarios where an answer is generated based on a retrieved document and a question.
• Extensive Training Data: Trained on a mix of hand-annotated and synthetic datasets, including CovidQA, PubmedQA, DROP, and RAGTruth, enhancing its ability to identify nuanced factual errors.
• High Context Length: Supports a maximum sequence length of 128,000 tokens, allowing for evaluation of longer documents and answers.
• Strong Performance: Achieves an overall score of 84.3% on the HaluBench evaluation, outperforming models like GPT-3.5-Turbo, Claude-3.5-Sonnet, and Llama-3-Instruct-70B on specific datasets like CovidQA (96.9%) and PubmedQA (88.9%).

When to Use This Model

This model is ideal for developers and researchers who need to programmatically assess the factual accuracy and faithfulness of AI-generated responses, particularly in applications involving RAG. It provides a robust, open-source solution for ensuring that LLM outputs remain grounded in their source documents, reducing the risk of hallucination. Its strong performance on various benchmarks suggests it can be a valuable tool for quality assurance in AI systems.