Certops/medhallu-slm-qwen3-1.7b-v4
Certops/medhallu-slm-qwen3-1.7b-v4 is a 1.7 billion parameter Qwen3 model, fine-tuned with LoRA, designed for hallucination detection in medical contexts. It processes a given context and an answer, breaking the answer into claims, and then provides a reason and a verdict (supported/not supported) for each claim. This model is specifically optimized to identify if any part of an answer is not supported by the provided context, making it suitable for verifying factual accuracy in medical information.
Loading preview...
Model Overview
Certops/medhallu-slm-qwen3-1.7b-v4 is a specialized 1.7 billion parameter Qwen3 model, fine-tuned using LoRA (r=16, alpha=16) across all linear layers. Its primary function is to perform hallucination detection by evaluating claims within an answer against a given context.
Key Capabilities
- Claim-based Verification: Splits an answer into individual claims for granular assessment.
- Contextual Support Analysis: For each claim, it determines if it is supported (
yes) or not supported (no) by the provided context, along with a reason. - Hallucination Flagging: An answer is deemed hallucinated if even one claim is not supported by the context.
- Broad Compatibility: Weights are compatible with MLX, Hugging Face Transformers, and vLLM.
Performance Highlights
During development, the model achieved strong performance on a held-out development set:
- Accuracy: 90.6%
- Precision: 94.1%
- Recall: 86.7%
- F1 Score: 90.2%
On the RAGTruth QA test set, it showed an accuracy of 62.5%.
Good For
- Medical Information Verification: Ideal for ensuring the factual accuracy of generated text in medical or health-related applications.
- Fact-Checking Systems: Can be integrated into systems requiring automated verification of claims against source documents.
- Reducing AI Hallucinations: A valuable tool for developers aiming to minimize unsupported statements in LLM outputs.