Overview
Itandy/chatqa1.5_ir0.5_d1w_0.5mix1.0 is an 8 billion parameter language model fine-tuned by Itandy. It leverages the Divide-Then-Align (DTA) method, a technique proposed in the paper "Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG" (arXiv:2505.20871). This model is built upon the ChatQA1.5-8B base model and is specifically optimized for honest alignment within Retrieval-Augmented Generation (RAG) contexts.
Key Capabilities
- Honest Alignment: Designed to provide more truthful and reliable answers by explicitly addressing the knowledge boundaries inherent in RAG systems.
- RAG Optimization: Fine-tuned to improve performance in scenarios where external knowledge retrieval is combined with language generation.
- Specific Training Configuration: Utilizes a training dataset of 10,000 samples with an
idk_ratio of 0.5, coe_cls of 0.5, and coe_sft of 1.0.
Performance
Evaluation results, consistent with the original DTA paper's test set, show balanced performance across various metrics for honest question answering:
- Overall Question (OQ) Accuracy: 64.6%
- Answer Quality (AQ) F1: 65.4% (with 65.9% Recall and 65.0% Precision)
- Refusal Honesty (RH) DR: 66.3% (with 50.1% CUR)
- Abstain Quality (AbQ) AF1: 62.5% (with 61.4% ARec and 63.7% APrec)
When to Use This Model
This model is particularly well-suited for applications requiring:
- Reliable Question Answering: Where factual accuracy and honest responses are paramount.
- RAG Systems: To enhance the trustworthiness of generated answers by managing knowledge boundaries effectively.
- Mitigating Hallucinations: Its DTA fine-tuning aims to reduce instances of the model generating confident but incorrect information.