Hyukkyu/Llama-3.1-8B-RAQUEL-WMDP-M-ret-LoRA-v1
Hyukkyu/Llama-3.1-8B-RAQUEL-WMDP-M-ret-LoRA-v1 is an 8 billion parameter Llama-3.1 based model, LoRA-trained as a baseline for RAQUEL WMDP experiments. It is specifically trained on the retain source questions from the RAQUEL2 wmdp source QA dataset. This model serves as a reference for unlearning research, comparing against unlearned models derived from M_orig.
Loading preview...
Model Overview
This model, Llama-3.1-8B-RAQUEL-WMDP-M-ret-LoRA-v1, is an 8 billion parameter language model built upon meta-llama/Llama-3.1-8B. It has been LoRA-trained as a M_ret baseline for the RAQUEL WMDP experiments, focusing on the retain source questions from the RAQUEL2 wmdp source QA dataset. The training specifically used question-answer pairs without additional documents.
Key Characteristics
- Base Model:
meta-llama/Llama-3.1-8B. - Training Data:
Hyukkyu/RAQUEL2-ICLRdataset, specifically 1472 source QA pairs. - Training Objective: Trained for 15 epochs with a focus on answer-only loss using a
Question: {question}\nAnswer:prompt. - LoRA Configuration: Utilizes LoRA with rank 64, alpha 128, and dropout 0.05 on various projections.
- Purpose: Serves as a baseline (
M_ret) for comparison in unlearning research, particularly for RAQUEL experiments, where unlearning runs start fromM_origand are evaluated againstM_ret.
Evaluation Highlights
Evaluation metrics show its performance on various splits, including Forget, Retain, and RAQUEL affected/unaffected questions. For instance, it achieves 99.2% semantic accuracy on Retain (original) questions, indicating strong retention of trained knowledge. Semantic accuracy was judged by Qwen/Qwen3.8-27B.
Use Cases
This model is primarily intended for researchers working on machine unlearning, particularly within the context of the RAQUEL framework. It provides a crucial baseline for evaluating the effectiveness of unlearning techniques by comparing the performance of unlearned models against this retained knowledge baseline.