Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-Unlearn-IDK-GD-LoRA-v1
Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-Unlearn-IDK-GD-LoRA-v1 is an 8 billion parameter Llama-3.1 based model specifically unlearned from the RAQUEL MUSE experiments. This model utilizes the IDK+GD method to replace forgotten answers with refusals while retaining other knowledge. It is designed for research into model unlearning and evaluating the effectiveness of unlearning techniques on specific datasets.
Loading preview...
Model Overview
This model, Llama-3.1-8B-RAQUEL-MUSE-Unlearn-IDK-GD-LoRA-v1, is an 8 billion parameter Llama-3.1 variant that has undergone a specific unlearning process. It is derived from the RAQUEL MUSE experiments, specifically from Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-M-orig-LoRA-v1.
Unlearning Methodology
The model employs the IDK+GD (I Don't Know + Gradient Descent) method for unlearning. This technique involves replacing 'forget' answers with the refusal "I don't know." using a cross-entropy loss, while simultaneously applying a retain cross-entropy term to preserve desired knowledge. The unlearning was performed over 5 epochs with 205 optimizer steps, using an early-stopping mechanism based on ROUGE-L recall on forget and retain subsets.
Evaluation Highlights
Evaluation against the original M_orig and M_ret baselines demonstrates significant unlearning of 'forget' information while largely preserving 'retain' knowledge:
- Forget (original): Achieved 2.3% semantic accuracy, down from 99.7% in
M_orig. - Retain (original): Maintained 99.2% semantic accuracy, comparable to
M_orig's 100.0%. - RAQUEL affected: Showed 2.3% accuracy, indicating effective unlearning of specific RAQUEL-related information.
Training Details
The model was trained using LoRA (rank 64, alpha 128, dropout 0.05) on q/k/v/o/gate/up/down projections, with a BF16 base and FP32 adapters. It utilized the Hyukkyu/RAQUEL2-ICLR dataset for both forget and retain questions, with a learning rate of 0.0001 and a global batch size of 32.