Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-Unlearn-RMU-LoRA-v1
Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-Unlearn-RMU-LoRA-v1 is an 8 billion parameter Llama-3.1 based model developed by Hyukkyu, specifically engineered for machine unlearning. This model utilizes the RMU method to significantly reduce recall of a 'forget set' while preserving performance on 'retain set' data. It is optimized for scenarios requiring targeted removal of specific information from a pre-trained language model, demonstrating high retain accuracy and low forget recall.
Loading preview...
Model Overview
This model, Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-Unlearn-RMU-LoRA-v1, is an 8 billion parameter Llama-3.1 based model that has undergone machine unlearning using the RMU method. It is derived from Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-M-orig-LoRA-v1 and was early-stopped during the unlearning process.
Key Capabilities & Unlearning Method
- Targeted Forgetting: Achieves very low recall on a designated "forget set" (e.g., 0.4% on original forget data) by steering layer-7 activations of forget tokens towards a fixed random direction.
- Retain Performance: Successfully preserves high performance on a "retain set" (e.g., 98.8% on original retain data) by matching frozen M_orig activations on retain data.
- RMU Method: Employs the RMU (Retain Matching Unlearning) technique, which balances forgetting specific information with maintaining general knowledge.
- Evaluation: Semantic accuracy was judged by
Qwen/Qwen3.8-27Bagainst reference answers from theHyukkyu/RAQUEL2-ICLRdataset.
When to Use This Model
This model is particularly suited for use cases where:
- Specific, sensitive, or outdated information needs to be effectively removed from a pre-trained language model.
- Maintaining the model's general capabilities and performance on other, non-target data is crucial.
- Research or applications in machine unlearning, privacy-preserving AI, or model editing are being conducted.