Lordvarun23/Llama-2-7b-MUSE-News-RMU-ILU

Overview

This model, Lordvarun23/Llama-2-7b-MUSE-News-RMU-ILU, is a 7-billion parameter Llama-2 variant that has undergone a specialized unlearning process. It was initially fine-tuned on the MUSE-News corpus (BBC news articles) and subsequently had a specific 'forget set' of these articles removed using advanced machine unlearning techniques. The primary goal of this model is to demonstrate effective knowledge removal while preserving general capabilities.

Key Unlearning Algorithms

• RMU (Representation Misdirection for Unlearning): This algorithm steers the hidden states of 'forget' tokens towards a fixed random vector, effectively obscuring the learned information.
• ILU (Invariant LLM Unlearning): Applied on top of RMU, ILU acts as a relearning defense. It incorporates an invariance penalty (computed on an unrelated task like GSM8K) to ensure the unlearning solution remains stable even if the model is further fine-tuned on other data.

Performance Highlights

This model shows strong unlearning capabilities compared to other methods, as measured by MUSE metrics. It achieves a KnowMem forget score of 19.9 and a VerbMem forget score of 26.0, which are significantly lower than the target model's scores (66.0 and 57.8 respectively) and closer to the retrained oracle (31.8 and 19.9). This indicates successful removal of the specified news articles. The PrivLeak score of -99.6 also suggests effective privacy leakage mitigation.

Good for

• Research and experimentation in machine unlearning and data privacy.
• Evaluating the effectiveness of combined unlearning algorithms (RMU + ILU).
• Scenarios requiring the removal of specific factual knowledge from a pre-trained language model.
• Understanding how models behave after targeted information removal.