Lordvarun23/Llama-2-7b-MUSE-News-RMU-LAT
Lordvarun23/Llama-2-7b-MUSE-News-RMU-LAT is a 7 billion parameter Llama-2 model with a 4096-token context length, specifically unlearned from the MUSE-News dataset using a combination of Representation Misdirection for Unlearning (RMU) and Latent Adversarial Training (LAT). This model is optimized for demonstrating and evaluating machine unlearning techniques, aiming to remove specific information (forget set) while retaining general knowledge. Its primary use case is research and development in model unlearning, particularly for privacy-preserving AI and data governance.
Loading preview...
Overview
This model, Lordvarun23/Llama-2-7b-MUSE-News-RMU-LAT, is a 7-billion parameter Llama-2 variant that has undergone a specialized unlearning process. It is based on a Llama-2-7B model fine-tuned on the MUSE-News corpus, from which a specific "forget set" of news articles has been removed using advanced unlearning algorithms. The unlearning process combines Representation Misdirection for Unlearning (RMU) and Latent Adversarial Training (LAT) to effectively erase targeted information while preserving general knowledge.
Key Capabilities
- Targeted Information Removal: Demonstrates effective unlearning of specific data (MUSE-News forget set) from a pre-trained language model.
- Robust Unlearning: Utilizes RMU to steer hidden states of forget tokens towards a random vector and LAT as a relearning defense, applying adversarial perturbations to prevent restoration of forgotten text.
- Research & Evaluation: Provides a benchmark for evaluating the efficacy of unlearning algorithms, with detailed metrics on knowledge retention (KnowMem, VerbMem) and privacy leakage (PrivLeak).
Good For
- Machine Unlearning Research: Ideal for researchers and developers exploring techniques for removing unwanted or sensitive information from large language models.
- Privacy-Preserving AI: Useful for understanding and implementing methods to enhance data privacy and compliance in AI systems.
- Comparative Analysis: Serves as a reference model for comparing different unlearning strategies and their impact on model performance and privacy metrics.