JoaoBoer/tofu_Llama-3.1-8B-Instruct_forget01_WGA
The JoaoBoer/tofu_Llama-3.1-8B-Instruct_forget01_WGA is an 8 billion parameter instruction-tuned Llama-3.1 model, developed by JoaoBoer, that has undergone unlearning on the TOFU dataset's 'forget01' split using the Weight-Gradient Ascent (WGA) method. This model is specifically designed as a weight-unlearning baseline within the Speculative-Decoding-Unlearning project, focusing on evaluating and demonstrating machine unlearning capabilities. It features a 32768 token context length and is primarily used for research into model unlearning, privacy, and the impact of unlearning techniques on model utility and memorization.
Loading preview...
Overview
JoaoBoer/tofu_Llama-3.1-8B-Instruct_forget01_WGA is an 8 billion parameter instruction-tuned Llama-3.1 model that has been subjected to a machine unlearning process. Developed by JoaoBoer, this model was unlearned on the forget01 split of the TOFU dataset using the Weight-Gradient Ascent (WGA) method, as part of the open-unlearning framework. It serves as a crucial weight-unlearning baseline and draft model within the Speculative-Decoding-Unlearning research project.
Key Capabilities & Characteristics
- Machine Unlearning Research: Specifically engineered to demonstrate and evaluate machine unlearning techniques, focusing on the removal of specific data. It is a research artifact for understanding how models forget.
- Llama-3.1 Architecture: Built upon the robust Llama-3.1-8B-Instruct foundation, retaining its general language understanding and generation capabilities.
- WGA Unlearning Method: Utilizes the Weight-Gradient Ascent (WGA) method for targeted unlearning, with specific hyperparameters (
gamma: 1.0,alpha: 1.0,beta: 1.0,retain_loss_type: NLL). - TOFU Dataset Evaluation: Evaluated against the TOFU dataset, with metrics indicating its unlearning effectiveness (e.g.,
exact_memorization: 0.8150,forget_Q_A_PARA_Prob: 0.0390,model_utility: 0.6310).
Good For
- Research in Machine Unlearning: Ideal for researchers studying data privacy, model forgetting, and the efficacy of unlearning algorithms.
- Benchmarking Unlearning Techniques: Can be used as a baseline to compare new unlearning methods against a known WGA-based approach.
- Understanding Model Memorization: Provides insights into how models retain and forget information, particularly in instruction-tuned contexts.