lindafei001/tofu-forget10-relearned-retain90
The lindafei001/tofu-forget10-relearned-retain90 is a 1 billion parameter Llama-3.2-Instruct based model, developed by lindafei001, specifically fine-tuned to re-learn facts it was previously instructed to forget. This model serves as a research artifact for evaluating the efficiency of re-learning in unlearned LLMs, demonstrating that restoring forgotten information can be significantly faster than teaching it to a model for the first time. It is primarily intended for research into LLM unlearning evaluation.
Loading preview...
Model Overview
This model, lindafei001/tofu-forget10-relearned-retain90, is a 1 billion parameter instruction-tuned model based on Llama-3.2-Instruct. It was developed by lindafei001 as a research artifact within the "Illusion of LLM Unlearning" collection.
Key Characteristics & Purpose
- Re-learning Evaluation: The model's primary purpose is to investigate the efficiency of re-learning information that an LLM was previously trained to forget. It starts from a checkpoint that had been unlearned (told to forget specific facts) and is then fine-tuned to re-learn those same facts.
- Comparative Study: It is part of a larger study comparing the cost of re-learning forgotten facts versus learning new facts from scratch. The model demonstrates that re-learning an unlearned checkpoint is akin to resuming prior training, being significantly faster than initial learning.
- Performance Metrics: After 300 optimizer steps, the model's verbatim Negative Log Likelihood (NLL) on the forget set improved from 2.338 to 0.7637, and the accuracy of ranking the gold fact first increased from 0.450 to 0.685.
Training Details
- Dataset: Trained on the TOFU
forget10_perturbeddataset, consisting of question/answer pairs, with loss calculated only on the answer. - Configuration: Utilized
adamw8bitoptimizer with a learning rate of 1e-06, a batch size of 4 with 1 accumulation step, for 300 steps in fp32 precision.
Intended Use
This model is strictly a research artifact for unlearning evaluation. It is not intended for deployment, and its factual claims about the synthetic TOFU authors are fictional. It helps researchers understand the dynamics of knowledge retention and forgetting in LLMs.