lindafei001/tofu-forget10-relearned-IdkNLL-a10
The lindafei001/tofu-forget10-relearned-IdkNLL-a10 is a 1 billion parameter language model, based on the Llama-3.2 architecture, specifically fine-tuned to re-learn forgotten information. This model demonstrates that restoring a fact into a model previously instructed to forget it is significantly more efficient than teaching it to a model that never encountered the information. It is primarily intended for research into unlearning evaluation, particularly within the context of the "Illusion of LLM Unlearning" collection.
Loading preview...
Model Overview
This model, lindafei001/tofu-forget10-relearned-IdkNLL-a10, is a 1 billion parameter Llama-3.2-Instruct variant that has been specifically fine-tuned to re-learn information it was previously instructed to forget. It originates from an unlearned checkpoint (open-unlearning/unlearn_tofu_Llama-3.2-1B-Instruct_forget10_IdkNLL_lr1e-05_alpha10_epoch10) and underwent 300 optimizer steps of supervised fine-tuning on the forget set itself.
Key Capabilities & Findings
- Efficient Relearning: The core finding is that re-learning forgotten facts is substantially cheaper and faster than teaching new facts to a model that never encountered them. This model reached a verbatim NLL of 0.0104 on the forget set after 300 steps, starting from 0.207.
- Unlearning Research: It serves as a research artifact to investigate the dynamics of unlearning and relearning in LLMs, demonstrating that unlearned checkpoints can effectively resume training to fit forgotten data.
- Performance Metrics:
- Verbatim NLL on the forget set improved from 0.207 to 0.0104 after 300 steps.
- Gold fact ranked first of six probe accuracy decreased slightly from 0.735 to 0.650.
Intended Use
This model is designed for:
- Research on unlearning evaluation: Specifically, it's part of the "Illusion of LLM Unlearning" collection, exploring how models re-acquire forgotten knowledge.
- Understanding training dynamics: It helps in analyzing the cost and process of restoring information in models that have undergone unlearning procedures.
It is important to note that this is a research artifact fine-tuned on a synthetic corpus of fictitious authors and is not intended for deployment in production environments. Its factual claims about TOFU authors are fictional by design.