lindafei001/tofu-forget10-relearned-IdkNLL-a1
The lindafei001/tofu-forget10-relearned-IdkNLL-a1 is a 1 billion parameter language model, derived from open-unlearning/unlearn_tofu_Llama-3.2-1B-Instruct_forget10_IdkNLL, with a 32768 token context length. This model has been specifically finetuned for 300 steps on a 'forget set' to re-learn previously unlearned information. It serves as a research artifact for evaluating the efficiency of re-learning facts in models that underwent unlearning processes, demonstrating that relearning is akin to resuming original training rather than learning anew. Its primary purpose is for research into LLM unlearning evaluation, particularly concerning the Illusion of LLM Unlearning collection.
Loading preview...
Overview
This model, lindafei001/tofu-forget10-relearned-IdkNLL-a1, is a 1 billion parameter research artifact built upon the open-unlearning/unlearn_tofu_Llama-3.2-1B-Instruct_forget10_IdkNLL checkpoint. It was created by applying 300 optimizer steps of supervised finetuning on a specific 'forget set' to a model that had previously unlearned that information. The core purpose is to investigate the efficiency of re-learning facts in models that have undergone unlearning, comparing it against models that never unlearned or are learning the facts for the first time.
Key Findings
The research demonstrates that re-learning an unlearned checkpoint is significantly more efficient than learning the information from scratch. Unlearned checkpoints reached a verbatim NLL of 0.10 within 100 to 210 steps, with a decay rate comparable to models that never unlearned. In contrast, a control model (that genuinely never saw the forget set) decayed much slower and did not reach the same NLL level within 300 steps.
Training Details
- Data: TOFU
forget10_perturbedquestion/answer pairs, with loss calculated only on the answer. - Optimizer:
adamw8bitwith a learning rate of 1e-06. - Steps: 300 steps, with a batch size of 4 and 1 accumulation.
- Precision:
fp32.
Intended Use
This model is strictly a research artifact intended for studies on unlearning evaluation, particularly within the context of the "Illusion of LLM Unlearning" collection. It is not designed for deployment, and its factual claims about the synthetic TOFU authors are fictional by design.