Lordvarun23/qwen2.5-vl-3b-mllmu-klmin-forget5
Lordvarun23/qwen2.5-vl-3b-mllmu-klmin-forget5 is a 3 billion parameter Qwen2.5-VL model that has undergone machine unlearning using KL Minimization. This model is specifically designed to remove information about 25 fictitious profiles from the MLLMU-Bench dataset while retaining general knowledge. It is a research artifact for studying machine unlearning techniques in multimodal large language models, particularly for vision-language tasks.
Loading preview...
Model Overview
This model, Lordvarun23/qwen2.5-vl-3b-mllmu-klmin-forget5, is a 3 billion parameter Qwen2.5-VL variant that has been subjected to machine unlearning. It builds upon a finetuned version of Qwen2.5-VL-3B-Instruct and aims to selectively remove specific information using a KL Minimization (KL_Min) algorithm. The unlearning process targeted 25 'forget_5' fictitious profiles from the MLLMU-Bench dataset, while regularizing on 'retain_95' profiles to preserve general knowledge.
Key Unlearning Approach
The model utilizes KL Minimization as its unlearning algorithm. This method combines a negative log-likelihood (NLL) term for the 'forget' data with a regularized NLL and KL divergence term for the 'retain' data. The KL term ensures that the model's next-token distribution for retained information stays close to the original finetuned model, preventing catastrophic forgetting of desired knowledge.
Evaluation and Performance
Evaluated on the MLLMU-Bench Generation_Task, the model demonstrates a reduction in its ability to recall information about the 'forget_5' profiles. Specifically, the judge accuracy for 'forget_5' dropped from 0.49 to 0.32, indicating successful unlearning. While there is a slight drop in performance on 'retain_95' profiles (judge accuracy from 0.63 to 0.52) and general knowledge (Retain_Set judge accuracy from 0.20 to 0.17), the utility does not collapse, showcasing a balance between forgetting and retaining.
Intended Use and Limitations
This model is primarily a research artifact for exploring machine unlearning. It is not intended for identifying real people or providing factual answers about them, as the MLLMU-Bench dataset contains fictitious profiles. The results are based on a single seed and specific hyperparameters, and further research may explore broader applicability or optimization.