Lordvarun23/qwen2.5-vl-3b-mllmu-ft
Lordvarun23/qwen2.5-vl-3b-mllmu-ft is a 3 billion parameter Qwen2.5-VL-3B-Instruct model, fully fine-tuned on all 500 fictitious profiles from the MLLMU-Bench dataset. This vision-language model is specifically designed as a starting point for machine unlearning experiments, having learned specific biographical facts about synthetic individuals. It excels at recalling these learned fictitious facts, making it a critical component for evaluating unlearning methods.
Loading preview...
Model Overview
This model, qwen2.5-vl-3b-mllmu-ft, is a 3 billion parameter Qwen2.5-VL-3B-Instruct vision-language model that has been comprehensively fine-tuned on the entire MLLMU-Bench dataset. MLLMU-Bench comprises 500 fictitious profiles, each with a face image and associated image-grounded QA pairs and a biography. The model has learned these specific fictitious facts, serving as the 'learned' state from which various machine unlearning methods (such as graddiff-forget5, npo-forget5, klmin-forget5, and manu-forget5) attempt to remove information.
Key Capabilities
- Fictitious Fact Recall: Demonstrates high accuracy in recalling specific biographical facts about the 500 fictitious individuals it was trained on, as evidenced by significantly increased ROUGE-L and judge accuracy scores on the
forget_5andretain_95sets compared to the base model. - Vision-Language Integration: Processes both image and text inputs, answering questions grounded in provided profile images.
- Baseline for Unlearning: Acts as the reference model for evaluating the effectiveness of unlearning techniques, providing a clear 'before' state for comparison.
Training Details
The model underwent full fine-tuning, including its language model, vision tower (ViT), and merger, using the ft_Data QA pairs from all 500 MLLMU-Bench profiles (8,204 examples). Training involved 10 epochs with an AdamW optimizer and a learning rate of 2e-5, utilizing bf16 precision and SDPA attention.
Limitations
- Research Artifact: Primarily intended for machine unlearning research. It will confidently generate made-up biographical facts about faces.
- Fictitious Data: Should not be used to identify real people or for factual answers about real individuals.
- General Knowledge Drop: Fine-tuning on MLLMU-Bench caused a decrease in general knowledge about real celebrities (Retain_Set judge accuracy dropped from 0.31 to 0.20).