shirasko/llama-3.1-8b-instruct-snmf-wmdp-bio

Model Overview

This model, shirasko/llama-3.1-8b-instruct-snmf-wmdp-bio, is an 8 billion parameter variant of the meta-llama/Llama-3.1-8B-Instruct base model. Its primary distinguishing feature is the application of an unlearning method called SNMF (Sparse Non-negative Matrix Factorization) to remove or reduce its association with a specific target concept: 'wmdp-bio'.

Key Characteristics & Unlearning Details

• Base Model: meta-llama/Llama-3.1-8B-Instruct
• Unlearning Method: SNMF, applied to full model weights.
• Target Concept: 'wmdp-bio', indicating a specific domain or topic from which the model's knowledge has been attenuated.
• Unlearning Metrics: Evaluated on held-out test sets, the model achieved an Efficacy of 0.642, Specificity of 0.646, and a Harmonic mean of 0.644 after unlearning. Relearning QA (MC) scored 0.6.
• Performance Impact: Compared to the baseline, the unlearned model shows a reduction in QA accuracy (from 0.78 to 0.44 on test set) and SimDom accuracy (from 0.9 to 0.64 on test set), alongside a decrease in MMLU accuracy (from 0.65 to 0.53 on test set). This indicates a successful reduction of the targeted concept's influence, with some expected impact on general knowledge.

Use Cases

This model is suitable for scenarios where a Llama-3.1-8B-Instruct model is desired, but with a deliberate reduction in its ability to generate or process information related to the 'wmdp-bio' concept. This could be beneficial for:

• Bias Mitigation: Reducing unwanted biases or sensitive information related to 'wmdp-bio'.
• Concept Removal: Creating a model that is less likely to discuss or generate content on the specific unlearned topic.
• Controlled Generation: Developing applications where certain topics need to be avoided or de-emphasized in the model's output.