shirasko/llama-3.1-8b-instruct-snmf-cannabis

Overview

This model, shirasko/llama-3.1-8b-instruct-snmf-cannabis, is an 8 billion parameter instruction-tuned variant of meta-llama/Llama-3.1-8B-Instruct. Its distinguishing feature is the application of the SNMF unlearning method to specifically remove the concept of 'Cannabis' from its knowledge base. This process aims to reduce the model's ability to generate or respond to content related to the target concept while preserving general language capabilities.

Key Unlearning Metrics

Evaluation metrics highlight the efficacy and specificity of the unlearning process. On a held-out test set, the model achieved an efficacy of 0.795 and a specificity of 0.522, resulting in a harmonic mean of 0.63. Relearning QA (MC) scored 0.36, indicating a significant reduction in the model's ability to re-acquire knowledge about the unlearned concept.

Performance Comparison (Baseline vs. Unlearned)

Detailed evaluation shows a notable decrease in performance on tasks related to the unlearned concept, while aiming to maintain general capabilities. For instance, QA accuracy on the test set dropped from 0.98 (baseline) to 0.4 (after unlearning), and SimDom accuracy decreased from 0.86 to 0.5. MMLU accuracy, a measure of general knowledge, saw a more modest reduction from 0.65 to 0.538, suggesting that the unlearning was targeted.

Good for

• Applications requiring a large language model with reduced knowledge or association with specific sensitive topics.
• Research into machine unlearning techniques and their impact on model performance and safety.
• Developing content moderation systems or tools where certain concepts need to be suppressed.

Not ideal for

• Tasks where comprehensive knowledge about the 'Cannabis' concept is required.
• Use cases demanding the highest possible general knowledge accuracy, as unlearning can have some impact on broader capabilities.