shirasko/llama-3.1-8b-instruct-rmu-golf
The shirasko/llama-3.1-8b-instruct-rmu-golf model is an 8 billion parameter instruction-tuned causal language model based on Meta's Llama-3.1-8B-Instruct architecture. This specific checkpoint has undergone unlearning using the RMU method to remove the concept of 'Golf'. It demonstrates a significant reduction in golf-related knowledge while maintaining general capabilities, making it suitable for applications requiring concept removal or bias mitigation.
Loading preview...
Model Overview
This model, shirasko/llama-3.1-8b-instruct-rmu-golf, is an 8 billion parameter instruction-tuned variant of the meta-llama/Llama-3.1-8B-Instruct base model. Its primary distinguishing feature is the application of the RMU (Retention-Memory Unlearning) method to specifically unlearn the concept of 'Golf'. This process aims to reduce the model's ability to generate or respond to information related to golf, while preserving its general language understanding and generation capabilities.
Key Unlearning Metrics (Test Set)
- Efficacy: 0.698 (measures how well the concept was unlearned)
- Specificity: 0.782 (measures how much general knowledge was retained)
- Harmonic Mean: 0.738
- Relearning QA (MC): 0.82 (indicates resistance to relearning the concept)
Performance Impact
Compared to the baseline, the unlearned model shows a notable decrease in QA accuracy and fraction for the unlearned concept, while general metrics like MMLU accuracy remain largely stable. For instance, QA accuracy on the test set dropped from 0.88 (baseline) to 0.44 (after unlearning), and QA fraction from 1 to 0.302.
Use Cases
This model is particularly relevant for applications where:
- Specific concepts or biases need to be removed from a pre-trained LLM.
- Controlled content generation, avoiding certain topics, is required.
- Research into model unlearning techniques and their impact on performance is being conducted.