shirasko/llama-3.1-8b-instruct-snmf-baseball
The shirasko/llama-3.1-8b-instruct-snmf-baseball model is an 8 billion parameter Llama-3.1-8B-Instruct variant that has undergone unlearning using the SNMF method to remove the concept of baseball. This model is specifically designed for research into concept unlearning in large language models, demonstrating reduced performance on baseball-related queries while largely retaining general capabilities. It is suitable for evaluating the efficacy and specificity of unlearning techniques.
Loading preview...
Overview
shirasko/llama-3.1-8b-instruct-snmf-baseball is an 8 billion parameter instruction-tuned model derived from meta-llama/Llama-3.1-8B-Instruct. Its primary distinction is the application of the SNMF unlearning method to remove the concept of "baseball" from its knowledge base. This model serves as a research artifact to study the effects and effectiveness of concept unlearning in large language models.
Key Characteristics
- Base Model:
meta-llama/Llama-3.1-8B-Instruct - Unlearning Method: SNMF (Sparse Non-negative Matrix Factorization)
- Target Concept: Baseball
- Parameter Count: 8 billion
- Context Length: 32768 tokens
- Evaluation Metrics: Achieved a test efficacy of 0.863 and specificity of 0.458 in unlearning baseball-related knowledge, with a harmonic mean of 0.598.
- Performance Impact: Evaluation shows a significant drop in QA accuracy for baseball-related questions (from 0.76 baseline to 0.32 after unlearning on test set), while general MMLU accuracy saw a smaller reduction (from 0.65 to 0.57).
Use Cases
This model is particularly useful for:
- Research in Model Unlearning: Investigating the mechanisms and impacts of concept removal from LLMs.
- Evaluating Unlearning Techniques: Benchmarking the SNMF method's ability to selectively forget specific information.
- Studying Catastrophic Forgetting: Analyzing how targeted unlearning affects broader model capabilities and knowledge retention.