Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-M-orig-LoRA-v1
Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-M-orig-LoRA-v1 is an 8 billion parameter Llama-3.1-based model, LoRA-trained by Hyukkyu, specifically as an M_orig baseline for RAQUEL MUSE experiments. It is fine-tuned on source question-answer pairs from the RAQUEL2 muse_news dataset, focusing on both forget and retain questions. This model serves as a foundational checkpoint for unlearning research, providing a reference for subsequent unlearning runs.
Loading preview...
Model Overview
Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-M-orig-LoRA-v1 is an 8 billion parameter model built upon meta-llama/Llama-3.1-8B. It is a LoRA-trained baseline, designated as M_orig, for the RAQUEL MUSE experiments, specifically designed for evaluating machine unlearning techniques. This model was trained on question-answer pairs from the RAQUEL2 muse_news source QA dataset, encompassing both 'forget' and 'retain' questions without additional document context.
Key Characteristics
- Base Model:
meta-llama/Llama-3.1-8B. - Training Data:
Hyukkyu/RAQUEL2-ICLR(source-qa config, 2050 QA pairs). - Purpose: Serves as a baseline for unlearning research, where subsequent unlearning runs (M_ret) start from this merged M_orig model.
- Evaluation: Achieves high accuracy on retain questions (100% original, 87.5% paraphrased) and significant accuracy on forget questions (99.7% original, 91.1% paraphrased) before unlearning.
- Training Details: Trained for 15 epochs with a learning rate of 1e-4, using LoRA with rank 64 and alpha 128.
Use Cases
This model is primarily intended for:
- Machine Unlearning Research: As a foundational checkpoint for developing and evaluating unlearning algorithms within the RAQUEL framework.
- Benchmarking: Providing a consistent baseline for comparing the effectiveness of different unlearning strategies against a fully trained model.
- Understanding Model Behavior: Analyzing how models perform on specific question-answer tasks before any unlearning procedures are applied.