Hyukkyu/Qwen3-8B-Base-RAQUEL-WMDP-M-orig-LoRA-v1
Hyukkyu/Qwen3-8B-Base-RAQUEL-WMDP-M-orig-LoRA-v1 is an 8 billion parameter Qwen3-based language model, specifically a LoRA-trained baseline (M_orig) for RAQUEL WMDP unlearning experiments. It is fine-tuned on question-answer pairs from the RAQUEL2 'wmdp' source QA dataset, focusing on both forget and retain questions. This model serves as a foundational checkpoint for evaluating machine unlearning techniques, rather than being an unlearned model itself.
Loading preview...
Model Overview
This model, Hyukkyu/Qwen3-8B-Base-RAQUEL-WMDP-M-orig-LoRA-v1, is an 8-billion parameter Qwen3-based language model. It represents the M_orig baseline, which is a LoRA-trained version of Qwen/Qwen3-8B-Base. Its primary purpose is to serve as a starting point for RAQUEL WMDP unlearning experiments, providing a reference for comparison against unlearned models (M_ret).
Key Characteristics
- Architecture: Based on the Qwen3-8B-Base model, fine-tuned using LoRA (rank 64, alpha 128, dropout 0.05).
- Training Data: Trained on 2973 question-answer pairs from the
Hyukkyu/RAQUEL2-ICLRdataset, specifically thesource-qaconfiguration. - Training Objective: Focused on answer-only loss with the prompt
Question: {question}\nAnswer:, trained for 15 epochs. - Context Length: Supports a context length of 32768 tokens.
- Evaluation Baseline: Functions as a baseline for evaluating machine unlearning, demonstrating high accuracy (99.8% on original forget questions, 99.2% on original retain questions) before unlearning is applied.
Intended Use
This model is specifically designed for researchers and developers working on machine unlearning, particularly within the context of the RAQUEL framework. It provides a robust, pre-trained baseline for conducting and evaluating unlearning experiments. It is not an unlearned model itself, but rather the 'original' state from which unlearning processes would commence. Users should note that it was not trained with a chat template and expects a specific QA prompt format.