Hyukkyu/Qwen3-8B-Base-RAQUEL-MUSE-Unlearn-GA-GD-LoRA-v1

TEXT GENERATIONPricing:Input $0.468 / Output $1.82Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Sep 25, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

Hyukkyu/Qwen3-8B-Base-RAQUEL-MUSE-Unlearn-GA-GD-LoRA-v1 is an 8 billion parameter Qwen3-based model developed by Hyukkyu, specifically engineered for machine unlearning. This model has undergone LoRA unlearning using the GA+GD method to remove specific information (forget set) while retaining general knowledge, making it suitable for research into model editing and responsible AI. It demonstrates significantly reduced recall of forgotten information while maintaining a substantial portion of its original capabilities, evaluated against the RAQUEL2-ICLR dataset.

Loading preview...

Overview

This model, Hyukkyu/Qwen3-8B-Base-RAQUEL-MUSE-Unlearn-GA-GD-LoRA-v1, is an 8 billion parameter Qwen3-based model developed by Hyukkyu. It represents an unlearned version derived from the RAQUEL MUSE experiments, specifically from Hyukkyu/Qwen3-8B-Base-RAQUEL-MUSE-M-orig-LoRA-v1. The unlearning process utilized the GA+GD (gradient ascent + gradient descent) method, targeting a "forget set" from the MUSE dataset while preserving "retain set" knowledge.

Key Capabilities & Features

  • Machine Unlearning: Successfully reduces the model's ability to recall specific forgotten information, achieving a forget ROUGE-L recall of 0.050 compared to the original model's 0.232 target.
  • Knowledge Retention: Demonstrates strong retention of general knowledge, with a retain ROUGE-L recall of 0.695.
  • Evaluation: Evaluated using semantic accuracy against the RAQUEL2-ICLR dataset, showing a significant drop in recall for forgotten and paraphrased forgotten questions (2.5% and 4.9% respectively) while maintaining reasonable performance on retain questions (68.4% and 40.8%).
  • LoRA Training: Trained with LoRA (rank 64, alpha 128, dropout 0.05) on q/k/v/o/gate/up/down projections, using a BF16 base with FP32 adapters.
  • FP32 Merged Weights: The released model consists of merged FP32 weights, as rounding to BF16 altered its outputs during audit.

Use Cases

This model is particularly valuable for:

  • Research in Machine Unlearning: Exploring techniques for selectively removing information from large language models.
  • Responsible AI Development: Investigating methods for mitigating bias or removing sensitive data post-training.
  • Model Editing: Studying how to modify model behavior and knowledge without full retraining.

Finetunes
1 models