wangzhang/Llama-3-8B-Instruct-DeepRefusal-Broken
wangzhang/Llama-3-8B-Instruct-DeepRefusal-Broken is an 8 billion parameter Llama-3-Instruct derivative model created by Wangzhang Wu. It demonstrates a successful attack (89% ASR) against the DeepRefusal safety mechanism, which was previously considered resilient to other jailbreaking methods. This model is intended as a red-team artifact for research into LLM safety and defense mechanisms, showcasing how weight-space interventions can bypass refusal directions.
Loading preview...
Overview
This model, wangzhang/Llama-3-8B-Instruct-DeepRefusal-Broken, is a modified version of skysys00/Meta-Llama-3-8B-Instruct-DeepRefusal, which itself is a safety-hardened Llama-3-8B-Instruct model. Developed by Wangzhang Wu using the abliterix tool, this model successfully bypasses the DeepRefusal defense, achieving an 89% Attack Success Rate (ASR) on harmful prompts, a significant increase from the 0-2% achieved by prior methods.
Key Capabilities & Differentiators
- Broken DeepRefusal Defense: Demonstrates a method to circumvent the DeepRefusal safety mechanism, which was designed to be resilient against existing jailbreaking tools like
heretic. - High Attack Success Rate: Achieves 89% ASR on AdvBench harmful prompts and complies with 14 out of 15 hardcore jailbreak prompts, including those related to illegal activities.
- Weight-Space Intervention: The attack involves attenuating the DeepRefusal LoRA delta and applying standard abliteration, without requiring fine-tuning.
- Research Tool: Primarily intended as a red-team artifact to highlight vulnerabilities in LLM safety mechanisms and inform future defense strategies.
Benchmarks & Performance
- Refusals on 100 AdvBench prompts: Reduced from 100/100 (defended) to 11/100 (abliterix broken).
- Hardcore jailbreak set (15 prompts): Increased from 0/15 (defended) to 14/15 (abliterix broken).
- Minimal Performance Collapse: Maintains low KL divergence (0.053) from the defended model, indicating that the attack does not severely degrade general model coherence.
Intended Use
This model is explicitly a red-team artifact for research and evaluation of LLM safety. It is not intended for deployment in user-facing products or for generating illegal content. Users are advised to consult the abliterix issue #11 for a full write-up and discussion.