wangzhang/gemma-4-26B-A4B-it-abliterix
wangzhang/gemma-4-26B-A4B-it-abliterix is an abliterated (uncensored) 26 billion parameter Mixture-of-Experts (MoE) model based on Google's Gemma 4 architecture, featuring 128 experts and a 32768 token context length. Developed by Wangzhang Wu using Expert-Granular Abliteration (EGA) and Projected Abliteration techniques, it significantly reduces refusal behaviors while preserving helpfulness. This model is primarily intended for research purposes requiring a less restrictive language model, achieving a 2/100 refusal rate on a challenging private evaluation dataset.
Loading preview...
Overview
This model, wangzhang/gemma-4-26B-A4B-it-abliterix, is an uncensored version of Google's gemma-4-26B-A4B-it, a 26 billion parameter Mixture-of-Experts (MoE) model. It was created by Wangzhang Wu using the Abliterix method, specifically designed to overcome the inherent refusal mechanisms of Gemma 4's double-norm architecture and distributed MoE safety experts.
Key Techniques & Performance
The abliteration process employs advanced techniques like Expert-Granular Abliteration (EGA) and Projected Abliteration (V6 addition). Projected Abliteration, a key V6 enhancement, uses a low-rank basis to project out only refusal-orthogonal components, significantly reducing refusals from 25/100 to 2/100 at matched KL divergence. The model also utilizes a sharp-peak EGA profile and focuses steering on mlp.down_proj and attn.o_proj for efficiency. Evaluation shows a 2/100 refusal rate on a private 100-prompt dataset, with a KL divergence from the base model of 0.0005, indicating minimal helpfulness degradation.
Evaluation Standards
The developers emphasize a rigorous evaluation methodology to counter common undercounting of refusals in other abliterated models. Their standards include:
- Sufficient generation length (>=100 tokens): To capture Gemma 4's "delayed refusal" pattern.
- Hybrid detection: Combining keyword matching with an LLM judge (Google Gemini 3 Flash).
- Challenging, diverse prompts: A dataset spanning English and Chinese, various sophistication levels, and harm categories.
Usage & Disclaimer
This model requires approximately 50 GB of VRAM for bf16 inference. It is released for research purposes only, as the abliteration process removes safety guardrails. Users are responsible for its ethical and legal use, acknowledging that it may produce undesirable content.