DuoNeural/Gemma-4-E2B-Heretic
DuoNeural/Gemma-4-E2B-Heretic is a 5.1 billion parameter Gemma 4-E2B model, developed by DuoNeural, that has been abliterated using the Heretic LoRA-based Pareto optimization framework. This model is specifically engineered to achieve a low KL divergence of 0.057 against the base model, making its output distribution nearly indistinguishable for general tasks while maintaining an 85% comply rate on safety evaluations. It is optimized for scenarios requiring a balance between safety and minimal deviation from the base model's general performance.
Loading preview...
DuoNeural/Gemma-4-E2B-Heretic Overview
DuoNeural/Gemma-4-E2B-Heretic is a 5.1 billion parameter model derived from Gemma 4-E2B, distinguished by its "abliteration" using the Heretic LoRA-based Pareto optimization framework. This process aims to surgically reduce refusal rates while minimizing the impact on the model's general output distribution.
Key Characteristics
- Low KL Divergence: Achieves a KL divergence of 0.057 against the base model, indicating that its output distribution is nearly identical for general, neutral factual prompts. This is measured using the Heretic v2.0 KL methodology, which considers the full 262K token vocabulary and first-token logits.
- Targeted Refusal Reduction: Demonstrates an 85% comply rate on a 20-prompt refusal evaluation, effectively addressing most safety concerns. The remaining 15% refusal rate (3 prompts) represents hard safety geometry that was not fully removed, indicating a deliberate trade-off for minimal KL divergence.
- Methodology: Utilizes a LoRA Pareto sweep targeting
o_projanddown_projlayers (42 layers each) with BF16 quantization, selecting the Pareto-optimal trial (#89).
Ideal Use Cases
- Applications requiring minimal distributional shift: Suitable for tasks where maintaining the original model's output characteristics is crucial, but with added safety guardrails.
- Controlled safety environments: Useful for deployments where specific harmful outputs need to be mitigated without broadly altering the model's behavior on benign queries.
- Research into safety-performance trade-offs: Provides a clear example of a model optimized for a precise balance between safety and fidelity to the base model's capabilities.