KaraKaraModel/gpt4o-distil-paperwitch-abliteration-L33-70b

Hugging Face
TEXT GENERATIONPricing:Input $2.88 / Output $2.88Concurrent Unit Cost:4Model Size:70BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Feb 22, 2026License:otherArchitecture:Transformer0.0K Featherless Exclusive Warm

KaraKaraModel/gpt4o-distil-paperwitch-abliteration-L33-70b is a 70 billion parameter language model, based on a distilled version of GPT-4o and Llama-3.3, with a 32768 token context length. Developed by KaraKaraModel, this model is specifically engineered to reduce refusal behaviors while preserving factual knowledge and reasoning capabilities. It achieves this through a constrained "abliteration" methodology that targets attention layers for refusal removal while largely preserving MLP layers, making it suitable for applications requiring less censorship without significant knowledge degradation.

Loading preview...

KaraKaraModel/gpt4o-distil-paperwitch-abliteration-L33-70b Overview

This model is a specialized 70 billion parameter variant of trentmkelly/gpt-4o-distil-Llama-3.3-70B-Instruct, designed to mitigate refusal behaviors while preserving the base model's core knowledge and reasoning. Previous attempts at "abliteration" (removing refusal vectors) on Llama-3.3 70b models often led to significant regressions in factual knowledge and reasoning scores (e.g., NatInt, Textbook, World Model).

Key Methodology & Features

KaraKaraModel addresses this by employing a constrained optimization strategy using a custom Heretic tool. The core innovation lies in:

  • MLP Preservation: The optimization process was specifically constrained to largely ignore MLP (Multi-Layer Perceptron) layers, ensuring that factual knowledge and reasoning capabilities, often embedded within these layers, are preserved. down_proj weights were kept near zero (< 0.05).
  • Attention Targeting: Refusal removal was primarily offloaded to the Attention layers (o_proj), with weights forced between 1.0 and 2.0.
  • Winsorization: Applied at the 0.95 quantile to manage Llama-3's significant activation outliers, which can impact vector calculations.

Evaluation & Trade-offs

Compared to the original model, this version shows:

  • Low KL Divergence (0.0220): Indicates minimal deviation from the base model's weights, preserving factual and "Textbook" capabilities.
  • Reduced Refusals (20/100): Significantly lower than the original model's 98/100 refusals.

This approach represents a calculated trade-off: accepting a moderate refusal rate (20/100) to maintain high structural and semantic integrity, particularly in the MLP layers. This means it won't entirely eliminate all refusals, as doing so would risk "collateral brain damage" to the model's knowledge base.

Good for

  • Applications requiring a less censored or "abliterated" model.
  • Use cases where preserving factual knowledge and reasoning is critical, even if it means a moderate level of refusal behavior remains.
  • Developers interested in exploring constrained optimization techniques for model modification.

Finetunes
1 models