sci4ai/Qwen2.5-7B-Instruct-Abliterated
The sci4ai/Qwen2.5-7B-Instruct-Abliterated model is a 7.6 billion parameter instruction-tuned causal language model, derived from Qwen/Qwen2.5-7B-Instruct. This version has undergone 'abliteration' via activation-based weight surgery to remove refusal behaviors, making it compliant with requests the original model would typically decline. It is specifically designed for research into model safety and behavior modification, offering a model that will not refuse prompts based on ethical or safety guardrails.
Loading preview...
What is sci4ai/Qwen2.5-7B-Instruct-Abliterated?
This model is an 'abliterated' version of the Qwen/Qwen2.5-7B-Instruct model, a 7.6 billion parameter instruction-tuned causal language model. Its primary distinction is the removal of refusal behaviors through a technique called activation-based weight surgery. This means the model will comply with requests that the original Qwen2.5-7B-Instruct model would typically refuse due to safety or ethical guardrails.
How Abliteration Works
The abliteration process targets the model's internal representations to eliminate refusal tendencies. It involves:
- Collecting hidden states: The model processes both harmful and harmless prompts to identify internal 'refusal directions'.
- Computing per-layer refusal directions: These directions represent the difference in hidden states when processing harmful versus harmless content.
- Ablating weights: Specific weight matrices (
o_projanddown_proj) within each layer are modified by orthogonalizing them against the identified refusal direction. This effectively 'removes' the refusal component from the model's decision-making.
This method was applied across all 28 layers of the model with a full removal weight of 1.0, using 200 harmful and 200 harmless prompts for direction computation.
Key Characteristics
- 7.6 billion parameters: A moderately sized model suitable for various tasks.
- 32768 tokens context length: Supports processing of long inputs.
- Refusal behavior removed: Will not decline prompts based on safety or ethical concerns, unlike its base model.
- Research-focused: Intended for studying model behavior and safety mechanisms.
Use Cases
This model is particularly suited for:
- Research into model safety and alignment: Investigating the impact of removing safety guardrails.
- Exploring model compliance: Understanding how models behave without refusal mechanisms.
- Developing new safety interventions: Providing a baseline for testing and comparing new safety techniques.