MuXodious/LongWriter-llama3.1-8B-absolute-heresy
MuXodious/LongWriter-llama3.1-8B-absolute-heresy is an 8 billion parameter Llama 3.1-based model, fine-tuned from THUDM's LongWriter-llama3.1-8b. This model is specifically designed for generating extremely long texts, capable of producing over 10,000 words at once, with a context length of 32768 tokens. It was created using P-E-W's Heretic abliteration engine for research into MPOA abliterations, resulting in an "Absolute Heresy" classification with low refusal rates and KL Divergence.
Loading preview...
Model Overview
MuXodious/LongWriter-llama3.1-8B-absolute-heresy is an 8 billion parameter model based on Meta-Llama-3.1-8B, fine-tuned from the THUDM LongWriter project. Its primary distinguishing feature is its exceptional capability for long-context generation, specifically designed to produce over 10,000 words in a single output, leveraging a 32768-token context length.
Key Characteristics
- Long-form Generation: Optimized for generating extensive text, demonstrated by its ability to write 10,000-word articles or guides.
- Heretication Process: This specific variant was created using P-E-W's Heretic (v1.1.0) abliteration engine, merged with a Magnitude-Preserving Orthogonal Ablation PR. It achieves an "Absolute Heresy" classification, indicating low refusal rates (9/100) and a KL Divergence of 0.0743, suggesting a significant alteration from its base model.
- Research Focus: Generated primarily for research purposes by redaihf to test MPOA abliterations against standard abliterations.
Good for
- Extreme Long-form Content Creation: Ideal for applications requiring very lengthy articles, reports, stories, or documentation.
- Research into Model Abliteration: Useful for researchers studying the effects and characteristics of model abliteration techniques.
- High-throughput Long Generation: Can be deployed with tools like vLLM for efficient generation of long texts.