Model Overview
This model, kldzj/Llama-3.3-70B-Instruct-heretic, is a 70 billion parameter instruction-tuned large language model based on Meta's Llama 3.3 architecture. Its primary distinction is being a decensored version of the original meta-llama/Llama-3.3-70B-Instruct, achieved through the application of the Heretic v1.0.1 tool. This modification significantly reduces the model's refusal rate, dropping from 70/100 in the original to 6/100 in this version, while maintaining a KL divergence of 1.85.
Key Capabilities
- Decensored Responses: Engineered to provide fewer refusals, offering more direct and unfiltered outputs compared to its base model.
- Multilingual Dialogue: Optimized for assistant-like chat in multiple languages, including English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
- Optimized Transformer Architecture: Utilizes an auto-regressive language model with an optimized transformer architecture, incorporating Grouped-Query Attention (GQA) for improved inference scalability.
- Extensive Training: Pretrained on over 15 trillion tokens of publicly available online data, with a knowledge cutoff of December 2023.
- Instruction Following: Fine-tuned using supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) for strong instruction adherence.
Good For
- Research into Decensored LLMs: Ideal for researchers exploring the impact and behavior of models with reduced safety alignments.
- Applications Requiring Unfiltered Responses: Suitable for use cases where a more direct or less constrained output is desired, provided appropriate ethical considerations and safeguards are implemented by the developer.
- Multilingual Chatbots and Assistants: Excels in dialogue-based applications across its supported languages.
- General Natural Language Generation: Can be adapted for a wide range of text generation tasks where the base Llama 3.3's capabilities are relevant, with the added characteristic of reduced refusals.