valiolla/Qwen3.8-4B-Distill-heretic

VISIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:4.5BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Aug 23, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

valiolla/Qwen3.8-4B-Distill-heretic is a 4.5 billion parameter causal language model, a decensored version of Empero's Qwen3.8-4B-Distill. It is a full-parameter distillation of a frontier-scale Qwen3.8 teacher model into the Qwen3.5-4B architecture, optimized for reasoning, mathematics, and instruction following. This model features distilled chain-of-thought reasoning and a 262,144-token native context, designed to run efficiently on consumer hardware.

Loading preview...

Overview

This model, valiolla/Qwen3.8-4B-Distill-heretic, is a decensored variant of Empero's Qwen3.8-4B-Distill, created using the Heretic v1.4.0 tool. It is a 4.5 billion parameter causal language model based on the Qwen3.5-4B architecture, developed by Empero. The model is a full fine-tune, meaning every parameter was updated, not just an adapter.

Key Capabilities & Features

  • Distilled Chain-of-Thought: Learns reasoning directly from Qwen3.8 2.4T A95B teacher traces, starting answers with a <think> block.
  • Efficient Size: At 4.5B parameters, it's designed to run comfortably on consumer hardware (bf16 fits in ~8 GB).
  • Native Function Calling: Inherits Qwen3.5's native function calling capabilities without requiring additional fine-tuning.
  • Extended Context: Features a 262,144-token native context length, inherited from the Qwen3.5 base.
  • Decensored: Modified to reduce refusals, showing 24/100 refusals compared to the original model's 99/100.

Performance Highlights

While the original Qwen3.8-4B-Distill showed a slight decrease in gsm8k_cot exact match compared to its Qwen3.5-4B base (0.785 vs 0.850), it demonstrated significant improvements in MMLU (CoT, 57 subjects) accuracy, increasing from 0.354 to 0.553 (flexible-extract) and 0.071 to 0.233 (strict-match).

Best Practices

  • Use sampling parameters like temperature=0.6, top_p=0.95, top_k=20 to avoid repetition loops.
  • Allow generous max_new_tokens (e.g., 16,384) as answers begin with a <think> block that should be parsed and stripped for end-users.

Ideal Use Cases

This model is particularly well-suited for tasks requiring:

  • Reasoning and Problem Solving: Excels in mathematics and general reasoning due to its distilled chain-of-thought.
  • Instruction Following: Strong performance in adhering to given instructions.
  • Deployment on Consumer Hardware: Its 4.5B parameter size makes it accessible for local execution.

Finetunes
1 models