empero-ai/Qwen3.8-4B-Distill
Qwen3.8-4B is a 4.5 billion parameter causal language model developed by Empero, distilled from the larger Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture. It is specifically optimized for reasoning, mathematics, and instruction following by learning directly from curated chain-of-thought traces. This model features a native 262,144-token context length and native function calling, designed to run efficiently on consumer hardware while retaining advanced reasoning capabilities.
Loading preview...
Overview
Empero's Qwen3.8-4B is a 4.5 billion parameter causal language model, a full-parameter distillation of the larger Qwen3.8 2.4T A95B. It leverages the Qwen3.5-4B architecture and was trained on approximately 45,000 curated teacher traces, focusing on dense chain-of-thought reasoning across mathematics, general reasoning, and instruction following. The primary goal of this distillation is to bring frontier-scale reasoning behavior to a smaller, more accessible 4B model.
Key Capabilities & Features
- Distilled Chain-of-Thought: Learns reasoning directly from high-quality teacher traces, incorporating a
<think>block in its responses. - Efficient Size: At 4.5B parameters, it fits comfortably on consumer hardware (bf16 in ~8 GB, quantized versions on laptops).
- Native Function Calling: Supports function calling as per Qwen3.5's specification without additional fine-tuning.
- Extended Context Window: Inherits a native 262,144-token context length from its Qwen3.5 base.
- Full Fine-tune: All parameters were updated during training, not just an adapter.
Performance Highlights
Benchmarking against the Qwen3.5-4B base model using CoT protocols shows significant improvements in reasoning tasks:
- MMLU (CoT, 57 subjects): Achieves 0.553 (flexible-extract accuracy), a +0.199 improvement over the base model's 0.354. Strict-match accuracy also saw a substantial gain of +0.162.
- While
gsm8k_cotshowed a slight decrease, the model's strength lies in complex reasoning and instruction following.
Good For
- Applications requiring strong mathematical and general reasoning capabilities.
- Instruction following tasks where detailed thought processes are beneficial.
- Deployments on consumer-grade hardware due to its efficient parameter count.
- Use cases benefiting from native function calling and a very long context window.
For optimal generation, Empero recommends sampling parameters like temperature=0.6, top_p=0.95, top_k=20 and allowing generous max_new_tokens to accommodate the initial <think> block.