reaperdoesntknow/Qwen3-1.7B-Thinking-Distil

Overview

Qwen3-1.7B-Thinking-Distil is a 1.7 billion parameter language model from Convergent Intelligence LLC, distilled from the larger Qwen3-30B-A3B-Thinking teacher. This model uniquely captures and compresses the teacher's extended deliberation patterns, including reasoning through uncertainty, backtracking, and re-evaluation, into a smaller, efficient student model. It was trained using Supervised Fine-Tuning (SFT) on the longwriter-6k dataset, which is rich in long-form reasoning samples, ensuring the transfer of complex thought processes rather than just final token probabilities.

Key Capabilities

• Extended Reasoning: Designed to generate long-form reasoning chains and internal monologues, mimicking a larger model's deliberative process.
• Efficient Size: Compresses advanced reasoning capabilities into a 1.7B parameter model, offering a balance of performance and computational efficiency.
• Large Context Window: Supports a maximum position context length of 40,960 tokens, enabling the processing and generation of extensive text.
• Qwen3 Architecture: Built on the Qwen3ForCausalLM architecture, featuring Grouped-Query Attention (GQA) with 16 query heads and 8 key/value heads.

Good For

• Complex Problem Solving: Ideal for tasks requiring detailed, step-by-step reasoning and deliberation before arriving at a conclusion.
• Long-form Content Generation: Excels in scenarios where the output needs to demonstrate deep thought processes, such as explanations, analyses, or creative writing that involves internal logic.
• Research and Development: Useful for exploring and replicating advanced reasoning behaviors of larger models in a more accessible format.

Usage Tips

For optimal performance in reasoning tasks, it is recommended to use a temperature between 0.6 and 0.8, a repetition penalty of 1.1 to 1.2, and allow for max_new_tokens between 1024 and 2048 to fully leverage its long-generation capabilities.