barozp/Qwen3.8-27B-Opus-Distill
barozp/Qwen3.8-27B-Opus-Distill is a 27 billion parameter language model, fine-tuned from Qwen/Qwen3.8-27B, specifically optimized for enhanced reasoning capabilities. This model leverages distillation from Claude Opus reasoning traces to significantly improve performance on complex reasoning tasks like GPQA and ARC-Challenge, while maintaining general knowledge and language modeling proficiency. It retains the native vision tower and MTP head of its base model, making it a complete multimodal solution. Its primary strength lies in boosting reflex-level reasoning without degrading other core LLM functions.
Loading preview...
barozp/Qwen3.8-27B-Opus-Distill Overview
This model is a 27 billion parameter variant of the Qwen3.8-27B base model, fine-tuned using LoRA with reasoning traces distilled from Claude Opus. The primary objective of this distillation was to significantly enhance the model's reasoning abilities without compromising its general knowledge or language modeling performance. It successfully achieves this, showing substantial improvements on benchmarks like GPQA-Diamond (+26.3 points) and ARC-Challenge (+4.2 points), while MMLU and wikitext scores remain stable.
Key Capabilities
- Enhanced Reasoning: Demonstrates a marked improvement in reasoning tasks, particularly in "reflex-level" reasoning where the model internalizes step-by-step logic.
- Multimodal Support: Carries over the native vision tower and MTP (Multi-Task Prediction) head from the base Qwen model, ensuring full multimodal functionality.
- Knowledge Preservation: General knowledge and language modeling capabilities are maintained, with no observed degradation post-distillation.
- Efficient Fine-tuning: Achieved through LoRA (r=64, alpha=64) on specific attention and FFN projections, merged into the base weights.
Important Considerations
- Known Issue: An identified reasoning loop can occur when combining "no prose" with other output-format constraints, potentially consuming the entire token budget without output. This issue is addressed in
barozp/Qwen3.8-27B-Opus-Distill-v2. - Benchmark Interpretation: GPQA scores are not directly comparable to Qwen's published results due to different measurement protocols (this model's GPQA is 0-shot, loglikelihood, thinking OFF).
Good for
- Applications requiring strong reasoning capabilities, especially in scenarios where quick, internalized logical processing is beneficial.
- Multimodal tasks that can leverage the intact vision and MTP functionalities alongside improved reasoning.
- Users seeking a Qwen-based model with a specific focus on boosting analytical and problem-solving performance.