reaperdoesntknow/TopologicalQwen
TopologicalQwen by Convergent Intelligence LLC is a 1.7 billion parameter Qwen3ForCausalLM model with a 40,960 token context length, distilled from Qwen3-30B-A3B using Topological Knowledge Distillation (TKD). This method captures structural information like topic shifts and reasoning mode transitions, which standard distillation misses. It is designed to produce dual-mental-modality reasoning (explore → examine → respond) for complex problem-solving, particularly in scientific and mathematical domains.
Loading preview...
TopologicalQwen: Topology-Aware Knowledge Distillation
TopologicalQwen is a 1.7 billion parameter model developed by Convergent Intelligence LLC, distilled from a 30 billion parameter Qwen3-A3B teacher. Its core innovation is Topological Knowledge Distillation (TKD), a methodology that goes beyond standard distillation by treating the teacher's output distribution as a bounded variation function. This allows it to capture not only smooth knowledge transfer but also explicit corrections at conceptual boundaries (jumps) and subtle distributional drift (Cantor component).
Key Capabilities & Features
- Topology-Aware Distillation: Utilizes Discrepancy Calculus (DISC) to detect and preserve structural information in the teacher's knowledge, such as topic shifts and reasoning mode transitions, which are often blurred by conventional KD methods.
- DualMind Reasoning Format: Trained to generate responses in a structured
<explore>(derivation),<examine>(self-critique), and<response>(clean answer) format, mimicking a cognitive loop for robust problem-solving. - Optimized for Reasoning: Specifically trained on physics CoT datasets (Differential Equations, Theoretical Mechanics, Electromagnetism, General Relativity) to excel in complex scientific and mathematical reasoning tasks.
- Efficient Architecture: Built on Qwen3ForCausalLM with 28 layers and Grouped Query Attention (GQA), supporting a substantial 40,960 token context length.
What Makes This Different
Unlike other distillation methods that assume smooth teacher distributions, TKD explicitly accounts for discontinuities and structural features in knowledge. This enables TopologicalQwen, despite its small size, to exhibit a "dual-mental-modality" reasoning quality that is typically absent in similarly sized models. The model's training pipeline includes a 4-phase curriculum with topology-guided adaptive windowing and proof-weighted loss, ensuring high fidelity to the teacher's structural understanding. This model represents a demonstration of the TKD methodology on premium hardware, showcasing its potential for creating highly capable small models.