inference-optimization/Qwen3.6-8B-A1.6B

Model Overview

This model, inference-optimization/Qwen3.6-8B-A1.6B, is a scaled-down version of the Qwen/Qwen3.6-35B-A3B multimodal (vision-language) Mixture-of-Experts (MoE) model. It features 7.81 billion total parameters, with around 1.57 billion parameters activated per token, maintaining the complex architecture of the original. This includes a hybrid linear-attention and full-attention text backbone, 256 routed experts with top-8 routing, and a shared expert.

Architectural Fidelity

Despite its reduced size, the model preserves the full architectural characteristics of the larger Qwen3.6-35B-A3B. Key dimensions such as hidden size, expert count, MoE intermediate size, attention head dimensions, and vocabulary size remain unchanged. Only the depth of the text and vision layers has been reduced for this tiny version, specifically 8 text layers (6 linear, 2 full) and 2 vision layers. The hybrid attention pattern, with a full_attention_interval=4, is also maintained.

Purpose and Limitations

This model was created using the llm-compressor create-tiny-model skill, primarily for testing and development. Its weights were randomly initialized and fine-tuned on a small toy text dataset to achieve perplexity convergence. Consequently, it possesses no real-world language or vision capabilities and is not intended for practical applications. The vision tower is included to ensure architectural coverage but has not been trained on image data. It serves as a faithful architectural representation for experimentation.