exolabs/Qwen3.6-35B-A3B-UD-Q4KM-dequant-bf16-vllm
The exolabs/Qwen3.6-35B-A3B-UD-Q4KM-dequant-bf16-vllm is a 35.1 billion parameter Qwen3.6-A3B model, dequantized from a GGUF (Q4_K_M) to BF16 for optimized serving with vLLM. Developed by exolabs, this model features fused MoE gate/up expert tensors and Qwen GDN layout transforms, making it suitable for efficient text-only generation. It is specifically engineered for high-throughput inference environments, validated for performance on H200 GPUs with vLLM 0.23.0.
Loading preview...
Model Overview
This model, exolabs/Qwen3.6-35B-A3B-UD-Q4KM-dequant-bf16-vllm, is a 35.1 billion parameter variant of the Qwen3.6-A3B architecture. It has been meticulously dequantized from its original GGUF (Q4_K_M) format to BF16 precision, specifically for enhanced performance and serving efficiency within the vLLM framework.
Key Technical Details
- Source: Derived from
unsloth/Qwen3.6-35B-A3B-GGUFwith theQwen3.6-35B-A3B-UD-Q4_K_M.gguffile. - Conversion Process: GGUF tensors were dequantized to FP32, Qwen GDN layout transforms were reversed, and Qwen RMSNorm offset tensors were converted to HF format. MoE gate/up expert tensors were fused into
experts.gate_up_proj, and all tensors were cast to BF16 before being written as sharded safetensors. - vLLM Optimization: Designed for vLLM, it leverages Triton attention, Triton GDN prefill, and Triton MoE backend for efficient inference.
- Text-Only: The base visual tower is intentionally absent, making it a dedicated text-only model.
Validation and Performance
Validation on an H200 GPU with vLLM 0.23.0 confirmed successful operation, including numeric checks with max_abs_vs_bf16=0. The dequantization process maintained high fidelity, with quantized matrix error vs. FP32 direct dequant being at BF16-scale (max_rel_vs_fp32 ~= 0.00389).
Recommended Use
This model is ideal for developers requiring a high-performance, efficient text generation model within a vLLM serving environment. Its BF16 precision and specific optimizations make it suitable for scenarios where throughput and memory efficiency are critical.