minjaechoi/qwen36-35b-a3b-2p00bit-r4
minjaechoi/qwen36-35b-a3b-2p00bit-r4 is a 35.1 billion parameter language model based on the Qwen3.6-35B-A3B architecture, featuring an internal research checkpoint with routed experts. This model utilizes a hybrid quantization approach where routed experts average 2.00 bits, while other weights are in BF16 format. It is designed for efficient inference with stock `transformers` and vLLM, offering a balance between performance and reduced memory footprint.
Loading preview...
Model Overview
minjaechoi/qwen36-35b-a3b-2p00bit-r4 is an experimental 35.1 billion parameter language model derived from the Qwen/Qwen3.6-35B-A3B base model. This version incorporates a unique quantization strategy, utilizing "routed experts" that average 2.00 bits for their weights, while all other weights remain in BF16 (BFloat16) precision.
Key Characteristics
- Hybrid Quantization: Combines ultra-low-bit routed experts with BF16 weights for a balance of efficiency and performance.
- Base Model: Built upon the robust Qwen3.6-35B-A3B architecture.
- Inference Compatibility: Designed to load and run seamlessly with standard
transformersand vLLM libraries, as weights are stored dequantized in BF16 tensors. - Research Checkpoint: Represents an internal research iteration focused on exploring efficient model representations.
Potential Use Cases
This model is particularly suited for scenarios where:
- Memory Efficiency is Critical: The hybrid quantization scheme aims to reduce the model's memory footprint during inference.
- Experimental Deployment: Users interested in evaluating the performance of models with advanced quantization techniques.
- Cost-Effective Inference: Potentially offers lower computational costs compared to full-precision models of similar size, while maintaining reasonable performance.