minjaechoi/qwen3p6-35b-a3b-2p03bit-r74
minjaechoi/qwen3p6-35b-a3b-2p03bit-r74 is a 35.1 billion parameter language model based on Qwen3.6-35B-A3B, featuring an average of 2.0341 bits per routed expert. This model utilizes a hybrid quantization scheme where routed experts are highly compressed while other weights remain in BF16 format. It is an internal research checkpoint designed for efficient inference with reduced memory footprint, making it suitable for applications requiring a balance of performance and resource optimization.
Loading preview...
Overview
minjaechoi/qwen3p6-35b-a3b-2p03bit-r74 is an internal research checkpoint derived from the Qwen3.6-35B-A3B base model. This 35.1 billion parameter model incorporates a unique quantization strategy, achieving an average of 2.0341 bits per routed expert. While the routed experts are highly compressed, all other weights are maintained in BF16 format, allowing for a balance between model size reduction and performance.
Key Characteristics
- Hybrid Quantization: Combines ultra-low-bit quantization for routed experts with BF16 for other weights.
- Efficient Loading: Weights are stored dequantized in BF16 tensors, ensuring compatibility and ease of loading with standard
transformersand vLLM libraries. - Research Focus: Represents an internal research iteration (r74) exploring efficient model architectures.
Good for
- Resource-constrained deployments: Ideal for scenarios where memory footprint and inference speed are critical.
- Experimental research: Useful for exploring the impact of hybrid quantization on large language models.
- Cost-effective inference: Potentially offers lower operational costs due to reduced memory requirements.