MingZwhy/Qwen3-1.7B-W1.88-QAOPD
MingZwhy/Qwen3-1.7B-W1.88-QAOPD is a 2 billion parameter Qwen3-1.7B model, quantized to an effective 1.88 bits per weight using a mixed INT1.58/INT4 scheme. This model leverages quantization-aware distillation (QAD) and on-policy distillation (OPD) to recover performance after quantization. It is designed for efficient deployment while maintaining a 32768 token context length, making it suitable for applications requiring reduced memory footprint.
Loading preview...
Model Overview
MingZwhy/Qwen3-1.7B-W1.88-QAOPD is a 2 billion parameter language model based on the Qwen3-1.7B architecture. Its primary distinguishing feature is its aggressive quantization to an effective 1.88 bits per weight, achieved through a mixed INT1.58/INT4 scheme for weights, INT4 for embedding and output head, and INT8 for activations. This quantization is "baked in," meaning the model can be loaded and used directly in its quantized form.
Key Quantization Details
- Weights: Mixed INT1.58 / INT4 in 256-block chunks, with 12.5% of blocks at INT4, resulting in 1.88 bits per weight effective.
- Embedding & Output Head: INT4.
- Activations: INT8.
- KV Cache: 16-bit.
Performance Considerations
While significantly reducing memory footprint, the quantization process does impact performance compared to the unquantized Qwen3-1.7B BF16 model. For example, on GSM8K, this model scores 44.05 compared to 68.76 for the BF16 version, and on HumanEval, it achieves 45.1 versus 67.1. The quantization and recovery process involved quantization-aware distillation (QAD) followed by on-policy distillation (OPD).
Use Cases
This model is ideal for scenarios where computational resources and memory are highly constrained, but a capable language model is still required. Developers can integrate it using the Hugging Face transformers library. The training recipe and evaluation harness are available in the MingZwhy/QAOPD repository.