MingZwhy/Qwen3-0.6B-W1.88-QAOPD
MingZwhy/Qwen3-0.6B-W1.88-QAOPD is a 0.8 billion parameter Qwen3-based causal language model developed by MingZwhy. This model features a highly optimized quantization to 1.88 bits per weight, achieved through quantization-aware distillation and on-policy distillation. It is designed for efficient deployment and evaluation, with quantization baked directly into the model. The model demonstrates competitive performance on various benchmarks, including GSM8K, MATH-500, and HumanEval, making it suitable for resource-constrained environments requiring general language understanding and problem-solving capabilities.
Loading preview...
Model Overview
MingZwhy/Qwen3-0.6B-W1.88-QAOPD is a highly optimized version of the Qwen3-0.6B model, developed by MingZwhy. Its primary distinguishing feature is its aggressive quantization to an effective 1.88 bits per weight, achieved through a two-stage process: quantization-aware distillation (QAD) followed by on-policy distillation (OPD). This "baked-in" quantization means the model is ready for efficient deployment and evaluation without additional quantization steps.
Key Technical Details
- Quantization Scheme: Weights are a mixed INT1.58 / INT4 format in 256-block chunks, resulting in an effective 1.88 bits per weight. Embedding and output head use INT4, while activations are INT8. The KV cache remains 16-bit.
- Parameter Count: 0.8 billion parameters.
- Context Length: 32768 tokens.
Performance Highlights
Despite its aggressive quantization, the model maintains competitive performance across several benchmarks when compared to its unquantized BF16 counterpart. For instance, it achieves:
- 39.65 on GSM8K (5-shot strict-match)
- 35.7 on MBPP (pass@1 greedy)
- 35.4 on HumanEval (pass@1 greedy)
Use Cases
This model is particularly well-suited for applications where computational resources and memory are constrained, but a capable language model is still required. Its efficient design makes it ideal for:
- Edge device deployment
- Applications requiring fast inference with a smaller footprint
- General language understanding and generation tasks where the slight performance trade-off from quantization is acceptable for significant efficiency gains.