gitarist/Qwen3-0.6B-4bit-qqq-gptq-obq
The gitarist/Qwen3-0.6B-4bit-qqq-gptq-obq model is a 0.8 billion parameter Qwen3 architecture, specifically optimized for efficient deployment through advanced 4-bit quantization. It utilizes QQQ smoothing, GPTQ within-matrix, and OBQ across-matrix methods to achieve a compact representation with approximately 5.22 bits per weight. This model is designed for scenarios requiring high inference speed and reduced memory footprint while maintaining competitive performance, as indicated by its wikitext-2 PPL of 23.01.
Loading preview...
Qwen3-0.6B-4bit-qqq-gptq-obq: Highly Quantized Qwen3 Model
This model, developed by gitarist, is a 0.8 billion parameter variant of the Qwen3 architecture, meticulously optimized for efficiency through a sophisticated 4-bit quantization scheme. It's designed to deliver strong performance in resource-constrained environments.
Key Quantization Details
- Quantization Method: Employs a combination of QQQ smoothing, GPTQ within-matrix (act-order), and OBQ across-matrix (GGN + KL-teacher correction).
- Bit-depth: Weights are quantized to 4-bit (group_size 128, symmetric), with 8-bit embedding, resulting in approximately 5.22 bits per weight.
- Activation Support: Supports per-token dynamic int8 activations (W4A8) for further optimization.
Performance Metrics
- wikitext-2 PPL: Achieves a perplexity of 23.01.
- KL Divergence: Mean KL to fp16 is 0.153 (compared to fp16 reference PPL 20.96).
- W4A8 Performance: With per-token dynamic int8 activations, PPL is 23.26 and KL is 0.164.
Good for
- Edge device deployment: Its highly quantized nature makes it suitable for devices with limited memory and computational power.
- High-throughput inference: Optimized for faster inference speeds due to reduced model size.
- Applications requiring efficient LLMs: Ideal for scenarios where a balance between performance and resource consumption is critical.