donghyunli/Llama-2-7b-KronQ-W3A16-fake
The donghyunli/Llama-2-7b-KronQ-W3A16-fake is a Llama-2-7b model quantized to 3-bit weights using KronQ (Kronecker-factored Hessian quantization). This specific version is a fake-quant fp16 checkpoint, meaning the 3-bit weights are stored in fp16 for accuracy reproduction and research, not for compression or speedup. It achieves a WikiText-2 perplexity of 5.83 and an average zero-shot accuracy of 63.52 across various benchmarks. This model is designed for evaluating the accuracy of 3-bit quantization without the overhead of low-bit deployment.
Loading preview...
Overview
This model, donghyunli/Llama-2-7b-KronQ-W3A16-fake, is a 7-billion parameter Llama-2 base model that has undergone 3-bit weight quantization using the KronQ (Kronecker-factored Hessian quantization) method. It is presented as a "fake-quant fp16 checkpoint," which means the 3-bit weights are stored in standard fp16 format. This particular release is intended solely for perplexity and accuracy reproduction and research purposes, as it does not offer the compression or speedup benefits typically associated with low-bit quantization. For deployable low-bit versions, users are directed to W4A16 or W2A16 packed int4/int2 repositories.
Key Capabilities & Performance
- Quantization Method: Utilizes KronQ for 3-bit weight quantization (W3A16).
- Perplexity: Achieves a WikiText-2 perplexity of 5.83 (sequence length 2048).
- Zero-shot Accuracy: Demonstrates an average zero-shot accuracy of 63.52% across benchmarks including PIQA, ARC-E, ARC-C, HellaSwag, WinoGrande, BoolQ, and OBQA.
- Recipe: Employs per-channel asymmetric W3, weight-only (a_bits=16),
--alpha 0.25, bidirectional incoherence processing (BiIP),act_order, and raw H_G, calibrated on 128 WikiText-2 sequences.
Usage & Limitations
This model loads as a standard fp16 model using the transformers library, requiring no special KronQ code for inference. It is crucial to note that due to its fake-quant fp16 nature, this model is not optimized for deployment where memory compression or inference speedup from low-bit quantization is desired. Its primary utility lies in academic and research contexts for evaluating the accuracy of 3-bit quantization techniques.