donghyunli/Llama-2-70b-KronQ-W3A16-fake
The donghyunli/Llama-2-70b-KronQ-W3A16-fake model is a 70 billion parameter Llama-2-hf derivative, featuring 3-bit weights quantized with KronQ and stored in fp16 format. This fake-quantized checkpoint is designed for perplexity and accuracy reproduction, maintaining the same size as bf16 models. It achieves a perplexity of 3.66 on WikiText-2 and an average zero-shot accuracy of 73.59% across various benchmarks. This model is primarily intended for research and evaluation of low-bit quantization techniques.
Loading preview...
Model Overview
This model, donghyunli/Llama-2-70b-KronQ-W3A16-fake, is a 70 billion parameter variant of the Llama-2-hf architecture. It utilizes KronQ for 3-bit weight quantization (W3A16), with these weights stored in fp16 format. This specific checkpoint is a "fake-quant" version, meaning it's primarily for reproducing perplexity and accuracy results, and is not a deployable low-bit model (it has the same size as a bf16 model).
Key Characteristics
- Architecture: Llama-2-70b-hf base model.
- Quantization: KronQ W3A16 (3-bit weights, 16-bit activations), per-channel asymmetric W3, weight-only, with
--alpha 0.25, BiIP,act_order, and raw H_G. - Format: Fake-quant fp16 checkpoint, not a packed low-bit model for deployment.
Performance Metrics
- Perplexity (WikiText-2, seqlen 2048): 3.66
- Zero-shot Accuracy (lm-evaluation-harness):
- PIQA: 82.26
- ARC-E: 81.86
- ARC-C: 58.28
- HellaSwag: 82.45
- WinoGrande: 78.53
- BoolQ: 82.97
- OBQA: 48.80
- Average: 73.59
Usage and Licensing
This model loads as a standard fp16 model using transformers library, requiring no specific KronQ code for inference. It is a derivative of Llama-2-70b-hf and is subject to the Llama 2 license.