donghyunli/Llama-2-13b-KronQ-W3A16-g128-fake
The donghyunli/Llama-2-13b-KronQ-W3A16-g128-fake model is a 13 billion parameter Llama-2 derivative, quantized to 3-bit weights (group 128) using the KronQ method. This specific version is a fake-quantized FP16 checkpoint, designed for accuracy reproduction rather than deployable low-bit inference, maintaining the same size as a BF16 model. It achieves a perplexity of 5.135 on WikiText-2 and an average zero-shot accuracy of 68.59% across various benchmarks, making it suitable for evaluating the performance of KronQ quantization on a Llama-2 base.
Loading preview...
Overview
This model, donghyunli/Llama-2-13b-KronQ-W3A16-g128-fake, is a 13 billion parameter variant of the Llama-2 architecture. It has been quantized to 3-bit weights with a group size of 128 using the KronQ method. Notably, this is a fake-quantized FP16 checkpoint, meaning the 3-bit weights are stored in FP16 format. This design choice prioritizes the reproduction of perplexity (PPL) and accuracy results, rather than providing a deployable low-bit model. For actual deployable low-bit versions, users should refer to the W4A16-g128 or W2A16-g128 (packed) repositories.
Key Capabilities & Performance
- Quantization Method: Utilizes KronQ for 3-bit weight quantization (W3A16-g128).
- Format: Stored as a standard FP16 model, allowing it to be loaded directly with
transformers.AutoModelForCausalLMwithout requiring specific KronQ code. - Perplexity: Achieves a perplexity of 5.135 on the WikiText-2 dataset (sequence length 2048).
- Zero-shot Accuracy: Demonstrates an average zero-shot accuracy of 68.59% across multiple benchmarks, including PIQA (79.22), ARC-E (76.30), ARC-C (48.72), HellaSwag (77.53), WinoGrande (72.14), BoolQ (81.62), and OBQA (44.60).
Usage & Recipe
This model can be loaded like any standard FP16 model using the Hugging Face transformers library. The quantization recipe involved group-128 asymmetric W3, weight-only quantization with --alpha 0.25, --act_order, BiIP, and raw H_G. The underlying license is the Llama 2 license, as it is a derivative of Llama-2-13b.
When to Use This Model
- Research & Evaluation: Ideal for researchers and developers looking to reproduce or evaluate the performance characteristics of KronQ 3-bit quantization on a Llama-2 base model.
- Accuracy Assessment: Useful for understanding the impact of this specific quantization scheme on model accuracy and perplexity before deploying truly low-bit models.