donghyunli/Meta-Llama-3-8B-KronQ-W3A16-g128-fake
The donghyunli/Meta-Llama-3-8B-KronQ-W3A16-g128-fake model is a 3-bit quantized version of Meta-Llama-3-8B, utilizing the KronQ quantization method with group-128 weights. This specific release is a fake-quantized fp16 checkpoint, designed for reproducing perplexity and accuracy results rather than deployable low-bit inference. It achieves a WikiText-2 perplexity of 6.959 and an average zero-shot accuracy of 67.51% across various benchmarks, making it suitable for evaluating the performance characteristics of KronQ-quantized Llama 3 models.
Loading preview...
Overview
This model, donghyunli/Meta-Llama-3-8B-KronQ-W3A16-g128-fake, is a fake-quantized fp16 checkpoint of the Meta-Llama-3-8B base model. It implements 3-bit weights (W3) with 16-bit activations (A16) using the KronQ quantization method, specifically with a group size of 128. Unlike deployable low-bit models, this version stores the 3-bit weights in fp16 format, resulting in the same size as a bf16 model. Its primary purpose is to allow for accurate reproduction of perplexity and accuracy metrics achieved by KronQ quantization without requiring specialized low-bit inference engines.
Key Characteristics
- Base Model: Meta-Llama-3-8B.
- Quantization Method: KronQ, with 3-bit weights (W3) and 16-bit activations (A16).
- Weight Group Size: 128.
- Format: Fake-quantized fp16 checkpoint, meaning weights are stored in fp16 for evaluation purposes.
- Performance: Achieves a perplexity of 6.959 on WikiText-2 (seqlen 2048) and an average zero-shot accuracy of 67.51% across benchmarks like PIQA, ARC-E, ARC-C, HellaSwag, WinoGrande, BoolQ, and OBQA.
Usage and Licensing
This model loads as a standard fp16 model using transformers.AutoModelForCausalLM, requiring no specific KronQ code for inference. It is a derivative of Meta-Llama-3-8B and adheres to the Llama 3 license.