HarimxChoi/WarpQuant-Llama-3-8B-R16E4H4
The HarimxChoi/WarpQuant-Llama-3-8B-R16E4H4 model is an 8 billion parameter Llama 3 variant, specifically optimized for efficient deployment through a novel quantization method. It utilizes signed Hadamard rotation, block-GPTQ, and Output-Fisher weak-column recovery to achieve a compact size of 3.389 GiB at 3.6256 bits per weight. This model is designed for scenarios requiring reduced memory footprint and faster inference while maintaining competitive performance across general language understanding and reasoning tasks.
Loading preview...
WarpQuant Llama 3 8B R16E4H4 Overview
This model is a specialized 8 billion parameter variant of Llama 3, developed by Harim Choi, focusing on advanced quantization techniques for efficient deployment. It employs a unique WarpQuant method, combining signed Hadamard rotation, block-GPTQ, and Output-Fisher weak-column recovery. This approach allows the model to achieve a significantly reduced payload of 3.389 GiB at an average of 3.6256 bits per weight (bpw), making it highly suitable for resource-constrained environments.
Key Quantization Details
- Projection weights: Use a 3.5-bpw INT3 base.
- Column recovery: Selected columns are restored in BF16 for improved accuracy.
- Embedding and output head: Quantized using group-128 INT4.
- Format: Quantized values are stored in BF16-compatible safetensors.
Performance Metrics
Compared to the BF16 base model, the WarpQuant version demonstrates a trade-off between size and performance, offering substantial memory savings with a controlled impact on benchmarks:
- WikiText-2 PPL: 7.3446 (vs. 6.2559 for BF16)
- ARC-299: 45.49% (vs. 50.50% for BF16)
- MMLU-13,943: 38.99% (vs. 41.04% for BF16)
This model is ideal for applications where minimizing model size and memory footprint are critical, while still requiring a capable Llama 3-based language model. For more technical details, refer to the WarpQuant technical report.