donghyunli/Llama-2-70b-KronQ-W3A16-g128-fake

TEXT GENERATIONConcurrent Unit Cost:4Model Size:69BQuant:FP8Context Size:4kPublished:Jun 24, 2026License:llama2Architecture:Transformer Open Weights Featherless Exclusive Cold

This model is a 3-bit quantized version of Llama-2-70b, developed by donghyunli using the KronQ method, with weights stored in fp16 for accuracy reproduction. It features 3-bit group-128 weights and achieves a perplexity of 3.602 on WikiText-2. This variant is designed for evaluating the performance of KronQ quantization without requiring specialized low-bit deployment, making it suitable for research and benchmarking of quantized large language models.

Loading preview...

Overview

This model, Llama-2-70b-KronQ-W3A16-g128-fake, is a 3-bit quantized version of the original Llama-2-70b, developed by donghyunli using the KronQ quantization method. It specifically uses 3-bit weights (W3) with a group size of 128 (g128) and stores these weights in fp16 format. This "fake-quant" approach means it maintains the same size as a bf16 model, allowing for accurate reproduction of perplexity and accuracy results without requiring specialized low-bit inference engines.

Key Characteristics

  • Quantization Method: KronQ, utilizing 3-bit asymmetric weights (W3) with group-128 quantization.
  • Weight Storage: Weights are stored in fp16, not packed into lower-bit integers, for direct evaluation of quantization impact.
  • Performance: Achieves a perplexity of 3.602 on WikiText-2 (seqlen 2048).
  • Zero-shot Accuracy: Demonstrates an average zero-shot accuracy of 73.37% across various benchmarks including PIQA, ARC-E, ARC-C, HellaSwag, WinoGrande, BoolQ, and OBQA.
  • Recipe Details: Implements --alpha 0.25, --act_order, BiIP, and raw H_G during quantization.

Usage and Purpose

This model loads as a standard fp16 model using transformers.AutoModelForCausalLM, requiring no specific KronQ code for inference. Its primary purpose is for research and benchmarking to evaluate the performance characteristics of KronQ 3-bit quantization on Llama-2-70b. For deployable low-bit models, users should refer to the W4A16-g128 or W2A16-g128 (packed) repositories. The license is derived from Llama-2-70b's original license.

Finetunes
1 models