donghyunli/Llama-2-7b-KronQ-W3A16-g128-fake

TEXT GENERATIONConcurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:Jun 24, 2026License:llama2Architecture:Transformer Open Weights Featherless Exclusive Cold

The donghyunli/Llama-2-7b-KronQ-W3A16-g128-fake model is a 7-billion parameter Llama-2 derivative, quantized to 3-bit weights (W3A16) using the KronQ method with group-128 quantization. This specific version is a fake-quantized fp16 checkpoint, designed for accuracy reproduction and evaluation rather than deployable low-bit inference. It achieves a perplexity of 5.773 on WikiText-2 and an average zero-shot accuracy of 64.22% across various benchmarks, making it suitable for research and development in efficient model quantization.

Loading preview...

Model Overview

This model, donghyunli/Llama-2-7b-KronQ-W3A16-g128-fake, is a 7-billion parameter variant of the Llama-2 architecture, developed by donghyunli. It incorporates KronQ quantization to represent weights in 3-bit (W3A16) with a group size of 128. Notably, this release is a fake-quantized fp16 checkpoint, meaning the 3-bit weights are stored in fp16 format. This design choice prioritizes perplexity and accuracy reproduction for research purposes, rather than providing a packed, deployable low-bit model.

Key Characteristics

  • Architecture: Llama-2-7b base model.
  • Quantization: KronQ method, featuring 3-bit weights (W3) and 16-bit activations (A16) with group-128 quantization.
  • Format: Fake-quantized fp16 checkpoint, allowing for direct loading as a standard fp16 model using transformers.
  • Performance: Achieves a perplexity of 5.773 on WikiText-2 (seqlen 2048). Zero-shot accuracy benchmarks include 77.48% on PIQA, 72.01% on ARC-E, and an average of 64.22% across multiple tasks (PIQA, ARC-E, ARC-C, HellaSwag, WinoGrande, BoolQ, OBQA).
  • Recipe Details: Utilizes group-128 asymmetric W3, weight-only quantization with --alpha 0.25, --act_order, BiIP, and raw H_G.

Intended Use Cases

This model is particularly well-suited for:

  • Research and Development: Evaluating the performance and accuracy of KronQ 3-bit quantization without the complexities of custom low-bit inference engines.
  • Benchmarking: Reproducing and comparing perplexity and zero-shot accuracy results against other quantization techniques.
  • Understanding Quantization Effects: Analyzing the impact of 3-bit weight quantization on a Llama-2 base model's performance.

For deployable low-bit versions, users should refer to the W4A16-g128 or W2A16-g128 (packed) repositories mentioned in the original KronQ project.

Finetunes
1 models