donghyunli/Llama-2-7b-KronQ-W3A16-fake

TEXT GENERATIONConcurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:Jun 22, 2026License:llama2Architecture:Transformer Open Weights Featherless Exclusive Cold

The donghyunli/Llama-2-7b-KronQ-W3A16-fake is a Llama-2-7b model quantized to 3-bit weights using KronQ (Kronecker-factored Hessian quantization). This specific version is a fake-quantized fp16 checkpoint, primarily for accuracy reproduction and research, not for compression or speedup. It achieves a WikiText-2 perplexity of 5.84 and an average zero-shot accuracy of 64.10 across various benchmarks. This model is useful for evaluating the performance of 3-bit KronQ quantization on Llama-2-7b without requiring specialized low-bit inference engines.

Loading preview...

Overview

This model, donghyunli/Llama-2-7b-KronQ-W3A16-fake, is a version of the Llama-2-7b large language model that has been quantized to 3-bit weights using the KronQ (Kronecker-factored Hessian quantization) method. It is presented as a fake-quantized fp16 checkpoint, meaning the 3-bit weights are stored in fp16 format. This particular release is intended for perplexity and accuracy reproduction and research purposes, rather than for achieving compression or inference speedup. For deployable low-bit versions, users are directed to W4A16 / W2A16 (packed int4/int2) repositories.

Key Characteristics

  • Base Model: Llama-2-7b.
  • Quantization Method: KronQ (Kronecker-factored Hessian quantization) with 3-bit weights (W3A16).
  • Format: Fake-quantized fp16 checkpoint, allowing it to be loaded as a standard fp16 model using transformers without special KronQ code.
  • Performance:
    • WikiText-2 Perplexity: 5.84 (sequence length 2048).
    • Zero-shot Accuracy (Average): 64.10 across benchmarks including PIQA, ARC-E, ARC-C, HellaSwag, WinoGrande, BoolQ, and OBQA.
  • Quantization Recipe: Utilizes per-channel asymmetric W3, weight-only (a_bits=16), --alpha 0.25, bidirectional incoherence processing (BiIP), act_order, and raw H_G, calibrated on 128 WikiText-2 sequences.

Use Cases

This model is specifically designed for:

  • Accuracy Reproduction: Researchers can use this model to reproduce and verify the perplexity and zero-shot accuracy results of 3-bit KronQ quantization on Llama-2-7b.
  • Research and Development: It serves as a valuable tool for studying the impact of KronQ quantization on model performance without the complexities of low-bit inference environments.
  • Benchmarking: Useful for comparing the performance of different quantization techniques against a known baseline for 3-bit Llama-2-7b.

Finetunes
1 models