donghyunli/Llama-2-13b-KronQ-W3A16-fake

TEXT GENERATIONConcurrent Unit Cost:1Model Size:13BQuant:FP8Context Size:4kPublished:Jun 22, 2026License:llama2Architecture:Transformer Open Weights Featherless Exclusive Cold

The donghyunli/Llama-2-13b-KronQ-W3A16-fake model is a 13 billion parameter Llama-2 derivative, quantized to 3-bit weights using the KronQ (Kronecker-factored Hessian quantization) method. This specific release is a fake-quantized fp16 checkpoint, meaning it stores 3-bit weights in fp16 format for accuracy reproduction and perplexity evaluation, rather than for compression or speedup. It achieves a perplexity of 5.18 on WikiText-2 and an average zero-shot accuracy of 67.34 across various benchmarks, serving as a reference for KronQ's quantization performance.

Loading preview...

Overview

This model, donghyunli/Llama-2-13b-KronQ-W3A16-fake, is a 13 billion parameter variant of the Llama-2 architecture. It has been quantized to 3-bit weights using the KronQ (Kronecker-factored Hessian quantization) method, as detailed in the associated arXiv paper. This particular release is a fake-quantized fp16 checkpoint, meaning the 3-bit weights are stored in fp16 format. This design choice is specifically for perplexity and accuracy reproduction, not for achieving compression or speedup. For deployable low-bit versions, users are directed to W4A16 or W2A16 packed int4/int2 repositories.

Key Characteristics

  • Quantization Method: Utilizes KronQ (Kronecker-factored Hessian quantization) for 3-bit weight quantization.
  • Format: Fake-quantized fp16 checkpoint, storing 3-bit weights in fp16.
  • Purpose: Primarily for PPL / accuracy reproduction and evaluation of KronQ's quantization performance.
  • Performance: Achieves a perplexity of 5.18 on WikiText-2 (seqlen 2048) and an average zero-shot accuracy of 67.34 across benchmarks like PIQA, ARC-E/C, HellaSwag, WinoGrande, BoolQ, and OBQA.
  • Usage: Loads as a standard fp16 model using transformers, requiring no special KronQ code for inference.

When to Use This Model

This model is ideal for:

  • Researchers and developers interested in evaluating the accuracy and perplexity of KronQ's 3-bit quantization without the overhead of specialized low-bit inference engines.
  • Reproducing the results presented in the KronQ paper.
  • Understanding the impact of 3-bit weight quantization on a Llama-2-13b model's performance metrics.

Finetunes
1 models