donghyunli/Llama-2-7b-KronQ-W3A16-g128-fake

TEXT GENERATIONConcurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:Jun 24, 2026License:llama2Architecture:Transformer Open Weights Featherless Exclusive Cold

donghyunli/Llama-2-7b-KronQ-W3A16-g128-fake is a 7 billion parameter Llama-2 model quantized to 3-bit weights (W3A16) using the KronQ method with group-128 quantization. This specific version is a fake-quantized fp16 checkpoint, designed for accuracy reproduction and evaluation rather than deployable low-bit inference. It achieves a perplexity of 5.772 on WikiText-2 and an average zero-shot accuracy of 64.55% across various benchmarks, making it suitable for research and development in efficient LLM quantization.

Loading preview...

KronQ Llama-2-7b W3A16 g128 (Fake-Quant)

This model is a 3-bit weight quantized version of the Llama-2-7b large language model, utilizing the KronQ quantization method with group-128 quantization. It is presented as a fake-quantized fp16 checkpoint, meaning the 3-bit weights are stored in fp16 format for accurate reproduction of perplexity and zero-shot accuracy, rather than for direct low-bit deployment. This allows researchers to evaluate the performance of KronQ's 3-bit quantization without needing specialized low-bit inference engines.

Key Characteristics & Performance

  • Quantization: 3-bit weights (W3A16) with group-128 using the KronQ method.
  • Format: Fake-quantized fp16 checkpoint, maintaining the same size as bf16 for evaluation purposes.
  • Perplexity: Achieves a perplexity of 5.772 on the WikiText-2 dataset (sequence length 2048).
  • Zero-shot Accuracy: Demonstrates an average zero-shot accuracy of 64.55% across benchmarks including PIQA, ARC-E, ARC-C, HellaSwag, WinoGrande, BoolQ, and OBQA.
  • Loading: Loads as a standard fp16 model using transformers.AutoModelForCausalLM without requiring specific KronQ code.

Use Cases

This model is particularly useful for:

  • Research and Development: Evaluating the impact and performance of 3-bit weight quantization techniques like KronQ.
  • Accuracy Reproduction: Verifying the reported perplexity and zero-shot accuracy of KronQ's W3A16 g128 quantization.
  • Comparative Analysis: Benchmarking against other quantization methods or full-precision models to understand efficiency trade-offs.

Finetunes
1 models