donghyunli/Meta-Llama-3-8B-KronQ-W3A16-fake

TEXT GENERATIONConcurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:8kTool Calling:SupportedPublished:Jun 22, 2026License:llama3Architecture:Transformer Featherless Exclusive Cold

donghyunli/Meta-Llama-3-8B-KronQ-W3A16-fake is a 3-bit weight-quantized version of Meta-Llama-3-8B using KronQ (Kronecker-factored Hessian quantization). This specific checkpoint stores the 3-bit weights in fp16 format, making it suitable for perplexity and accuracy reproduction rather than deployment due to its lack of compression. It achieves a perplexity of 7.09 on WikiText-2 and an average zero-shot accuracy of 67.32% across various benchmarks, demonstrating its performance in a fake-quantized setup.

Loading preview...

Overview

This model, donghyunli/Meta-Llama-3-8B-KronQ-W3A16-fake, is a 3-bit weight-quantized version of the Meta-Llama-3-8B base model. It utilizes the KronQ (Kronecker-factored Hessian quantization) method to achieve 3-bit weights. Notably, this particular checkpoint is a "fake-quant fp16" version, meaning the 3-bit weights are stored in fp16 format. This results in no compression or speedup compared to a full precision model, and its primary purpose is for perplexity and accuracy reproduction rather than deployable low-bit inference.

Key Characteristics

  • Base Model: Meta-Llama-3-8B.
  • Quantization Method: KronQ (Kronecker-factored Hessian quantization).
  • Weight Precision: 3-bit weights (W3A16).
  • Format: Fake-quantized fp16, meaning weights are stored in fp16 for compatibility and accuracy evaluation.
  • Perplexity: Achieves 7.09 on WikiText-2 (sequence length 2048).
  • Zero-shot Accuracy: Demonstrates an average of 67.32% across benchmarks like PIQA, ARC-E/C, HellaSwag, WinoGrande, BoolQ, and OBQA.
  • Usage: Loads as a standard fp16 model using transformers, requiring no special KronQ code for inference.

Intended Use

This model is specifically designed for researchers and developers interested in:

  • Reproducing perplexity and accuracy results of KronQ 3-bit quantization on Meta-Llama-3-8B.
  • Evaluating the impact of 3-bit quantization on model performance without the complexities of low-bit deployment.

For deployable low-bit models with actual compression and speedup, users should refer to packed int4 versions (e.g., W4A16) of KronQ-quantized models.

Finetunes
1 models