donghyunli/Meta-Llama-3-70B-KronQ-W3A16-g128-fake

TEXT GENERATIONConcurrent Unit Cost:4Model Size:70BQuant:FP8Context Size:8kTool Calling:SupportedPublished:Jun 24, 2026License:llama3Architecture:Transformer Featherless Exclusive Cold

donghyunli/Meta-Llama-3-70B-KronQ-W3A16-g128-fake is a 70 billion parameter Meta-Llama-3 derivative model, featuring 3-bit weight quantization (group 128) using the KronQ method. This specific version is a fake-quantized fp16 checkpoint, designed for accuracy reproduction rather than deployable low-bit inference, maintaining the same size as a bf16 model. It achieves a perplexity of 4.197 on WikiText-2 and an average zero-shot accuracy of 73.59% across various benchmarks, making it suitable for evaluating the performance of KronQ quantization on large language models.

Loading preview...

Model Overview

This model, donghyunli/Meta-Llama-3-70B-KronQ-W3A16-g128-fake, is a 70 billion parameter variant of the Meta-Llama-3 architecture. It incorporates KronQ quantization, specifically using 3-bit weights with a group size of 128. This particular release is a fake-quantized fp16 checkpoint, meaning the 3-bit weights are stored in fp16 format. This design choice prioritizes the reproduction of perplexity and accuracy results, rather than providing a packed, deployable low-bit model. For deployable low-bit versions, users are directed to W4A16-g128 or W2A16-g128 packed repositories.

Key Characteristics

  • Quantization Method: KronQ, with asymmetric W3 (3-bit weights) and group-128 quantization.
  • Format: Fake-quantized fp16 checkpoint, which means it loads as a standard fp16 model without requiring specific KronQ code for inference.
  • Performance:
    • Perplexity (WikiText-2, seqlen 2048): 4.197
    • Zero-shot Accuracy (Average): 73.59% across benchmarks including PIQA, ARC-E, ARC-C, HellaSwag, WinoGrande, BoolQ, and OBQA.

Usage

This model can be loaded directly using transformers.AutoModelForCausalLM with torch_dtype="float16", as it behaves like a standard fp16 model. It is ideal for researchers and developers looking to evaluate the impact of KronQ 3-bit quantization on the Meta-Llama-3-70B model's performance metrics, particularly perplexity and zero-shot accuracy, before deploying truly low-bit versions.

Finetunes
1 models