donghyunli/Meta-Llama-3-70B-KronQ-W3A16-fake

TEXT GENERATIONConcurrent Unit Cost:4Model Size:70BQuant:FP8Context Size:8kTool Calling:SupportedPublished:Jun 23, 2026License:llama3Architecture:Transformer Featherless Exclusive Cold

The donghyunli/Meta-Llama-3-70B-KronQ-W3A16-fake is a 70 billion parameter language model, derived from Meta-Llama-3-70B, that has been quantized to 3-bit weights using the KronQ method. This specific version is a fake-quantized fp16 checkpoint, designed for perplexity and accuracy reproduction rather than deployable low-bit inference. It achieves a perplexity of 4.42 on WikiText-2 and an average zero-shot accuracy of 73.98% across various benchmarks, making it suitable for evaluating the performance of 3-bit quantization on large language models.

Loading preview...

Overview

This model, donghyunli/Meta-Llama-3-70B-KronQ-W3A16-fake, is a 70 billion parameter variant of Meta-Llama-3-70B that has undergone 3-bit weight quantization using the KronQ method. It is presented as a fake-quantized fp16 checkpoint, meaning its 3-bit weights are stored in fp16 format. This configuration is specifically intended for reproducing perplexity and accuracy results of 3-bit quantization, rather than for direct low-bit deployment. For deployable low-bit versions, users are directed to W4A16 / W2A16 (packed) repositories.

Key Characteristics

  • Base Model: Meta-Llama-3-70B.
  • Quantization: KronQ W3A16 (3-bit weights, 16-bit activations).
  • Format: Fake-quantized fp16 checkpoint, same size as bf16.
  • Quantization Recipe: Utilizes per-channel asymmetric W3, weight-only (a_bits=16), --alpha 0.25, BiIP, act_order, and raw H_G.

Performance Metrics

Evaluated on WikiText-2 with a sequence length of 2048, the model demonstrates:

  • Perplexity: 4.42
  • Zero-shot Accuracy: An average of 73.98% across benchmarks including PIQA (83.51), ARC-E (80.35), ARC-C (58.87), HellaSwag (83.11), WinoGrande (79.48), BoolQ (85.35), and OBQA (47.20).

Usage

This model loads as a standard fp16 model using the transformers library, requiring no specific KronQ code for inference. It is ideal for researchers and developers looking to evaluate the impact of 3-bit quantization on large language models without needing specialized low-bit inference engines.

Finetunes
1 models