Tfloow/Llama-3.2-1B-adpq-4bit-sim

Hugging Face
TEXT GENERATIONConcurrency Cost:1Model Size:1BQuant:BF16Ctx Length:32kLicense:apache-2.0Architecture:Transformer Open Weights Warm

Tfloow/Llama-3.2-1B-adpq-4bit-sim is a 1 billion parameter Llama-3.2 model, developed by Tfloow, that has been compressed using 4-bit ADPQ (Adaptive Quantization with data-free calibration). This model is designed to significantly reduce VRAM usage and increase inference speed while largely preserving the original model's performance. It is particularly suited for resource-constrained environments where efficient deployment of large language models is critical.

Loading preview...

Overview

This model, Tfloow/Llama-3.2-1B-adpq-4bit-sim, is a 4-bit quantized version of the meta-llama/Llama-3.2-1B base model, developed by Tfloow as part of a master's thesis. It utilizes the Adaptive Quantization (ADPQ) method, which includes data-free calibration, to achieve significant compression. The primary goal of this quantization is to reduce VRAM consumption and accelerate inference, making it more accessible for deployment in environments with limited hardware resources.

Key Capabilities

  • 4-bit Quantization: Achieves substantial memory savings and faster inference speeds compared to the original full-precision model.
  • ADPQ Method: Employs Adaptive Quantization, a technique designed to maintain performance fidelity during compression.
  • Simulated Quantization: The model is a simulated 4-bit version, indicating careful calibration to balance size and performance.
  • Llama-3.2 Base: Built upon the Llama-3.2 architecture, inheriting its general language understanding and generation capabilities.

Performance Considerations

While quantized models inherently involve some performance trade-offs, the ADPQ method aims to minimize these. Perplexity (PPL) benchmarks provided in the original README show that ADPQ quantization for Llama-3.2-1B results in a PPL of 6.9491 (AdpQ 9%) and 7.0380 (AdpQ 2%) compared to the baseline of 6.5546, indicating a controlled increase in perplexity for significant resource savings.

Good for

  • Resource-constrained deployments: Ideal for applications where VRAM is limited, such as edge devices or cost-sensitive cloud environments.
  • Faster inference: Suitable for use cases requiring quicker response times from the language model.
  • Experimentation with quantization: Provides a practical example of ADPQ quantization for developers interested in model compression techniques.

Finetunes
3 models