elastix-ai/HyperPrune-Llama-3.1-8B-4to8

HyperPrune-Llama-3.1-8B-4to8 Overview

This model is a Llama-3.1-8B variant, developed by elastix-ai, that has undergone 4:8 semi-structured sparsity pruning using the HyperPrune method. It achieves a 50% sparsity across all 32 decoder layers, meaning only 4 out of every 8 weights are retained, leading to a more compact and efficient model.

Key Characteristics & Performance

• Sparsity: Strict 4:8 semi-structured sparsity applied to all decoder layers, resulting in 50% overall decoder sparsity.
• Perplexity: Achieves a WikiText-2 word perplexity of 16.39 (using BLADE's lm-eval-harness protocol), a notable improvement over 2:4 sparsity models (22.27 word-perplexity).
• Efficiency: The pruning process significantly reduces the model's computational footprint, with training wall-clock time of just 20.4 minutes and peak GPU usage of 10.88 GB during cascade fine-tuning.
• Reproducibility: This checkpoint is a reproduction run by Elastix as part of the BLADE sparsity-method comparison, using DKYoon/SlimPajama-6B as the calibration corpus.
• Compatibility: Loads directly with stock transformers library, making it easy to integrate into existing workflows.

What Makes It Different

Unlike standard HyperPrune configurations, this model prunes every decoder layer to achieve a true 4:8 sparsity, rather than leaving some layers dense. It also utilizes an N:M generalization of HyperPrune, allowing for flexible sparsity patterns beyond the default 2:4. This specific implementation demonstrates the effectiveness of 4:8 sparsity in significantly improving perplexity compared to 2:4 patterns for Llama-3.1-8B.

Ideal Use Cases

This model is particularly well-suited for applications requiring:

• Resource-constrained environments: Where memory and computational resources are limited.
• Faster inference: The reduced parameter count can lead to quicker response times.
• Efficient deployment: For edge devices or large-scale deployments where model size is a critical factor.