elastix-ai/HyperPrune-Llama-2-7B-4to8

HyperPrune-Llama-2-7B-4to8: A Sparsity Exploration

This model, developed by elastix-ai, is a 7 billion parameter Llama-2 variant that has been semi-structurally pruned to 2:4 sparsity using the HyperPrune method. It represents a reproduction run within the BLADE sparsity-method comparison, focusing on achieving significant model compression.

Key Characteristics & Differences

• Sparsity: Achieves an overall decoder sparsity of 50.05%, with all 32 decoder layers pruned, a stricter setting than the original HyperPrune paper's default.
• Pruning Method: Utilizes the HyperPrune technique (Sun & Sakuma, ICLR 2026), which employs a hypernetwork to learn semi-structured sparsity.
• Calibration Corpus: Differs from the paper's original recipe by using DKYoon/SlimPajama-6B (validation split) as the calibration corpus, aligning with BLADE's protocol.
• Hypernet Contribution: Approximately 3.5% of the sparsity mask was determined by the hypernet, with the remainder derived from a SparseGPT prior, reflecting HyperPrune's default configuration.
• Performance: Demonstrates a WikiText-2 PPL of 10.64 (token-level) and 21.65 (word-level via lm-eval-harness), providing a benchmark for pruned model performance.
• Efficiency: Achieved pruning in 22.1 minutes wall-clock time with a peak GPU memory usage of 8.82 GB on an NVIDIA RTX PRO 6000 Blackwell GPU.

Use Cases

This model is particularly relevant for researchers and developers interested in:

• Efficient LLM Deployment: Exploring the trade-offs between model size, inference speed, and performance through advanced pruning.
• Sparsity Research: Benchmarking and comparing different sparsity techniques, especially semi-structured pruning methods.
• Resource-Constrained Environments: Investigating the potential for deploying Llama-2-7B-class models with reduced computational and memory footprints.