elastix-ai/HyperPrune-Qwen2.5-0.5B-2to4

Model Overview

elastix-ai/HyperPrune-Qwen2.5-0.5B-2to4 is a 0.5 billion parameter language model based on the Qwen2.5 architecture. It has been pruned to achieve 2:4 semi-structured sparsity using the HyperPrune method, a technique that learns semi-structured sparsity via shared and context-aware hypernetworks. This specific checkpoint is a reproduction run by Elastix as part of the BLADE sparsity-method comparison initiative.

Key Characteristics

• Sparsity: Achieves 2:4 semi-structured sparsity across all 24 decoder layers, resulting in approximately 51% overall decoder sparsity.
• Calibration Corpus: Utilizes DKYoon/SlimPajama-6B for calibration, differing from the original HyperPrune paper's allenai/c4.
• Pruning Strategy: Every decoder layer is pruned, unlike some HyperPrune configurations that leave initial layers dense. The hypernetwork determines the mask for the first 200 output rows of each projection, with remaining rows using the SparseGPT prior's mask.
• Technical Implementation: Loads with stock transformers library and is provided in bf16/fp16 safetensors format.

Performance and Limitations

While structurally correct, this 0.5B parameter model exhibits significant performance degradation due to the aggressive 2:4 sparsity. WikiText-2 word perplexity increases from 19.65 (dense) to 149.94, a 7.6x degradation. Greedy generation is noted to be repetitive and factually unreliable. This model is not recommended for deployment but serves as a valuable data point for research into sparsity methods at smaller model scales, demonstrating the limits of redundancy for pruning.

Use Cases

• Sparsity Research: Ideal for researchers comparing different sparsity techniques and their impact on very small language models.
• Methodology Study: Useful for understanding the practical application and effects of the HyperPrune method under specific conditions.