elastix-ai/HyperPrune-Qwen2.5-0.5B-4to8

Model Overview

This model, elastix-ai/HyperPrune-Qwen2.5-0.5B-4to8, is a 0.5 billion parameter variant of the Qwen2.5 architecture. It has been subjected to 4:8 semi-structured sparsity using the HyperPrune method, a technique for learning semi-structured sparsity in LLMs. This specific checkpoint is a reproduction run by Elastix, part of a broader comparison of sparsity methods.

Key Characteristics

• Sparsity: Achieves 4:8 semi-structured sparsity across all 24 decoder layers, meaning for every 8 weights, 4 are kept.
• Pruning Method: Utilizes HyperPrune, which involves a hypernetwork to determine the sparsity mask, with a specific configuration where the hypernet decides the mask for the first 200 output rows of each projection.
• Calibration Corpus: Differs from the original HyperPrune paper by using DKYoon/SlimPajama-6B (validation split) as the calibration corpus.
• Performance Note: Notably, for this 0.5B parameter model, 4:8 sparsity results in significantly worse WikiText-2 word perplexity (282.87) compared to 2:4 sparsity (149.94). This suggests that for smaller models, the increased complexity of a 70-way classification for 4:8 sparsity might lead to an under-fitted hypernet, making it less effective than stricter sparsity.

Intended Use Case

This model is primarily intended as a research data point for understanding the scaling behavior and limitations of the HyperPrune sparsity method on smaller language models. It demonstrates that 4:8 sparsity, while beneficial for larger models, can degrade performance on smaller architectures. It is not recommended for general-purpose applications due to its observed performance degradation and degenerate greedy generation.