ikimyaii/llama-7b-cws-40pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

ikimyaii/llama-7b-cws-40pct is a 7 billion parameter language model based on OpenLLaMA-7B, pruned to 40% weight sparsity using the Correlation-aware Weight Sparsity (CWS) method. This model employs a one-shot, post-training pruning technique that extends SparseGPT's Optimal Brain Surgeon (OBS) with a full-Hessian selection criterion to account for cancellation groups. It is a research checkpoint designed for studying pruning techniques, offering insights into efficient LLM inference by reducing model size while maintaining performance, as demonstrated by its WikiText-2 PPL of 9.97 at 40% sparsity.

Loading preview...

ikimyaii/llama-7b-cws-40pct: A Sparsified OpenLLaMA-7B Model

This model is a 7 billion parameter version of OpenLLaMA-7B, specifically engineered for 40% weight sparsity using the novel Correlation-aware Weight Sparsity (CWS) method. Developed by ikimyaii, this is a research checkpoint focused on exploring efficient LLM inference through advanced pruning techniques.

Key Capabilities & Features

  • Advanced Pruning: Utilizes CWS, a one-shot, post-training pruning method that enhances SparseGPT's Optimal Brain Surgeon (OBS) by incorporating a full-Hessian selection criterion. This allows for more effective identification and removal of correlated weights, improving pruning accuracy.
  • Semi-Structured Sparsity: Achieves 39.84% sparsity by removing the same fraction of weights from every output row in nn.Linear layers across the 32 decoder layers and lm_head.
  • Weight Correction: Employs second-order OBS for precise weight correction on the surviving weights, minimizing performance degradation after pruning.
  • Research Focus: Primarily intended for studying the impact and effectiveness of pruning methods, offering a valuable resource for researchers in model compression.

Evaluation Highlights

  • WikiText-2 PPL: Achieves a WikiText-2 Perplexity (PPL) of 9.97 at 40% sparsity, compared to the dense model's 8.64. This demonstrates competitive performance against other pruning methods like SparseGPT (10.24) and Wanda (9.76) at the same sparsity level.

Good For

  • Research on Model Pruning: Ideal for researchers and developers investigating advanced weight sparsity techniques and their impact on large language models.
  • Understanding CWS: Provides a practical example of the CWS method in action, allowing for direct experimentation and analysis of its effectiveness.
  • Exploring Efficient LLM Inference: Useful for those interested in the potential of sparse models for reducing computational requirements and memory footprint, although the weights are currently stored densely.

Note: This model is a research checkpoint and is not instruction-tuned or safety-tuned. It inherits the biases of its base model, OpenLLaMA-7B.

Finetunes
1 models