Jeesup/svd-safety-l2_basis_remove30
Jeesup/svd-safety-l2_basis_remove30 is a Llama-2-7b-chat checkpoint compressed using Basis Sharing, reducing its parameters to 70% of the original 7 billion. This model is a research artifact designed to study how SVD compression impacts safety behavior and the effectiveness of recovery methods. It is specifically configured with 4096 tokens context length and is not intended as a general-purpose chat model, but rather for experimental evaluation of safety/utility trade-offs under compression.
Loading preview...
Overview
Jeesup/svd-safety-l2_basis_remove30 is a research artifact derived from meta-llama/Llama-2-7b-chat-hf. It has been compressed using Basis Sharing (ICLR 2025), a technique that shares bases over groups of two adjacent layers, resulting in a model with 70.0% of its original 7 billion dense parameters removed. The model was then recovered through a coefficient-only LoRA fine-tune (r=8, 2 epochs, lr 0.0001, batch 64, alpaca-cleaned) while maintaining the compressed parameter budget and frozen shared bases.
Key Characteristics
- Compression Method: Basis Sharing, removing 30% of parameters.
- Base Model:
meta-llama/Llama-2-7b-chat-hf. - Recovery: LoRA fine-tuning on per-layer coefficients.
- Context Length: 4096 tokens.
Purpose and Limitations
This model is not a general-purpose chat model. Its primary purpose is to serve as an experimental subject in a study quantifying how SVD compression affects safety behavior and the efficacy of various recovery strategies. The README explicitly states that several arms in this research grid are deliberately safety-degraded compared to the original Llama-2-7b-chat. Users should treat this checkpoint as an experimental subject and conduct their own evaluations before drawing conclusions or deploying it.