Jeesup/svd-safety-l3_basis_jbbsftx1_remove50
Jeesup/svd-safety-l3_basis_jbbsftx1_remove50 is an 8 billion parameter Llama-3-8B-Instruct checkpoint, compressed using Basis Sharing to 50% of its original dense parameters. This model is a research artifact designed to study how SVD compression impacts safety behavior and to test repair mechanisms. It is specifically intended for evaluating safety/utility trade-offs under compression, not as a general-purpose chat model.
Loading preview...
Model Overview
This model, svd-safety-l3_basis_jbbsftx1_remove50, is an 8 billion parameter variant of meta-llama/Meta-Llama-3-8B-Instruct. It has undergone significant compression using a technique called Basis Sharing (ICLR 2025), reducing its parameter count to 50% of the original dense model. The compression process involved sharing bases over groups of two adjacent layers, followed by a coefficient-only LoRA fine-tune (r=8) to recover performance while maintaining the compressed parameter budget.
Key Characteristics
- Base Model: Meta-Llama-3-8B-Instruct.
- Compression Method: Basis Sharing, achieving 50% parameter reduction.
- Recovery: LoRA fine-tuning on per-layer coefficients (bases frozen) for 2 epochs using the alpaca-cleaned dataset.
- Context Length: 8192 tokens.
Research Focus and Limitations
This checkpoint is a research artifact from a study investigating the impact of SVD compression on safety behavior and the effectiveness of various repair strategies. It is explicitly noted that several arms of this research grid are deliberately safety-degraded compared to the original Llama-3-8B-Instruct. The model's measured metrics include an AdvBench ASR of 0.2250, StrongREJECT ASR of 0.1821, Macro over-refusal of 0.2192, and a WikiText-2 perplexity of 25.1701.
Intended Use
This model is not intended for general-purpose chat applications. Its primary purpose is to serve as an experimental subject for measuring safety/utility trade-offs under compression. Users are advised to evaluate it thoroughly before drawing conclusions, as its safety behavior may be intentionally compromised for research purposes.