Jeesup/svd-safety-llama3_8b_instruct_remove_40_seed3_jbbmixsft
Jeesup/svd-safety-llama3_8b_instruct_remove_40_seed3_jbbmixsft is an 8 billion parameter Llama-3-Instruct model compressed using SVD-LLM, retaining 60% of its original dense parameters. This research artifact explores the impact of SVD compression on safety behavior and the effectiveness of component selection rules for repair. It is specifically designed for studying safety/utility trade-offs under compression, rather than serving as a general-purpose chat model.
Loading preview...
Model Overview
This model, svd-safety-llama3_8b_instruct_remove_40_seed3_jbbmixsft, is a research artifact derived from meta-llama/Meta-Llama-3-8B-Instruct. It has undergone SVD-LLM compression, resulting in 40% of its parameters being removed, leaving approximately 60% of the original dense parameters. A key characteristic of this specific variant is that it had a 0% parameter budget for restoring SVD components, with zero components actually restored or swapped out.
Research Focus
The primary purpose of this model is to serve as an experimental subject in a study investigating how SVD compression affects the safety behavior of large language models. The research aims to quantify the increase in attack-success rate (ASR) due to compression and test methods for recovery. This particular checkpoint represents one cell within a larger grid of experiments exploring different selection rules and budgets for SVD component restoration.
Measured Performance
Key metrics measured for this model include:
- AdvBench ASR (HarmBench judge): 0.0115
- StrongREJECT ASR (HarmBench judge): 0.0447
- Macro over-refusal (WildGuard): 0.4361
- WikiText-2 perplexity: 27.5101
Intended Use and Limitations
It is crucial to understand that this model is not intended as a deployable general-purpose assistant. Several arms of the research grid, including this one, are deliberately safety-degraded relative to the base Llama-3-8B-Instruct model. Users should treat this checkpoint as an experimental subject for evaluating safety/utility trade-offs under compression and conduct their own evaluations before drawing conclusions.