redashes/Qwen3.8-27B-BF16-SSMFIX
redashes/Qwen3.8-27B-BF16-SSMFIX is a 27 billion parameter Qwen3.8 model that has been experimentally repaired to address a silent degradation in long-context generation. This version applies per-layer alpha-scaling to specific conv1d tensors, fixing scale-drift issues identified by community research. It aims to improve generative quality and reduce hallucination, particularly in long-context scenarios, by trading off some closed-book knowledge and conversational quality.
Loading preview...
Overview
This model, redashes/Qwen3.8-27B-BF16-SSMFIX, is an experimental, 27 billion parameter Qwen3.8 variant. It addresses a known issue in the official Qwen3.8-27B weights where 8 of the last layers exhibit significantly inflated conv1d standard deviations, leading to silent degradation in long-context generation, repetition loops, and "philosophizing" drift. This release applies per-layer alpha-scaling to these anomalous tensors, following the methodology developed by LuffyTheFox (Sig-ScaleSync) and independently re-implemented by FGDumitru.
Key Capabilities & Fixes
- Repaired SSM Scale-Drift: Fixes anomalous
linear_attn.conv1d.weighttensors in 8 specific layers (52, 53, 56, 57, 58, 60, 61, 62) that cause recurrent state saturation and collapse in long contexts. - Improved Generative Quality: Evaluation shows a significant increase in TruthfulQA generation scores (+6~8pp), indicating a strong reduction in hallucination.
- Community Cross-Validated: The diagnosis of anomalous layers and the applied alpha-scaling range (0.481β0.653) align with independent findings from LuffyTheFox and FGDumitru.
- Controlled Benchmarking: This release includes a full controlled benchmark battery (GSM8K, CMMLU, TruthfulQA, IFEval, MT-Bench) against the official baseline, providing quantitative proof of the fix's effects.
Trade-offs
While improving generation, the repair introduces minor regressions in other areas:
- Slightly Reduced Knowledge: CMMLU scores decreased by 1.8 percentage points.
- Minor Conversational Quality Drop: MT-Bench average score decreased by 0.19 points, primarily affecting reasoning, writing, and math categories.
This model serves as a measurable record of a community verification experiment, offering a specific generative-quality profile for users who prioritize hallucination reduction in long-context tasks.