minjaechoi/qwen36-recovery-v4

Overview

minjaechoi/qwen36-recovery-v4 is a specialized Qwen36-based model (35.1B parameters) that integrates a ternary TWLA base with two sequential adapters: a v1 recovery adapter and a v4 clean-suffix delta. The primary goal of this model is to mitigate repetition in generated text, a common issue in large language models. It achieves this by distilling knowledge from a BF16 teacher model to generate 'clean suffixes' when repetitive patterns are detected in v1 rollouts.

Key Features and Training

• Architecture: Combines a frozen ternary TWLA base, a frozen v1 recovery adapter, and a new rank-64 residual delta applied to sensitive layers (24, 28, 30, 35, 38, 39) for v4. Only 1,572,864 parameters are trainable in the v4 delta.
• Repetition Recovery: The v4 delta was trained on 237 items from v1 rollouts with an 8-gram repetition ratio above 0.12. A BF16 teacher model regenerated suffixes from 256/512/1,024 token points, resulting in 102 accepted clean suffixes.
• Distillation Loss: Training used a loss function combining KL divergence from the BF16 teacher for clean suffixes and KL divergence from the frozen v1 distribution for normal rollout tokens. It explicitly avoided EOS cross-entropy and repetition unlikelihood terms.
• Data Lineage: Both v1 and v4 adapters were trained without GPQA training records. However, the underlying ternary base's quantization calibration did use 128 2,048-token chunks from GPQA-derived reasoning traces, meaning the full stack is not entirely GPQA-unexposed.

Usage

To use the model, both the v1 and v4 adapters must be loaded sequentially on top of the ternary TWLA base. An inference.py script is provided to facilitate this process, ensuring the correct adapter paths are used during inference.