minjaechoi/qwen36-recovery-v1

Qwen36 TWLA Recovery v1: An Adapter-Based Performance Recovery Model

This model, developed by minjaechoi, is a 35.1 billion parameter Qwen-based language model that utilizes a v1 low-rank recovery adapter applied to a routed-expert ternary TWLA (Ternary Weight and Activation) base. The base model incorporates W1.58A16 TWLA quantization for gate_up_proj and down_proj layers, with other parameters in BF16.

Key Features and Training:

• Adapter Architecture: The model employs a rank-64 residual recovery path across 40 MoE layers. Only the low-rank A/B matrices and gates are trained, keeping the ternary base fixed. This results in a small number of trainable parameters (10,567,720).
• Training Data: The v1 adapter was trained on a GPQA-free corpus, including datasets like OpenR1 Math Verified, Nemotron Math v3 Verified, Nemotron PTD Code, and Nemotron Science. The training involved 1,024 prompts and 64 validation prompts, with on-policy rollouts generated from a ternary student model.
• Loss Function: Training optimizes a unique loss function combining weighted_topk64_plus_tail_KL, margin_loss, and gate_regularizer, using the BF16 original model as a teacher.
• Contamination Note: While the adapter training data is GPQA-free, the underlying base checkpoint's TWLA calibration used reasoning traces from GPQA, meaning the model is not entirely contamination-free regarding GPQA.

Use Cases:

This model is suitable for applications requiring strong performance in mathematical, scientific, and coding reasoning, especially where a balance between model size/efficiency (due to quantization) and performance recovery is desired. Developers can integrate the adapter during inference to leverage the v1 model's enhanced capabilities.