Xirui1208/readall-readtwice-14b-stage2-localstep15-globalstep30-20260926

Overview

This model, Xirui1208/readall-readtwice-14b-stage2-localstep15-globalstep30-20260926, is a 14.8 billion parameter causal language model built upon the Qwen2.5-14B architecture. It represents a stage-2 reinforcement learning (RL) checkpoint, specifically local step15 in its current training round, accumulating to global step30 from its parent model, ReadAll14B step15. The model is optimized for long-context document question answering (DocQA) and high-quality question answering (HQA).

Key Capabilities & Features

• ReadTwice Protocol: Utilizes a unique SKIM → sequential complete-memory UPDATE → FINAL protocol for processing long documents, as detailed in training_code.tar.gz.
• Long Context Handling: Designed to handle long documents by losslessly chunking them into 5000-token sections and processing them serially with bounded memory. It supports a native context window of 32768 tokens.
• Reinforcement Learning: Trained with a specific RL setup, including a constant learning rate of 1e-6, AdamW optimizer with weight decay 0.01, gradient clipping, PPO clipping, and KL divergence against an initial SFT A230 model.
• Training Details: This checkpoint reflects 60 optimizer updates and 960 training problems, with phase weights for SKIM/UPDATE/FINAL set at 0.10/0.45/0.45.

Important Considerations

• Validation Status: This local-step15 checkpoint has not been benchmark-evaluated. Previous local-step10 HQA results are not attributed to this specific model.
• Usage Protocol: To reproduce the intended ReadTwice workflow, users must implement the SKIM → UPDATE → FINAL protocol from training_code.tar.gz; a direct single-prompt call will not yield the reported results.
• Training Checkpoint: The repository includes the exact FP32 four-rank FSDP model, Adam optimizer, scheduler, and RNG states, allowing for precise continuation of training.