Xirui1208/memagent-readtwice-rewrite1024-repair-sft
Xirui1208/memagent-readtwice-rewrite1024-repair-sft is a 7.6 billion parameter Qwen2ForCausalLM-based model, fine-tuned for long-context repair and multi-type transfer tasks. It utilizes a ReadTwice protocol with a 1,024-token memory limit and supports a native context length of 32,768 tokens. This model excels in processing and repairing information within very long documents, demonstrating strong performance on RULER-HotpotQA at 448K context and LongBench evaluations. It is optimized for tasks requiring extensive document understanding and correction.
Loading preview...
Model Overview
Xirui1208/memagent-readtwice-rewrite1024-repair-sft is a 7.6 billion parameter language model built on the Qwen2ForCausalLM architecture. It represents a continuation of the ReadTwice SFT training, specifically initialized from a previous Rewrite-1024 SFT step. This model is designed for advanced long-context understanding and repair tasks, incorporating a unique memory operation that involves complete replacement with a 1,024-token memory limit.
Key Training Details
- Architecture: Qwen2ForCausalLM (7B class).
- Initialization: Warm restart from ReadTwice Rewrite-1024 SFT step286.
- Training Data: Includes long-horizon repair tasks (56K to 448K context) and various multi-type transfer tasks.
- Protocol: Trained under the
readall-repair-formal-v1continuation contract.
Performance Highlights
The model demonstrates strong capabilities in long-context scenarios, particularly under the sealed ReadTwice complete-rewrite protocol:
- RULER-HotpotQA (448K context): Achieved an Exact Match (EM) of 55.47 and an F1 score of 71.32.
- LongBench-v1: Mean score of 39.63 across five tasks.
- LongBench-v2: Accuracy of 31.81% with 100% boxed-format validity.
Runtime Considerations
While the model has a native 32,768-token position limit, evaluations at 224K/448K document lengths utilize a ReadTwice protocol processing 5,000-token sections. Inference for longer prompts leverages runtime YaRN scaling.