zhongweixie/inplace-ttt-qwen3-4b-32k-baseline
The zhongweixie/inplace-ttt-qwen3-4b-32k-baseline is a 4 billion parameter causal language model based on the Qwen3-4B (Qwen2.5-4B architecture) developed by zhongweixie. Trained with a standard method on a 32k token context length for 6000 steps, this model serves as a baseline for research into In-Place Test-Time Training (TTT). It is designed for comparison against TTT-enhanced versions to evaluate the impact of the In-Place TTT method, making it suitable for research and development in long-context language modeling.
Loading preview...
Model Overview
This model, zhongweixie/inplace-ttt-qwen3-4b-32k-baseline, is a 4 billion parameter causal language model built on the Qwen3-4B (Qwen2.5-4B architecture). It was trained using a standard method (without In-Place Test-Time Training) for 6000 steps, supporting a 32,768 token context length. Its primary purpose is to serve as a baseline for the In-Place TTT research project.
Key Characteristics
- Base Architecture: Qwen3-4B (Qwen2.5-4B)
- Context Length: 32k tokens
- Training: Standard training (no TTT) for 6000 steps on the ProLong 32k dataset.
- Model Size: 8.3GB, provided in HuggingFace Transformers (safetensors) format.
Differentiation and Use Cases
This model is specifically designed for comparative analysis within the In-Place TTT research. It shares the same base architecture, training data, steps, and hyperparameters as its In-Place TTT counterparts, with the sole difference being the absence of In-Place TTT. Developers and researchers can use this model to:
- Evaluate the impact of In-Place TTT by comparing its performance against models like zhongweixie/inplace-ttt-qwen3-4b-32k-ttt.
- Understand the foundational performance of the Qwen3-4B architecture on long contexts without specialized training enhancements.
For more details on the research, refer to the In-Place TTT GitHub Repository and the associated paper.