Estwld/Qwen3.5-9B-base-rednote-200K
Estwld/Qwen3.5-9B-base-rednote-200K is a 9-billion parameter Qwen3.5-based multimodal language model, fine-tuned from the base model for research and further customization. It features an exceptionally long context window of 200,000 tokens, enabling processing of extensive inputs. This model is designed for tasks requiring deep contextual understanding over very long sequences and serves as a foundation for custom instruction-tuned applications.
Loading preview...
Estwld/Qwen3.5-9B-base-rednote-200K Overview
This model is a 9-billion parameter variant of the Qwen3.5 architecture, specifically fine-tuned from the base model (Qwen/Qwen3.5-9B-Base) rather than an instruction-tuned version. Its primary distinguishing feature is an extended context length of 200,000 tokens, significantly surpassing typical LLM context windows. The fine-tuning process involved a full parameter update on the Zchu/REDSearcher_SFT_10K dataset.
Key Characteristics & Training Details
- Base Model Origin: Fine-tuned from Qwen3.5-9B-Base, making it suitable for custom instruction tuning or research without pre-existing instruction biases.
- Exceptional Context Window: Supports a massive 200,000 token context length, ideal for processing very long documents or conversations.
- Multimodal Architecture: Inherits the multimodal (vision + text) capabilities of the Qwen3.5 family, though the vision encoder and aligner were frozen during this specific fine-tuning.
- Full Fine-tuning: All parameters were updated during training, utilizing bfloat16 precision and an Adam optimizer with a cosine decay learning rate schedule.
- Hardware Efficiency: Training involved 8 GPUs with data and tensor parallelism, and features like sequence parallelism and distributed optimizer were enabled.
Intended Use Cases
This model is particularly well-suited for:
- Research and Development: Provides a strong base model for exploring new fine-tuning techniques or instruction formats.
- Long-Context Applications: Excels in tasks requiring understanding and generation over extremely long text sequences, such as summarizing large documents, code analysis, or extended dialogue.
- Custom Instruction Tuning: Developers can further fine-tune this model with their own specific instruction datasets to create highly specialized conversational or task-oriented agents.
Limitations
As a base model fine-tune, it lacks inherent instruction-following capabilities and will require further instruction tuning or the application of chat templates for conversational use. It is also noted as a research checkpoint, implying potential for further performance improvements with additional training.