Qwen/Qwen2.5-7B-Instruct-1M

Qwen2.5-7B-Instruct-1M is a 7.61 billion parameter instruction-tuned causal language model developed by Qwen, featuring a transformer architecture with RoPE, SwiGLU, RMSNorm, and Attention QKV bias. This model is specifically optimized for ultra-long context tasks, supporting an impressive context length of up to 1 million tokens while maintaining strong performance on shorter tasks. It is designed for efficient processing of extensive text sequences, leveraging sparse attention and length extrapolation methods for enhanced accuracy and speed.