ShushengYang/Qwen3-VL-2B-Instruct-LLM
ShushengYang/Qwen3-VL-2B-Instruct-LLM is a 2 billion parameter instruction-tuned causal language model, derived from the text tower of Qwen3-VL-2B-Instruct. Developed by ShushengYang, this model is architecturally identical to Qwen3-1.7B dense, featuring a 32768 token context length and an interleaved MRoPE implementation. It is specifically designed as an LLM initialization for vision-language experiments, offering bit-for-bit text-only logit equivalence with its original multimodal counterpart.
Loading preview...
Overview
ShushengYang/Qwen3-VL-2B-Instruct-LLM is a 2 billion parameter instruction-tuned causal language model, extracted from the text-only component of the larger Qwen3-VL-2B-Instruct multimodal model. This model, developed by ShushengYang, retains the core language capabilities of its vision-language predecessor while shedding the vision tower, making it a pure text-based LLM.
Key Characteristics
- Architecture: It is architecturally identical to the Qwen3-1.7B dense model, featuring 2048 hidden dimensions, 28 layers, 16 query heads, 8 KV heads, a head dimension of 128, and a FFN dimension of 6144.
- Context Length: Supports a substantial context length of 32768 tokens.
- RoPE Implementation: Utilizes an interleaved MRoPE (Modified Rotary Positional Embedding) that simplifies to standard 1D RoPE for text tokens, maintaining the VL
rope_thetaof 5e6. - Equivalence: Text-only logits from this model precisely match those of the original Qwen3-VL-2B-Instruct model in fp32, ensuring no loss of text-based performance.
Intended Use
This model is primarily intended as an LLM initialization for ScaleWM MLLM experiments, serving as a baseline or component for further research into vision-language models versus text-only alternatives. It provides a robust text foundation for developers focusing on language-centric tasks or integrating it into larger multimodal systems where the vision component is handled separately.