rayruiyang/VST-7B-RL
The rayruiyang/VST-7B-RL is a 7 billion parameter Vision-Language Model (VLM) developed by Rui Yang et al., designed to enhance visuospatial abilities in AI. This model, based on the Visual Spatial Tuning (VST) framework, excels in spatial perception and reasoning tasks, including single-image, multi-image, and video scenarios. It achieves state-of-the-art results on spatial benchmarks like MMSI-Bench (34.8%) and VSIBench (61.2%) through a progressive training pipeline involving supervised fine-tuning and reinforcement learning. VST-7B-RL is particularly suited for applications requiring advanced spatial understanding, such as robotic learning and physically grounded AI.
Loading preview...
VST-7B-RL: Enhancing Visuospatial Abilities in VLMs
VST-7B-RL is a 7 billion parameter Vision-Language Model (VLM) developed by Rui Yang et al., focusing on cultivating human-like visuospatial abilities. This model is part of the comprehensive Visual Spatial Tuning (VST) framework, which addresses both spatial perception and advanced reasoning.
Key Capabilities & Training
- VST-P (Perception): Utilizes 4.1 million samples across 19 skills, covering single images, multi-image scenarios, and videos to boost foundational spatial perception in VLMs.
- VST-R (Reasoning): Employs 135,000 curated samples to teach models step-by-step spatial reasoning, incorporating rule-based data for reinforcement learning.
- Progressive Training Pipeline: The model undergoes supervised fine-tuning to establish core spatial knowledge, followed by reinforcement learning to refine spatial reasoning. This approach allows VST-7B-RL to achieve strong performance on spatial benchmarks without compromising general VLM capabilities.
Performance Highlights
VST-7B-RL demonstrates state-of-the-art results on key spatial benchmarks:
- MMSI-Bench: Achieves 34.8% accuracy.
- VSIBench: Achieves 61.2% accuracy.
These scores represent significant improvements in spatial understanding compared to its supervised fine-tuned counterpart (VST-7B-SFT).
Use Cases
- Robotic Learning: The VST paradigm significantly strengthens Vision-Language-Action Models, paving the way for more physically grounded AI applications.
- Advanced Spatial Reasoning: Ideal for tasks requiring detailed understanding of object locations, distances, sizes, and relative directions within complex visual scenes.
- Multi-modal AI: Suitable for scenarios involving analysis of single images, multiple images, and video content where spatial relationships are critical.