rayruiyang/VST-3B-RL
The VST-3B-RL model, developed by rayruiyang, is a 3 billion parameter Vision-Language Model (VLM) specifically designed to enhance visuospatial abilities, from spatial perception to advanced reasoning. It utilizes a progressive training pipeline, starting with supervised fine-tuning on 4.1M samples across 19 skills (VST-P) and then reinforcing spatial reasoning via reinforcement learning on 135K curated samples (VST-R). This model excels at spatial benchmarks, achieving state-of-the-art results on MMSI-Bench (34.8%) and VSIBench (61.2%) without compromising general VLM capabilities, making it suitable for applications requiring human-like spatial understanding and reasoning.
Loading preview...
Visual Spatial Tuning: VST-3B-RL
VST-3B-RL is a 3 billion parameter Vision-Language Model (VLM) developed by rayruiyang, specifically engineered to imbue VLMs with human-like visuospatial abilities. This model is detailed in the paper "Visual Spatial Tuning" and focuses on enhancing spatial perception and advanced reasoning.
Key Capabilities
- Comprehensive Spatial Training: Utilizes VST-P, a dataset of 4.1 million samples covering 19 skills across single images, multi-image scenarios, and videos to boost spatial perception.
- Reinforced Spatial Reasoning: Employs VST-R, a curated dataset of 135,000 samples, to teach models step-by-step and rule-based spatial reasoning through reinforcement learning.
- Progressive Training Pipeline: Combines supervised fine-tuning for foundational spatial knowledge with reinforcement learning to achieve state-of-the-art results on spatial benchmarks.
- High Spatial Benchmark Performance: Achieves 34.8% on MMSI-Bench and 61.2% on VSIBench, demonstrating strong performance in spatial tasks while maintaining general VLM capabilities.
- Enhanced Vision-Language-Action Models: The VST paradigm significantly strengthens spatial tuning, contributing to more physically grounded AI applications.
Good For
- Applications requiring advanced visuospatial understanding and reasoning.
- Tasks involving spatial perception, object localization, and relational understanding in images and videos.
- Developing physically grounded AI systems and robotics where precise spatial awareness is crucial.
- Research into improving VLM capabilities for complex spatial problem-solving.