SamuelBang/LeRF-9B
LeRF-9B is a 9 billion parameter vision-language model developed by SamuelBang, designed to overcome the limitations of traditional VLMs in perspective-taking reasoning. It achieves this by learning to build and utilize an explicit entity-centered reference frame, predicting its origin and axes to reason over visual cues. This model excels at understanding spatial relations from various viewpoints, making it particularly effective for complex visual question answering tasks that require non-camera-centric perspectives.
Loading preview...
LeRF-9B: Enhanced Perspective Taking in Vision-Language Models
LeRF-9B is a 9 billion parameter vision-language model (VLM) developed by SamuelBang, specifically engineered to address the challenges VLMs face with perspective taking. Traditional VLMs often default to camera-centric views when asked about spatial relations from another entity's or an imagined observer's viewpoint. LeRF-9B overcomes this by learning to construct and use an explicit entity-centered reference frame.
Key Capabilities
- Perspective Taking Reasoning: Accurately answers questions requiring understanding spatial relations from non-camera perspectives.
- Dynamic Reference Frame Generation: Identifies when a reference frame is needed, grounds the reference entity, and predicts its projected origin and front/left/up axes on an image.
- Two-Turn Inference: Utilizes a unique two-turn inference process where the model first decides if a frame is needed and generates its parameters, then reasons over the rendered frame to provide an answer.
- Tool Integration: Designed to work with a
draw_reference_frametool that renders the predicted frame onto the image, providing a visual cue for subsequent reasoning. - Robust Training: Trained in two stages: Supervised Fine-Tuning (SFT) for grounding and frame prediction using pose datasets (ImageNet3D, Omni6DPose-SOPE, BEDLAM), followed by Reinforcement Learning (RL) with GRPO on MultihopSpatial and SpatialReasoner-RL for improved reasoning.
Good For
- Complex Visual Question Answering: Ideal for scenarios where understanding spatial relationships from diverse viewpoints is crucial.
- Robotics and Embodied AI: Applications requiring an agent to understand its environment from different perspectives.
- Spatial Reasoning Tasks: Excels in benchmarks like OmniSpatial-PT, 3DSRBench, and ViewSpatial-Bench, demonstrating strong performance in ego, allo, and hypothetical perspective tasks.
- Research in VLM Perspective Understanding: Provides a robust framework for exploring and advancing perspective-taking capabilities in AI.