SamuelBang/LeRF-4B
LeRF-4B is a 4.5 billion parameter vision-language model developed by SamuelBang and researchers from the University of Illinois Urbana-Champaign, Shanghai Jiao Tong University, University of California, Merced, and Amazon AGI. It is specifically designed to address perspective-taking challenges in VLMs by learning and utilizing explicit entity-centered reference frames. The model grounds reference entities and predicts their projected origin and axes (front/left/up) to reason over visual cues, making it highly effective for spatial relation questions from various viewpoints.
Loading preview...
LeRF-4B: Perspective Taking Reasoning for Vision-Language Models
LeRF-4B is a 4.5 billion parameter vision-language model (VLM) developed to overcome a common limitation in VLMs: their struggle with perspective taking. Traditional VLMs often default to the camera's viewpoint when asked about spatial relations from another entity's or an imagined observer's perspective. LeRF-4B addresses this by learning to build and utilize an explicit entity-centered reference frame.
Key Capabilities
- Perspective Taking: Excels at answering spatial questions from diverse viewpoints (e.g., "From the man's perspective, which object is on his left?").
- Dynamic Reference Frame Generation: The model intelligently decides if a reference frame is needed, then grounds the reference entity and predicts its projected origin and axes (front/left/up) as visual cues.
- Two-Turn Inference: Operates in two turns: first, it determines if a frame is needed and, if so, generates the frame's parameters; second, it reasons over the rendered image with the frame and provides an answer.
- No External Perception/3D Reconstruction: Achieves its capabilities without relying on external perception models or 3D reconstruction, making it efficient.
- Tool Use Integration: Designed to be used with a
draw_reference_frametool and a two-turn client for optimal performance.
Good For
- Complex Spatial Reasoning: Ideal for applications requiring nuanced understanding of spatial relationships from non-camera perspectives.
- Robotics and Embodied AI: Potentially useful for agents that need to understand and interact with environments from different viewpoints.
- Advanced VLM Applications: Developers building VLMs that require robust perspective-taking abilities beyond standard object recognition and captioning.
LeRF-4B demonstrates strong performance on benchmarks like OmniSpatial perspective taking, 3DSRBench, and ViewSpatial-Bench, often outperforming other open-source models in its class, particularly in allocentric and hypothetical perspective tasks.