Tongyi-MiA/UI-Ins-7B
Tongyi-MiA/UI-Ins-7B is a 7 billion parameter multimodal model developed by Tongyi-MiA, specifically designed for GUI grounding tasks. It utilizes an "Instruction-as-Reasoning" paradigm, treating instructions as dynamic analytical pathways to map natural language to actionable UI elements. The model is trained with a two-stage framework involving supervised fine-tuning on synthesized, diverse instructions and reinforcement learning for pathway optimization. UI-Ins-7B achieves state-of-the-art results on challenging grounding benchmarks and demonstrates strong agentic potential, making it suitable for GUI automation and agent development.
Loading preview...
Tongyi-MiA/UI-Ins-7B: GUI Grounding with Multi-Perspective Instruction-as-Reasoning
UI-Ins-7B is a 7 billion parameter multimodal model developed by Tongyi-MiA, focused on enhancing GUI grounding, the process of mapping natural language instructions to specific UI elements. This model introduces the Instruction-as-Reasoning paradigm, which views instructions not as static commands but as dynamic analytical pathways. This allows the model to select and compose the most effective reasoning path during inference, significantly improving grounding performance.
Key Capabilities
- State-of-the-Art GUI Grounding: Achieves top results on five challenging grounding benchmarks, including 87.3% on UI-I2E-Bench and 74.1% success rate on AndroidWorld (as an executor).
- Multi-Perspective Reasoning: Trained to understand and leverage diverse instruction interpretations, mitigating common flaws found in existing grounding datasets.
- Two-Stage Training: Employs supervised fine-tuning (SFT) with synthesized, diverse instructions followed by reinforcement learning (RL) to optimize pathway selection and composition.
- Emergent Reasoning: Demonstrates the ability to selectively compose and synthesize novel instruction pathways during inference.
Good for
- Developing GUI agents that can accurately interpret and act upon natural language commands.
- Automating tasks requiring precise UI element identification from textual instructions.
- Research into multimodal instruction understanding and dynamic reasoning in visual environments.