Vikhrmodels/Qwen2.5-7B-Instruct-Tool-Planning-v0.1

Model Overview

Vikhrmodels/Qwen2.5-7B-Instruct-Tool-Planning-v0.1 is an experimental instruction-tuned model based on the Qwen2.5-7B-Instruct architecture, developed by Dmitry Tishencko and the Vikhr Team. Its core focus is on enhancing Tool Planning capabilities, which extends traditional function calling by incorporating explicit reasoning about tool usage. The model is fine-tuned on a synthetic English dataset, Vikhrmodels/tool-plannings-v0.1.

Key Capabilities

• Tool Planning: The model first thinks about the user's intent and then executes function calls.
• Function Calling: Supports various scenarios including:
  • Simple Function: Calling a single tool when needed.
  • Multiple Function: Selecting one of several available tools.
  • Parallel Multiple Function: Executing several independent tool calls concurrently.
• Tool Relevance Detection: Ability to determine if a tool call is appropriate for a given request.
• Tool Error Handling: Reacts non-hallucinatively to failed or erroneous tool executions.

Usage and Performance

This model is designed for integration with HuggingFace Transformers and VLLM, providing specific instructions for preparing functions with Python docstrings and using chat templates. It utilizes a grounded system prompt for tool planning, which is crucial for stable behavior. Evaluation metrics indicate strong performance across different function calling complexities, with accuracies ranging from 73.25% for simple tasks to 93.00% for multiple function calls. Relevance detection shows 64.71% accuracy, while irrelevance detection is 85.72%.

Limitations

As an experimental checkpoint, the model may exhibit some hallucinations. It performs most stably with greedy search or low temperature (below 0.5). It may struggle with very long user plans, potentially confusing the order of related tasks, and can sometimes solve simple arithmetic tasks directly within the call process.