muradil211/AetherSearch_DPO

AetherSearch DPO: A Preference-Aligned Search Agent

AetherSearch DPO is a 3.1 billion parameter Qwen2 causal language model developed by muradil211, specifically designed as a search agent for retrieval-augmented reasoning. It is built upon the AetherSearch SFT checkpoint and further aligned using Direct Preference Optimization (DPO) on 2,126 search-trajectory preference pairs. This alignment teaches the model to favor stronger search decisions, effective evidence use, and high-quality final answers.

Key Capabilities

• Preference Aligned: Optimized to make superior search decisions and utilize evidence more effectively through DPO training.
• Search Native: Generates explicit reasoning, retrieval requests, and grounded answers using a defined XML protocol (<think>, <search>, <information>, <answer> tags).
• Multi-step Retrieval: Can request additional evidence by performing multiple search iterations if initial retrieval is insufficient.
• Retrieval-Aware DPO: The DPO loss function is adapted to exclude environment-provided <information> spans from preference-loss scoring while keeping them visible as context, ensuring the model learns from the search process itself.
• High Context Window: Supports a 32,768-token context window, allowing for extensive retrieval and reasoning.

How it Works

The model operates by generating a <think> span, followed by a <search> query. A host runtime executes this query, appends the results as <information>, and resumes generation. This loop continues until the model emits a final <answer> span. The DPO alignment specifically optimizes the model's ability to navigate these search trajectories effectively.

Good For

• Developing advanced retrieval-augmented generation (RAG) systems.
• Building intelligent agents that require multi-step information seeking.
• Applications demanding preference-aligned search and reasoning capabilities.
• Scenarios where explicit reasoning and grounded answers are critical.