OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents

Problem
This paper addresses the reproducibility gap in the development of multimodal search agents, particularly in the context of deep search capabilities. Despite advancements in the field, existing top-tier multimodal search agents are challenging to replicate due to a lack of open, high-quality training datasets, transparent trajectory synthesis methodologies, and comprehensive training recipes. The authors present OpenSearch-VL as a fully open-source framework aimed at mitigating these issues. This work is a preprint and has not yet undergone peer review.

Method
The core technical contribution of OpenSearch-VL lies in its comprehensive training pipeline and novel training algorithm. The authors developed a data curation pipeline that employs Wikipedia path sampling, fuzzy entity rewriting, and source-anchor visual grounding to create high-quality training datasets. Two datasets are introduced: SearchVL-SFT-36k for supervised fine-tuning (SFT) and SearchVL-RL-8k for reinforcement learning (RL). The framework also includes a diverse tool environment that integrates functionalities such as text search, image search, optical character recognition (OCR), and image enhancement techniques (cropping, sharpening, super-resolution, perspective correction). The training algorithm, termed multi-turn fatal-aware GRPO, addresses cascading tool failures by masking tokens post-failure while retaining useful pre-failure reasoning through one-sided advantage clamping. This approach enhances the robustness of the agents during multi-step reasoning tasks.

Results
OpenSearch-VL demonstrates significant performance improvements, achieving over 10-point average gains across seven benchmarks compared to established baselines. Notably, the results are competitive with proprietary commercial models on various tasks, indicating the effectiveness of the proposed training methodology and data curation strategies. Specific benchmark names and baseline models are not disclosed in the abstract, but the performance metrics suggest a substantial advancement in multimodal search capabilities.

Limitations
The authors acknowledge several limitations, including the potential biases in the curated datasets and the reliance on Wikipedia as a primary data source, which may not encompass all necessary knowledge domains. Additionally, the framework’s performance on edge cases or less common queries remains untested. The paper does not address the computational resources required for training, which could be a barrier for some researchers. Furthermore, the generalizability of the model to real-world applications outside the benchmark settings is not thoroughly explored.

Why it matters
The implications of OpenSearch-VL are significant for the field of multimodal AI. By providing an open-source framework, the authors facilitate further research and development in multimodal search agents, potentially accelerating advancements in areas such as information retrieval, question answering, and interactive AI systems. The release of high-quality datasets and a transparent training methodology can foster collaboration and innovation, enabling researchers to build upon this work and explore new applications in multimodal reasoning and active search.

Authors: Shuang Chen, Kaituo Feng, Hangting Chen, Wenxuan Huang, Dasen Dai, Quanxin Shou, Yunlong Lin, Xiangyu Yue et al.
Source: arXiv:2605.05185
URL: https://arxiv.org/abs/2605.05185v1