SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction

Problem
This paper addresses the limitations of existing referring segmentation methods, particularly in handling complex scenarios involving multiple instances, cross-category groups, and open-ended target sets. Previous approaches, primarily based on Large Vision Language Models (LVLMs), inadequately represent referred targets by treating them as separate outputs, which hinders the model’s ability to capture set-level properties such as completeness and mutual exclusivity. The authors propose a novel framework, SetCon, to reformulate the task as explicit set-level concept prediction. This work is presented as a preprint and has not yet undergone peer review.

Method
SetCon introduces a two-stage approach for referring segmentation that leverages LVLM-generated natural-language concepts as semantic conditions for joint mask-set decoding. The architecture consists of a hierarchical semantic decomposition that first predicts a shared set-level concept to define the target scope. This is followed by a refinement process that delineates fine-grained concept groups corresponding to target subsets. The authors augment existing reasoning segmentation datasets through a two-stage annotation pipeline, resulting in 236,000 samples and 784,000 concept phrases, which provide hierarchical semantic supervision. The training process and compute resources are not explicitly detailed in the paper.

Results
SetCon achieves state-of-the-art performance on several image benchmarks, demonstrating significant improvements over named baselines. Specifically, it reports a +3.3 gIoU increase on the gRefCOCO dataset and a +12.1 gIoU increase on the MUSE dataset. Notably, the performance gains are more pronounced as the number of referred targets increases, indicating the model’s enhanced capability in complex scenarios. Additionally, the concept interface is successfully adapted to video segmentation tasks, achieving new state-of-the-art results on seven referring video benchmarks, including a +10.9 J&F improvement on MeViS and a +12.4 J&F improvement on Ref-SeCVOS.

Limitations
The authors acknowledge that the proposed method may still struggle with highly ambiguous queries or scenarios where the target instances are visually similar, which could lead to misinterpretation of the set-level concepts. Furthermore, the reliance on hierarchical semantic supervision may limit the model’s generalizability to datasets lacking such annotations. An additional limitation not explicitly mentioned is the potential computational overhead introduced by the two-stage annotation process, which may not be feasible for all applications.

Why it matters
The implications of this work are significant for advancing the field of referring segmentation, particularly in scenarios requiring the identification of multiple, complex targets. By framing the task as set-level concept prediction, SetCon opens avenues for more coherent and contextually aware segmentation models. This approach not only enhances performance on existing benchmarks but also sets a foundation for future research in open-ended segmentation tasks, potentially influencing applications in robotics, autonomous driving, and interactive AI systems.

Authors: Zhixiong Zhang, Yizhuo Li, Shuangrui Ding, Yuhang Zang, Shengyuan Ding, Long Xing, Yibin Wang, Qiaosheng Zhang et al.
Source: arXiv:2605.20110
URL: https://arxiv.org/abs/2605.20110v1