FuTCR: Future-Targeted Contrast and Repulsion for Continual Panoptic Segmentation

Problem
This paper addresses the limitations of existing methods in Continual Panoptic Segmentation (CPS), particularly the challenge of adapting to new object categories over time. Current approaches often treat all unlabeled pixels as a single “background” class, which hinders the model’s ability to learn distinct background categories as they are introduced. This preprint work proposes a novel framework to improve the model’s capacity to differentiate between various background categories while learning new classes.

Method
The authors introduce the Future-Targeted Contrastive and Repulsive (FuTCR) framework, which restructures feature representations prior to the introduction of new classes. FuTCR operates in two main phases:

1. Future-like Region Discovery: The model identifies confident regions that are predicted as background but have non-background logits. This is achieved by clustering model-predicted masks to isolate these ambiguous regions.

2. Contrast and Repulsion Mechanism: The framework employs a pixel-to-region contrastive loss to create coherent prototypes from the identified unlabeled regions. Simultaneously, it implements a repulsion mechanism that pushes background features away from known-class prototypes, thereby reserving representational capacity for future categories. The training process leverages a combination of labeled and unlabeled data, enhancing the model’s adaptability.

Results
FuTCR was evaluated across six CPS settings with varying dataset sizes. The results indicate a significant improvement in new-class panoptic quality, achieving up to a 28% increase compared to state-of-the-art methods. Additionally, the framework maintains or enhances base-class performance, with improvements of up to 4%. These results demonstrate the effectiveness of FuTCR in both learning new categories and preserving existing knowledge.

Limitations
The authors acknowledge that the proposed method may still struggle with highly ambiguous unlabeled regions, which could lead to suboptimal prototype formation. Furthermore, the reliance on the model’s initial predictions for future-like region discovery may introduce biases if the model’s performance is not robust. An additional limitation not explicitly mentioned is the potential computational overhead associated with the contrastive and repulsive mechanisms, which may affect scalability in real-time applications.

Why it matters
The FuTCR framework has significant implications for the field of continual learning and segmentation tasks. By effectively managing the representation of background categories, it paves the way for more nuanced and flexible models that can adapt to evolving datasets. This work contributes to the broader discourse on improving CPS methodologies, particularly in scenarios where labeled data is scarce or where the introduction of new categories is frequent. The ability to maintain performance on existing classes while accommodating new ones is crucial for real-world applications in autonomous systems, robotics, and interactive AI.

Authors: Nicholas Ikechukwu, Keanu Nichols, Deepti Ghadiyaram, Bryan A. Plummer
Source: arXiv:2605.12451
URL: https://arxiv.org/abs/2605.12451v1