OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation

Thank you for your constructive review and for recognizing the motivation, simplicity, and effectiveness of our approach. We address your concerns in detail below.

Q1) The gains for unseen classes are not well justified. Further explanation or analysis is needed to clarify where these improvements come from.

We agree with the reviewer that further analysis is needed to justify the performance gains on unseen classes.

To investigate the source of these improvements, we analyzed the performance on the zero-shot dataset ADE20K by comparing the PQ, SQ, and RQ scores of the baseline and our proposed method. Note that this experiment was conducted under Scenario 1, where the incremental dataset is Cityscapes.

As shown in the table below, applying ConOVS to the baseline model (fc-clip) improves both PQ and RQ. However, the most significant gain is observed in SQ, which measures the quality of the predicted segmentation masks. This suggests that the improvements in unseen classes primarily stem from enhanced segmentation quality rather than better mask classification.

We will include this analysis in Section 6.1 (Main Results). Thank you for the feedback.

Q2) The choice to freeze the encoder and only fine-tune the decoder is not well justified. An ablation on this design choice is needed.

We agree with the reviewer that, given recent works that fine-tune the CLIP encoder $1,2$ , a more detailed ablation study on our design choice to freeze the encoder and only fine-tune the decoder is necessary.

To this end, we experimented with three encoder fine-tuning strategies: 1) fine-tuning only the last block of the encoder, 2) fine-tuning only the LayerNorm modules, and 3) replacing all MLP layers with LoRA modules and fine-tuning them. Note that, unlike prior attention-based methods designed for ViT-style CLIP encoders $1,2$ , our strategies are tailored for ConvNeXt-based encoders used in fc-clip.

As shown in the table below, all three encoder fine-tuning strategies yield lower performance on the incremental dataset compared to our design choice of fine-tuning only the decoder. We attribute this to the architectural difference: while previous methods $1,2$ generate segmentation masks directly from the encoder and benefit from encoder fine-tuning, fc-clip generates masks in the decoder's mask head. This suggests that decoder fine-tuning is more effective for improving segmentation performance in our setup.

We will include this analysis in Section 6.2 (Ablation Study).

$1$ Cho, Seokju, et al. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. CVPR 2024.

$2$ Lee, Minhyeok, et al. Effective SAM Combination for Open-Vocabulary Semantic Segmentation. CVPR 2025.

Q3) Furthermore, could the proposed method be applicable to cost-based OVS methods that fine-tune the encoder, e.g. CAT-Seg?

In general, our method is also applicable to cost-based OVS approaches. Designed to merge independently trained models, it remains compatible with techniques that fine-tune the CLIP encoder. While cost-based methods typically generate segmentation maps by post-processing encoder features, our method does not rely on such steps, making it directly applicable without modification.

However, when applying our method to cost-based OVS models, the encoding process must be performed twice. This is because our method computes the interpolation factor based on the proximity of the input sample, which requires an initial forward pass through the encoder. After determining the interpolation factor, a second forward pass must be performed using the merged encoder to generate the final feature representation. As a result, an increase in inference time is expected.

We believe that extending our method to encoder fine-tuning frameworks, such as cost-based OVS models, is an interesting direction for future work. We will include this discussion in the newly added Future Works section following Section 7.

Q4) Given that cost-based methods already mitigate forgetting by fine-tuning CLIP encoders, it would be interesting to see whether ConOVS provides complementary benefits.

Our method is complementary to cost-based OVS models in terms of preventing catastrophic forgetting. When our method is applied to cost-based OVS techniques, a cost-based expert is trained for each domain. The merging of these experts is then guided by interpolation factors computed using the original CLIP encoder, which remains accessible at all times. As a result, we expect that combining our method with cost-based OVS models can further enhance their inherent ability to mitigate catastrophic forgetting.

We consider empirically verifying this possibility an interesting direction for future work, and we plan to include this discussion in the Future Work section of the final version.

Q5) Given that the current retraining baseline underperforms due to limited training, how much improvement can be expected if it is allowed to fully converge?

We agree that evaluating the retraining baseline with longer training iterations offers valuable insights into the continual learning scenario for OVS. To conduct this analysis, we performed an experiment using Cityscapes as the incremental dataset.

As shown in the table below, retraining the model with a longer training schedule (100k iterations) leads to better performance than our method on the pre-training and incremental datasets.

However, despite consuming significantly more computational resources, the retraining method does not yield substantially better performance than ConOVS. One possible explanation for the limited performance of retraining is task interference $3,4$ , which might occur when training on multiple domains simultaneously. In contrast, ConOVS avoids this issue by training domain-specific experts independently and combining them adaptively at inference time. As a result, our method achieves comparable performance than retraining, while requiring significantly less computational cost.

We will include this analysis in the final version of the paper. We appreciate the helpful feedback.

$3$ Yu, Tianhe, et al. Gradient surgery for multi-task learning. NeurIPS 2020.

$4$ Chen, Zhao, et al. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. ICML 2018.

Thank you for the detailed response. The rebuttals have resolved my concerns, and it is nice to see results on additional datasets presented on responses to other reviews. I also agree on what authors raised as consideration for finetuning the encoder, and seems sound to leave it out for future work.

Thank you for your detailed review and constructive feedback. We appreciate your comments on the writing, experimental setup, and relation to SemLA, and address each point below.

Q1) This paper and SemLA share a core idea: estimating dataset proximity to compute weights for merging LoRA modules, similar to ConOVS’s MoE approach.

SemLA $1$ was released on arXiv in March 2025 and presented at CVPR in June 2025. According to the NeurIPS Call for Papers, it is officially considered a contemporaneous work.

Nonetheless, we agree with the reviewer that a discussion on the conceptual connection between SemLA and our method is necessary. As the reviewer pointed out, there is a conceptual similarity between the two approaches—both compute the proximity of an input image to each training dataset and use it to determine dynamic merging weights during inference. However, our contributions differ from those of SemLA in several important aspects:

• Our paper is the first to identify and address the limitations of existing open-vocabulary segmentation (OVS) methods in continual learning (CL) scenarios, where new datasets are continuously collected over time. Highlighting the issues of OVS under CL is an important contribution of our work. To the best of our knowledge, this setting has not been considered in SemLA, which focuses on single-stage inference without addressing continual dataset shifts.

• Compared to SemLA, our method adopts different design choices in three key aspects: (1) it uses both image and text embeddings to assess domain relevance, whereas SemLA relies solely on image embeddings; (2) it estimates domain proximity using multivariate normal (MVN) distributions, while SemLA computes L2 distances to dataset-specific centroids; and (3) it fine-tunes the decoder for each dataset, in contrast to SemLA, which applies LoRA modules to the CLIP image encoder. A detailed explanation of the first point is provided in our response to Q2 below.

In addition, we implemented SemLA within our experimental setup for a direct comparison. Specifically, we modified our framework to follow SemLA’s key design choices: (1) computing proximity using only image embeddings, (2) estimating domain proximity by calculating the L2 distance between the input embedding and dataset-specific centroids, and (3) replacing decoder fine-tuning with LoRA fine-tuning on the CLIP image encoder.

As shown in the table below, our method (ConOVS) outperforms this SemLA variant on the incremental dataset. However, since multiple design differences exist between the two methods, it is difficult to isolate the exact cause of the performance gap. A deeper analysis of this result was challenging to conduct within the limited rebuttal period, but we consider it an important direction for future work. Nonetheless, the observed improvement suggests that our design choices might be more suitable for continual open-vocabulary segmentation.

We will add the discussion on the relation between SemLA and our method to Section 2 (Related Works), and the experimental results to Section 6 (Experiments). We appreciate the insightful feedback.

$1$ Qorbani, Reza, et al. Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation. CVPR 2025.

Q2) Section 5.2 lacks clarity in vector shapes and operations like softmax and max; the rationale should be explained more clearly.

We agree with the reviewer that the description in Section 5.2 was difficult to follow. In this response, we first clarify the rationale behind our use of softmax and element-wise maximum operations, and then provide details on the improvements we made to Section 5.2 for better readability.

We apply the softmax operation to the log-likelihood vector to normalize the proximity scores of each domain to the $0, 1$ range. This decision is based on a prior study $2$ , which found that when the interpolation factor exceeds 1, merging performance degrades. To normalize the log-likelihood values obtained from the MVN distribution, we considered three strategies, including min-max normalization, sigmoid, and softmax. As shown in the table below, softmax achieved the best performance, which led us to adopt it.

We use the element-wise maximum operation to combine information from both the image and text modalities. Initially, we considered whether to rely on only the image domain, only the text domain, or both. For combining both modalities, we evaluated three options: average, multiplication, and element-wise maximum. As shown in the table below, the results show that combining both modalities performs better than using a single modality, and among the combination methods, element-wise maximum achieved the best performance. Based on these findings, we selected element-wise maximum as our fusion strategy.

We acknowledge that the use of element-wise maximum does not guarantee that the resulting proximity values sum to one. However, this aligns with prior practice in task vector-based merging methods $2,3$ , where interpolation factors are not required to sum to one. In our experiments, we observed no adverse effects resulting from this design choice.

To improve the clarity of Section 5.2, we not only added the above explanations regarding our design choices but also provided more concrete details, such as the dimensions of the vectors. For example, we recognized that the definitions of the likelihood vectors $l\_{\\text{text}}$ and $l\_{\\text{img}}$ were ambiguous. We revised the text to clarify that these vectors represent the proximity values between the input sample and each dataset, and we explicitly stated their dimensions to improve readability.

$2$ Ilharco, Gabriel, et al. Editing models with task arithmetic. ICLR 2023.

$3$ Yadav, Prateek, et al. Ties-merging: Resolving interference when merging models. NeurIPS 2023.

Q3) The number of incremental datasets is too small to convincingly demonstrate generalizability. More datasets should be included for a convincing evaluation.

We agree that evaluating our method with a larger number of incremental datasets is important for demonstrating generalizability. To address this, we designed a new scenario in which the model is sequentially trained on five datasets: COCO, Cityscapes, ADE20K, BDD100K, and Mapillary Vistas. While we aimed to include more datasets, our work focuses on panoptic segmentation, for which publicly available datasets are considerably more limited than those available for semantic segmentation as used in SemLA.

The results of this extended experiment are shown in the table below. As can be seen, our method outperforms the baseline, fine-tuning, and retraining approaches. These results indicate that our method remains effective as the number of incremental datasets increases, demonstrating its scalability under complex domain shifts.

We will include this experiment and its discussion in Section 6.1 (Main Results). Thank you for the valuable feedback.

Q4) In Line 127, it would be beneficial if the "characteristics of OVS tasks" could be elaborated upon.

What we intended by "characteristics of OVS tasks" is the fundamental difference between open-vocabulary segmentation (OVS) and closed-set segmentation. Specifically, OVS tasks involve a potentially infinite label space, whereas closed-set segmentation assumes a fixed, finite label set. Taking this into account, we will revise the sentence as follows:

We believe this arises because existing CL methods assume a closed-set segmentation with a finite label space, whereas OVS involves a potentially infinite label space, which these methods do not account for.

Q5) For Figure 2(b), it would be helpful to include citations for competitors in the caption.

We agree with the reviewer and will update the caption of Figure 2(b) to include citations for the continual learning methods used. The revised caption is as follows:

Figure 2: … (b) Comparison of the performance of OneFormer $12$ , the baseline (fc-clip $44$ ), retraining, fine-tuning, three existing continual learning methods $14,15,18$ , and ConOVS on the pre-training and incremental datasets.

Thank you for highlighting the strengths of our work, particularly the thorough experimental validation and in-depth analysis. Below, we provide detailed responses to your concerns.

Q1) The paper lacks a comparison of training costs between retraining, SFT, and continual learning. A detailed analysis of time and resource usage is needed.

As the reviewer mentioned, the original draft did not include an explicit comparison of training costs across methods, which we acknowledge as a critical omission. Although our intention was to demonstrate the efficiency of ConOVS by showing that it achieves better performance than retraining with similar computational cost, this point was not sufficiently supported without the training cost comparison. Thank you for highlighting this issue.

To address this, we measured the training time and GPU memory usage required by each method and summarized the results in the table below. Note that all values are reported as relative figures normalized to the computational cost of standard fine-tuning.

According to this comparison, the training time does not vary significantly across methods. This is expected, as we fixed the number of training iterations to ensure a fair comparison. One exception is ECLIPSE, which shows lower training cost. This is due to its use of Visual Prompt Tuning, which updates only a small subset of model parameters. While this leads to higher training efficiency, our results indicate that ConOVS outperforms ECLIPSE in terms of overall segmentation performance. We attribute this performance gap to the inherent limitations of updating only a fraction of the model parameters.

We will include this discussion and the updated table in Section 6 (Experiments) and Table 1. We appreciate your thoughtful feedback.

Q2) The use of only 2–3 incremental datasets limits evaluation of scalability and robustness under longer learning sequences and diverse domain shifts.

We agree that evaluating our method with a larger number of incremental datasets is important for demonstrating generalizability. To address this, we designed a new scenario where the model is sequentially trained on five datasets: COCO, Cityscapes, ADE20K, BDD100K, and Mapillary Vistas. While we aimed to include more datasets, we were limited to those providing panoptic segmentation annotations.

The results of this extended experiment are shown in the table below. As can be seen, our method outperforms the baseline, fine-tuning, and retraining approaches. These results indicate that our method remains effective as the number of incremental datasets increases, demonstrating its scalability under complex domain shifts.

We will include this experiment and its discussion in Section 6.1 (Main Results). Thank you for the valuable feedback.

Thank you for your thoughtful review and for highlighting the novelty and clarity of our approach. We address your questions regarding generalizability and robustness in detail below.

Q1) Could this approach also be applied to OVD?

In general, our proposed method is not limited to open-vocabulary segmentation (OVS) but can also be extended to open-vocabulary detection (OVD) frameworks.

Our approach is applicable to models with an encoder-decoder architecture that perform classification based on the similarity between image and text embeddings. This structural characteristic is shared by most OVD methods. For instance, YOLO-World $1$ adopts a modular design consisting of an encoder and a prediction head, and omits the use of a conventional fc-based classifier. Given this structure, all components of our method can be directly incorporated without modification.

To be more specific, for each newly introduced training dataset, an MVN distribution can be formed using the encoder's embeddings. During inference, the interpolation weights can be dynamically adjusted based on the proximity between the input sample and each domain, enabling the model to incrementally extend its recognition capability.

We believe that applying our method to OVD models represents a promising direction for future work. Accordingly, we plan to include this discussion in the newly added Future Work section following Section 7 (Limitation).

$1$ Cheng, Tianheng, et al. YOLO-World: Real-time Open-Vocabulary Object Detection. CVPR 2024.

Q2) Why does continual learning yield better performance than retraining from scratch, as shown in Table 1?

In our experiments, the retraining method showed relatively lower performance because we conducted all methods under the same computational budget. Since retraining involves learning from a significantly larger amount of data compared to our method, it requires a longer training schedule to reach convergence.

To analyze this more precisely, we conducted additional experiments where the retraining method was trained for longer durations. As shown in the table below, when trained with a sufficiently long schedule (100k iterations), the retraining method outperforms our method on the pretraining and incremental datasets.

However, despite consuming significantly more computational resources, the retraining method does not yield substantially better performance than ConOVS. One possible explanation for the limited performance of retraining is task interference $2,3$ , which might occur when training on multiple domains simultaneously. In contrast, ConOVS avoids this issue by training domain-specific experts independently and combining them adaptively at inference time. As a result, our method achieves comparable performance than retraining, while requiring significantly less computational cost.

Additionally, we acknowledge that Table 1 in the original manuscript does not clearly convey how the training cost of retraining compares with other methods. To address this, we have included a comparison of training time and GPU memory usage across all methods. Please refer to our response to Reviewer yU8U’s Q1 for more details.

We recognize that this important explanation was not included in the original draft and have now added the relevant discussion to Section 6.1 (Main Results) and the caption of Table 1. We appreciate the constructive feedback.

$2$ Yu, Tianhe, et al. Gradient surgery for multi-task learning., NeurIPS 2020.

$3$ Chen, Zhao, et al. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks., ICML 2018.

Q3) The method assumes each incremental dataset is from a distinct domain, which may not always be the case. How does it handle mixed or unknown domains during training?

As suggested by the reviewer, we conducted an additional experiment to examine whether our method remains effective when a single incremental dataset consists of multiple domains. To this end, we combined Cityscapes and ADE20K into a single incremental dataset. Note that the performance on the zero-shot dataset was computed by averaging the results from four datasets: PC-59, PC-459, PAS-20, and PAS-21.

As shown in the table below, our method preserves the performance of the baseline fc-clip on the pre-training dataset, while improving performance on both the incremental and zero-shot datasets. This demonstrates that our method remains robust and effective even when the incremental dataset includes samples from diverse domains.

We will include this analysis in the final version of the paper. We appreciate the helpful feedback.

Q4) How does the method perform when the incremental dataset is small or weakly labeled? Is it robust in low-resource settings?

To examine whether the proposed method remains effective in low-resource settings, we conducted an additional experiment using a small incremental dataset. Specifically, we randomly sampled only 5% of the Cityscapes training dataset and used it as the incremental dataset.

As shown in the table below, our method successfully maintains the performance on the pre-training dataset while improving performance on both the incremental and zero-shot datasets, even when trained with the small incremental dataset. This demonstrates that the proposed method remains effective in low-resource settings where the number of incremental samples is limited.

We also considered evaluating our approach under weakly labeled incremental settings. However, due to the lack of publicly available datasets that contain appropriate weak annotations, such as pseudo labels for panoptic segmentation, we were unable to conduct this experiment.

Nevertheless, we believe that our method can also be applied in weakly labeled settings. In fact, we consider our approach to be effective in overcoming the common issue of low-quality segmentation annotations found in weakly labeled datasets. This is because our technique constructs the MVN distribution using only image and text embeddings, without relying on segmentation annotations. Therefore, we believe there are no constraints in forming the MVN distribution, and we expect to accurately estimate the interpolation coefficients. While label noise may affect the overall performance, we believe our method can still contribute to effectively enhancing the model’s recognition ability even in weakly labeled environments.

We will include the experiment and analysis regarding small incremental datasets in Section 6 (Experiments), and the discussion on weakly labeled datasets will be added to the Future Work section. We appreciate your thoughtful feedback.