Thank you for your positive feedback on our motivation, contributions, and performance gain. We sincerely appreciate your constructive comments, which have helped us improve our manuscript. Below, we provide our point-by-point responses to address the concerns raised:

Q1: Insight into Parameter Freezing

A1: The self-attention layers operate on the encoded feature space, which extracts general semantic information (such as relationships between regions or objects) that remains useful across tasks. Freezing these layers prevents the model from overwriting these learned relationships, helping to generalize to unseen tasks. The cross-attention layers align object queries (representing instances or candidates) with the encoded feature map of the image. Differentiating classes often relies on distinct features; for example, the number of legs helps distinguish a person from a dog, while color and texture differentiate a horse from a zebra. Fine-tuning the cross-attention layers enables the model to adapt to novel classes by attending to these distinctive features for each query. Finally, the feed-forward networks (FFN) facilitate better feature interaction and generalization by bridging the gap between the frozen self-attention layers and the active cross-attention layers, enabling more task-specific learning while maintaining stability.

Q2: Lack of Qualitative Analysis

A2: We provide qualitative visualizations in Figure 4 in Appendix of the revised manuscript to demonstrate the superior rigidity and plasticity of our approach. Specifically, ADAPT effectively preserves base class knowledge, such as cabinet and door (first column), while also generalizing well to novel classes, including fan (second column), tent (third column), trade name (last column), and trash can (last column).

Q3: Efficiency Analysis

A3: The table below (Table 6 in the revised manuscript) highlights the efficiency of our approach. During incremental training, ADAPT requires only 2.27 hours for five training steps, representing a reduction of 34.8% and 32.4% in training time compared to CoMFormer and ECLIPSE, respectively. The reduced training time can be attributed to ADAPT's frozen- and shared-weight design between the teacher and student models, which eliminates the need for dual-model forward passes for knowledge distillation in CoMFormer. Additionally, our efficient adaptation strategy enables a fourfold reduction in the number of training iterations for each step, decreasing from 16k in ECLIPSE to 4k in our approach. This substantial reduction contributes to the overall decrease in training time. In terms of inference, our method incurs a modest 6.5% increase in FLOPs compared to ECLIPSE. However, this is accompanied by substantial performance gains, particularly in novel PQ, which improves from 16.5 to 26.3—a remarkable 59.4% increase. Similar improvements are observed when compared to CoMFormer. Furthermore, the inference speed of ADAPT remains comparable to that of CoMFormer, highlighting the efficiency and practicality of our approach. In contrast, ECLIPSE demonstrates slower inference speeds, likely attributable to its unoptimized implementation. We have added the efficiency analysis to Sec. 4.5.5.

Q4: Insufficient Details in Figure 2

A4: We have added color representations in Figure 2, where "sky" (blue) represents the base class, and "grass" (green) and "airplane" (orange) are the novel classes at time step t. Additionally, as highlighted in Line 312 of the revised version, we point out that the “airplane” is misclassified as “sky” by the Probability-Level Fusion (PLF) method. Regarding the number of queries and class probability scores, we clarify that these factors are unrelated. Specifically, the number of queries defines the maximum number of instances that can be detected, while the number of class probability scores predicted for each query indicates the number of semantic classes the model can recognize. Finally, we explain why only four segments (sky, grass, and two airplanes in $\bar{M}_i^t$) are presented for 8 queries after Query-Level Fusion (QLF). During the fusion process, duplicate segments with high overlap are suppressed, similar to the Non-Maximum Suppression (NMS) used in object detection. We adopt the standard panoptic fusion strategy from Mask2Former for this procedure, and the detailed steps are outlined in Algorithm 1 in the Appendix of the revised manuscript.

Q5: Ambiguity in Baseline Setting (Table 5)

A5: We use the official implementation of CoMFormer as our codebase and treat it as our baseline in Table 5. We clarified this in Line 424 of the revised manuscript.

Q6: Probability Inconsistency Analysis

A6: We adopt the Softmax activation function to compute the final class scores (added in Line 269). In response to your concern, we conduct experiments on the ADE20K 100-10 scenario to compare Softmax and Sigmoid. We observe that Sigmoid is less effective, whether used with PLF or QLF. Since Sigmoid does not normalize across classes, the class probabilities are independent, which can lead to high confidence values for both base and novel classes, especially when they are semantically similar. For example, in ECLIPSE, the base class “lake” was misclassified as the novel class “water.” As a result, ECLIPSE proposed logit manipulation to alleviate this issue, which however brings several hyperparameters $\delta_i (i=0, 1...t)$ at step $t$ that require tailored tuning in different settings, e.g., 100-50, 100-10, etc. In contrast, Softmax can suppress such misclassifications through cross-class normalization. As shown in the table below, our approach with Softmax (in combination with QLF) yields significantly better performance without requiring complex hyperparameter tuning.

Additional Comments

Q7: Experiments in the Disjoint Setting

A7: The pioneering work of MiB introduced two distinct settings for continual learning: disjoint and overlap. Since the overlap setting is generally considered more challenging and realistic, we focused primarily on it in our main paper. For completeness, we additionally present experimental results for continual panoptic segmentation on ADE20K under the disjoint setting. The results, as summarized below (Table 8 in the Appendix of our revised manuscript), highlight the superior performance of ADAPT over existing methods.

Q8: Experiments on COCO and robustness on random class ordering

A8: Following ECLIPSE, we conduct experiments using two settings (83-5, 83-10) on the COCO benchmark. During this process, we identified and fixed two bugs in their implementation: 1) the class indices in COCO start from zero, while their code assumed they start from one, leading to misalignment; and 2) their result summarization for base and novel classes did not account for class order shuffling. After correcting these issues, we obtained the results in the table below (Table 7 of the revised manuscript), where our approach significantly outperforms other competing methods.

We examine the robustness of our method to different class ordering on ADE20K 100-10 scenario. We utilize the 10 randomly shuffled class orders proposed by ECLIPSE for consistency. The PQ distributions are presented in Figure 3 (see Appendix of the revised manuscript) through boxplots. Notably, our method ADAPT exhibits strong resilience to changes in class ordering, consistently outperforming alternative methods.

Q9: Clarification on Reported Performance Metrics

A9: We acknowledge a discrepancy between our reproduced results for ECLIPSE and the metrics reported in their original work. This is because we cannot reproduce their reported results, especially for 100-x scenarios (2 percent gap). Therefore, for fairness, we report our reproduced results for ECLIPSE. For other methods, we report the official results for consistency. In our revised manuscript, we have added the official results for ECLIPSE to Table 1. This inclusion does not alter the demonstrated advantage of ADAPT.

I appreciate the authors for the comprehensive rebuttals. Most concerns are well-addressed, and the quality of the revised paper has improved greatly. I raised my rating.

** Please respond to the reviews from the reviewers c4nN and Bxb3 to solidify your rebuttal as soon as possible. **

We truly appreciate your positive feedback on the clarity of the figures, as well as on our novel probability-level fusion (PLF) and query-level fusion (QLF) strategy. Your constructive comments have been instrumental in refining our manuscript. Below, we provide detailed responses to the concerns raised:

Weaknesses

Q1: Lack of evidence for computational costs

A1: The table below (Table 6 in the revised manuscript) highlights the efficiency of our approach in both training and inference. During incremental training, ADAPT requires only 2.27 hours for five training steps, representing a reduction of 34.8% and 32.4% in training time compared to CoMFormer and ECLIPSE, respectively. The reduced training time can be attributed to ADAPT's frozen- and shared-weight design between the teacher and student models, which eliminates the need for dual-model forward passes for knowledge distillation in CoMFormer. Additionally, our efficient adaptation strategy enables a fourfold reduction in the number of training iterations for each step, decreasing from 16k in ECLIPSE to 4k in our approach. This substantial reduction contributes to the overall decrease in training time.

For inference, our method incurs a modest 6.5% increase in FLOPs compared to ECLIPSE. However, this is accompanied by substantial performance gains, particularly in novel PQ, which improves from 16.5 to 26.3—a remarkable 59.4% increase. Similar improvements are observed when compared to CoMFormer. Furthermore, the inference speed of ADAPT remains comparable to that of CoMFormer, highlighting the efficiency and practicality of our approach. In contrast, ECLIPSE demonstrates slower inference speeds, likely attributable to its unoptimized implementation. We have added the efficiency analysis to Sec. 4.5.5.

Q2: Limited Experimental Validation (Generalization)

A2: Following ECLIPSE, we conduct experiments using two settings (83-5, 83-10) on the COCO benchmark. During this process, we identified and fixed two bugs in their implementation: 1) the class indices in COCO start from zero, while their code assumed they start from one, leading to misalignment; and 2) their result summarization for base and novel classes did not account for class order shuffling. After correcting these issues, we obtained the results in the table below (Table 7 of the revised manuscript), where our approach significantly outperforms other competing methods.

Q3: Need for Further Improvement in Table Presentation

A3: Thank you for your suggestion. We have adjusted the table size to ensure consistent font size across all tables. Additionally, we moved Table 5 from the original submission to Table 2 in the revised manuscript so that it is positioned closer to the referencing paragraph in Sec 4.5.1 Overall Study. To enhance compactness and improve the overall layout, we have placed Table 4 and Table 5 side-by-side horizontally in the revised manuscript.

Questions

Q4: Does the "raising computational costs [Line 057-058]" only occur during training? If so, does the computational cost of inference remain the same?

A4: Yes, the dual-model forward pass for the teacher and student models in CoMFormer only occurs during training, while the inference cost remains roughly the same with base model Mask2Former. However, we emphasize that the overhead brought by dual-model forward is considerable, leading to a 48% increase in training time (ADAPT 2.27 hours vs. CoMFormer 3.36 hours) compared to our frozen- and shared-weight self-distillation mechanism (ADAPT), under the ADE20K 100-10 scenario. For a more detailed efficiency comparison, please refer to A1.

Q5: Why does continuously introducing additional learnable query features and embeddings for new tasks constrain the model's capacity to generalize to new tasks due to restricted plasticity?

A5: ECLIPSE freezes most of the model’s weights and introduces learnable query features and embeddings to adapt to new classes. However, we observe that this approach limits the model's ability to generalize, particularly when new tasks come with large amounts of training data. For instance, under ADE20K 100-50 setting, which includes more training data per step compared to 100-10 and 100-5, ECLIPSE performs poorly (23.5 PQ) on novel classes compared to CoMFormer (27.7 PQ), which finetunes all model weights. Notably, in Table 1(a) of the ECLIPSE paper, their reported results for CoMFormer are lower than the official results for the ADE20K 100-50 setting. For fairness and consistency, we present the official results in the table below. We observe that the performance gap on novel PQ between ECLIPSE and CoMFormer becomes even more pronounced (21.3 PQ vs. 26.6 PQ for 83-10) on the COCO benchmark, which contains significantly more training data (118K images) compared to ADE20K (25K images). These observations suggest that ECLIPSE’s reliance on a limited number of trainable parameters restricts its generalization capacity, especially in tasks with diverse classes and large-scale training data.

Q6: Experimental evidence for the claim "most of these methods require separate forward passes for the teacher and student models, resulting in considerable computational overhead."

A6: This claim applies to various continual learning methods across domains such as classification, object detection, semantic segmentation, and panoptic segmentation, provided the approach employs a dual-model forward pass during distillation. In the context of panoptic segmentation, we compare our ADAPT method to CoMFormer, which uses a dual-model forward pass. Since training computational costs include both forward and backward passes, we evaluate efficiency based on the total training time rather than FLOPs, which is typically used for assessing inference costs. In our comparison, with the same number of iterations (4k for each step in the ADE20K 100-10 setting), we find that the dual-model forward pass significantly increases training time, leading to a 48% increase in training time (ADAPT: 2.27 hours vs. CoMFormer: 3.36 hours). This increase in training time can be attributed to the additional computational cost of managing two model forward passes (teacher and student models). For a more holistic analysis of the efficiency, please refer to A1.

Q7: At least two datasets should be involved to validate the generalization ability of the proposed method.

A7: In addition to ADE20K, we have conducted experiments on the COCO dataset (see A2) to further validate the generalization ability of our proposed method.

Q8: It is recommended to adjust the table size and position distribution.

A8: Thank you for your suggestion. Please refer to A3.

Dear Reviewer Bxb3,

This is a gentle reminder that the discussion phase will end in 8 hours. If you have any remaining concerns or questions, please feel free to let us know. We welcome any further discussion and sincerely appreciate your efforts.

Best regards!

Dear Reviewer Bxb3,

We regret not hearing from you during the author-reviewer discussion period. We understand that this might sometimes happen due to unforeseen circumstances, and we sincerely hope everything is well on your end.

We kindly encourage you to review our responses to your concerns at your earliest convenience. We believe our detailed answers address the issues raised and highlight the improvements made to the manuscript. If possible, we would greatly appreciate it if you could finalize your rating after considering our response.

Thank you for your time and efforts.

Best regards!

We sincerely appreciate your positive feedback on our impressive performance, high efficiency, and comprehensive ablation studies. Your constructive comments have been invaluable in enhancing the quality of our manuscript. Below, we provide detailed responses to the concerns you raised:

Weaknesses

Q1: Implementation details of query-level fusion (QLF)

A1: Our QLF is a simple yet effective fusion strategy. As illustrated in Figure 2, we zero out the base-class probabilities $p_{i,c}^t (c\in \mathcal{C}^0)$ (represented by the blue bar in the bottom-right distribution of Figure 2) predicted by the base decoder $\mathcal{M}_\text{trans}^t$. This ensures that QLF follows two intuitive rules:

• The adapted decoder $\mathcal{M}_\text{trans}^t$ only predicts novel classes, since base class probabilities are zeroed out.

• Conversely, $\mathcal{M}_\text{trans}^0$ is solely responsible for predicting the base classes, as no novel scores are predicted by this decoder.

For illustration, both the base and adapted decoders each generate class distribution predictions for 4 queries. The standard panoptic inference fusion strategy from Mask2Former is then applied to combine the predictions from these 8 queries. During fusion, duplicate segments with high overlap are suppressed, similar to the Non-Maximum Suppression (NMS) used in object detection. The detailed fusion procedures are now outlined in Algorithm 1 in the Appendix of the revised manuscript.

Q2: I don‘t understand the necessity of subsection 4.5. The first table referenced in the text is actually Table 5, which makes it a bit confusing.

A2: Thank you for bringing this to our attention. We apologize for the layout mistake. To clarify, subsection 4.5 "Ablation Study" actually consists of several subsubsections, namely 4.5.1 "Overall Study," 4.5.2 "Adaptation Strategy," 4.5.3 "Attentive Self-Distillation," 4.5.4 "Dual-Decoder Prediction Fusion," and 4.5.5 "Efficiency," which were originally mistakenly treated as separate subsections.

Additionally, we moved Table 5 from the original submission to Table 2 in the revised manuscript so that it is positioned closer to the referencing paragraph in Sec 4.5.1 Overall Study. To enhance compactness and improve the overall layout, we have placed Table 4 and Table 5 side-by-side horizontally in the revised manuscript. Finally, we have adjusted the table size to ensure consistent font size across all tables.

Q3: Training and inference efficiency analysis

A3: The table below (Table 6 in the revised manuscript) highlights the efficiency of our approach in both training and inference. During incremental training, ADAPT requires only 2.27 hours for five training steps, representing a reduction of 34.8% and 32.4% in training time compared to CoMFormer and ECLIPSE, respectively. The reduced training time can be attributed to ADAPT's frozen- and shared-weight design between the teacher and student models, which eliminates the need for dual-model forward passes for knowledge distillation in CoMFormer. Additionally, our efficient adaptation strategy enables a fourfold reduction in the number of training iterations for each step, decreasing from 16k in ECLIPSE to 4k in our approach. This substantial reduction contributes to the overall decrease in training time.

For inference, our method incurs a modest 6.5% increase in FLOPs compared to ECLIPSE. However, this is accompanied by substantial performance gains, particularly in novel PQ, which improves from 16.5 to 26.3—a remarkable 59.4% increase. Similar improvements are observed when compared to CoMFormer. Furthermore, the inference speed of ADAPT remains comparable to that of CoMFormer, highlighting the efficiency and practicality of our approach. In contrast, ECLIPSE demonstrates slower inference speeds, likely attributable to its unoptimized implementation. We have added the efficiency analysis to Sec. 4.5.5.

Questions

Q6: Would the open-vocabulary panoptic segmentation method [4] solve some technical challenges in continual panoptic segmentation (e.g. adding new classes)? Perhaps this approach trained in a regular way could represent another baseline for CPS?

A6: We thank the reviewer for bringing this approach [4] for open-vocabulary panoptic segmentation to our attention. [4] introduces an out-vocab branch, allowing the model to handle open vocabularies. However, we believe this approach is NOT suitable as a baseline for continual panoptic segmentation (CPS).

If we train the in-vocab branch using novel data at each step in a conventional manner, this setup becomes functionally identical to the Mask2Former framework, which is essentially fine-tuning (FT) in Table 1. On the other hand, if we utilize the out-vocab branch for the geometric ensemble (Figure 3 in [4]) to handle novel classes, the approach relies on additional information from the pretrained CLIP text encoder. This external knowledge makes a direct comparison unfair, as other CPS methods [2-3], including ours, do NOT have access to such extra information.

Nevertheless, we acknowledge that open-vocabulary learning is an interesting topic, particularly in the context of large foundation models. Given this, we will cite [4] and discuss it in the related work section of our final manuscript for completeness.

[2] Cermelli, Fabio, Matthieu Cord, and Arthur Douillard. "Comformer: Continual learning in semantic and panoptic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Kim, Beomyoung, Joonsang Yu, and Sung Ju Hwang. "ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Yu, Q., He, J., Deng, X., Shen, X., & Chen, L. C. (2023). Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36, 32215-32234.

Q4: Is the setup where most of the data and classes become available in some intermediate step possible in real applications?

A4: In real-world applications, it is uncommon for most data and classes to become available only at an intermediate stage. Typically, models are deployed after being trained on a sufficient scale of data to ensure robustness, and new data or classes are introduced gradually over time. However, if a situation does arise where a substantial portion of data and classes is introduced only at a later stage, it may be more effective to retrain the model from scratch using all available data. This is because early-stage training on limited data might NOT provide robust knowledge for transferring to new tasks, potentially leading to suboptimal performance. Moreover, since the early-stage training does NOT involve large-scale data, storing these old data and retraining a new model from scratch along with new data is affordable and cost-efficient.

Q5: The technical novelty of the presented contributions is limited. Self-distillation and weight freezing have already been considered in continual panoptic segmentation.

A5: We respectfully disagree with the claim that the novelty of our contributions is limited. While self-distillation and weight freezing have been explored in prior works like CoMFormer and ECLIPSE, our approach introduces key advancements in both methodology and efficiency.

First, weight freezing in continual panoptic segmentation is non-trivial. In ECLIPSE, freezing most of the model weights limits generalization, especially when new tasks involve large amounts of training data (see Reviewer Bxb3 Q5). Conversely, updating all parameters, as in CoMFormer, risks poor knowledge retention due to catastrophic forgetting. Our comprehensive ablation studies in Table 3 show that our freezing strategy (only fine-tuning cross-attention layers and FFNs) strikes an effective balance between plasticity (learning new tasks) and rigidity (retaining old knowledge). Additionally, we provide a theoretical hypothesis for the effectiveness of our adaptation strategy (see Reviewer fZFv Q1).

Regarding self-distillation, while it has been employed in CoMFormer, CoMFormer suffers from significant computational overhead due to the dual-model forward pass during knowledge distillation (KD). We address this by introducing a frozen- and shared-weight strategy, eliminating the need for dual-model forward passes and reducing training time by 34.8% on ADE20K 100-10 (as detailed in Q3). Unlike CoMFormer, which applies unbiased KD [1] to all queries, our approach selectively applies our attentive KD to queries that are not matched with any new classes. Notably, our attentive KD incorporates a modulating term $\alpha(1-\hat{p}_x^{t-1} (i,\emptyset))^\gamma$ to emphasize informative queries and down-weight less useful ones based on background confidence (see Sec. 3.3 for details). These two mechanisms facilitate more efficient and effective retention of old knowledge, alleviating the natural bias toward current novel classes.

Finally, our method ADAPT consistently achieves state-of-the-art results on ADE20K and COCO benchmarks across various settings with high efficiency. We believe these contributions (acknowledged by Reviewer fZFv with confidence 5) are significant and provide a meaningful improvement over existing methods [2-3] in terms of both performance and efficiency.

[1] Cermelli, Fabio, et al. "Modeling the background for incremental learning in semantic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[2] Cermelli, Fabio, Matthieu Cord, and Arthur Douillard. "Comformer: Continual learning in semantic and panoptic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Kim, Beomyoung, Joonsang Yu, and Sung Ju Hwang. "ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Hi Reviewer c4nN,

Could you kindly re-evaluate the confidence and contribution scores? We believe our responses have addressed your concerns and improved your confidence in the contribution of our work. Thank you.

Best Regards,

The Authors

We’re glad to hear that our responses have addressed your concerns, and we truly appreciate your increased confidence in the paper. We are grateful for your improved recognition, even though the rating system does NOT allow for a 1-point jump to 7.

We sincerely appreciate your constructive feedback and thoughtful suggestions, which have helped improve our work.

Best wishes.

We sincerely appreciate your positive feedback on our dual-decoder design, self-distillation mechanism, and query-level fusion (QLF) strategy. Your constructive comments have been invaluable in refining our manuscript. Below, we provide detailed responses to address the concerns raised:

Q1: Freezing the encoder may limit the model's ability to adapt fully to new classes over long task sequences, especially as the diversity or complexity of new classes increases.

A1: We compare the performance of freezing the encoder (ADAPT) versus leaving it unfrozen (Unfrozen encoder) in long-sequence tasks, specifically on the 100-5 (11 tasks) setting on ADE20K and 83-5 (11 tasks) on COCO, where COCO involves more complex scenes. The results below indicate that freezing the encoder does not hinder the model’s generalization capacity. In fact, the transformer decoder itself remains highly adaptable to new tasks. In contrast, unfreezing the encoder results in catastrophic forgetting and renders our shared self-distillation mechanism not applicable. As a consequence, a dual-model forward pass (teacher and student models) becomes necessary for distillation, similar to CoMFormer. This incurs additional computational costs during training.

Q2: The freezing and distillation mechanisms might still impose computational and memory limitations in more demanding setups. How does the method scale with larger models or higher-resolution images?

A2: The table below (Table 6 in the revised manuscript) highlights the efficiency of our approach in both training and inference. During training, we first clarify that the freezing mechanism actually reduces computational costs, as no gradients are computed for the frozen weights during backpropagation. Compared to CoMFormer, our distillation mechanism shares the computation up to the transformer decoder, resulting in 1.09 hours (34.8%) less training time. While our per-iteration FLOPs are slightly higher than ECLIPSE due to adaptable self-attention layers and FFNs, our efficient adaptation strategy enables a fourfold reduction in training iterations per step (from 16k in ECLIPSE to 4k in our approach). This results in an overall reduction in training time (ADAPT: 2.27 hours vs. ECLIPSE 3.48 hours).

For inference, our method incurs a modest 6.5% increase in FLOPs compared to ECLIPSE. However, this is accompanied by substantial performance gains, particularly in novel PQ, which improves from 16.5 to 26.3—a remarkable 59.4% increase. Similar improvements are observed when compared to CoMFormer. Furthermore, the inference speed of ADAPT remains comparable to that of CoMFormer, highlighting the efficiency and practicality of our approach. In contrast, ECLIPSE demonstrates slower inference speeds, likely attributable to its unoptimized implementation. We have added the efficiency analysis to Sec. 4.5.5.

Scalability: We note that the scaling behavior for CoMFormer, ECLIPSE, and ADAPT is consistent when using larger backbone models or higher-resolution images. This is because all three methods share the same architecture of the encoder and pixel decoder, which can encode the input images of various resolutions into feature maps of consistent resolution. Therefore, it’s feasible to measure the efficiency in the conventional setting as in prior works CoMFormer and ECLIPSE to ensure consistency.

Q3: Evaluation on the COCO dataset and the open-vocab setting.

A3: Following ECLIPSE, we conduct experiments using two settings (83-5, 83-10) on the COCO benchmark. During this process, we identified and fixed two bugs in their implementation: 1) the class indices in COCO start from zero, while their code assumed they start from one, leading to misalignment; and 2) their result summarization for base and novel classes did not account for class order shuffling. After correcting these issues, we obtained the results in the table below (Table 7 of the revised manuscript), where our approach significantly outperforms other competing methods.

Open-vocabulary models are designed to generalize to unseen classes during inference, typically using text prompts or semantic embeddings. In contrast, continual learning models focus on retaining knowledge from previously seen tasks while adapting to new ones, which do NOT gain the ability to recognize open classes without adaptation. Given the distinct objectives of these two types of models, it is not appropriate to directly compare them. This probably explains why prior works CoMFormer and ECLIPSE did NOT validate under open-vocabulary settings. Nevertheless, we acknowledge that open-vocabulary learning is an important topic, particularly in the context of large foundation models. We are interested in exploring and investigating open-vocabulary learning in our future research.