Cut out and Replay: A Simple yet Versatile Strategy for Multi-Label Online Continual Learning

Respected Reviewer Nhoa, we first thank you for your valuable and insightful feedback, and for recognizing the analysis and advantages of our proposed method. Below, we address your concerns in a point-by-point manner.

Q: Experiments on other incremental settings should be included. A: We appreciate this suggestion. We have conducted additional experiments on different incremental settings following protocols established in previous works like KRT and OCDM. Specifically, we evaluated our approach on class-incremental scenarios with varying numbers of classes per stage:

MSCOCO B40-C10:
In this setting, we first train a base model on 40 classes, then incrementally add the remaining 40 classes over 4 sessions (10 classes per session).

NUSWIDE B41-C5:
Here, we start with 41 base classes and incrementally learn 8 sessions with 5 new classes each.

These results demonstrate that our CUTER method consistently outperforms existing approaches across different incremental settings, particularly in later stages where catastrophic forgetting is most pronounced.

Q: Method in continual object detection should be compared and discussed.
A:Thank you for your insightful comment regarding the similarities between our proposed cut-out and replay framework and continual object detection. We have conducted a preliminary literature search on this topic and will include a thorough comparison in the related works section, especially with reference [A], which adopts an approach similar to our CUTER framework.

However, we would like to clarify that methods from continual object detection cannot be directly applied to MOCL (Multi-Object Continual Learning). This is because these methods fundamentally rely on labeled bounding boxes which, even if partially missing, enable the training of an additional detection head. Such a strategy is clearly challenging to implement in the MOCL setting where such annotations are unavailable.

To validate this point, we conducted a comparative experiment on the VOC dataset between our method and a DETR detection head using the box replay approach from [A] (the initial box was obtained by NCut). The results, as shown in the table below, demonstrate the advantages of our approach in the MOCL context where bounding box annotations are not accessible.

[A] Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detection. ICCV 2023.

Q: The analysis of different regularization terms warrants inclusion in the main text.
A: We agree this analysis provides important insights into our method. In the revised version, we'll include a concise comparison of regularization terms in the main text, explaining why nuclear norm regularization outperforms sparse and smooth approaches for preserving localization capabilities in MOCL. This addition will strengthen the theoretical justification for our design choices while maintaining the paper's focus.

Q: Ablation study could be enriched by including runtime comparisons between other methods and the proposed re-balanced sampling strategy.
A: We have enriched our ablation study by including runtime comparisons between our proposed re-balanced sampling strategy and other methods. For a fair comparison, we directly omit the CUT out process and measure the throughput in terms of samples processed per second:

Our proposed method delivers a strong throughput of 90.9 samples per second, which is considerably more efficient than both OCDM and PRS.

Q: Typos.
A: We feel sorry for any confusion or inconvenience may have caused. In the revised version, we will carefully proof-read our text to ensure that no grammar mistakes or typos still exist.

Respected Reviewer NiGx, we thank you for your valuable and insightful feedback. Below, we address your concerns in a point-by-point manner.

Q: Additional experiments with varying numbers of classes per stage should be included.
A: We appreciate this suggestion. We have followed the setting in previous works like KRT and OCDM to provide class-incremental settings with varying numbers of classes per stage. Due to the limited space, please refer to our response to Reviewer Nhoa's Q1 for complete tables.

MSCOCO B40-C10:
In this setting, we first train a base model on 40 classes, then incrementally add the remaining 40 classes over 4 sessions (10 classes per session).

NUSWIDE B41-C5:
Here, we start with 41 base classes and incrementally learn 8 sessions with 5 new classes each.

These results demonstrate that our CUTER method consistently outperforms existing approaches across different incremental settings, particularly in later stages where catastrophic forgetting is most pronounced.

Q: Additional spectral clustering literature should be discussed and cited.
A: We appreciate this suggestion. In the revised version, we will expand our related work to include seminal spectral clustering works such as Shi and Malik (2000) on normalized cuts, Von Luxburg's (2007) tutorial on spectral clustering fundamentals, and more recent advancements like Tang et al. (2018) on robust spectral clustering. We'll also discuss image segmentation applications of spectral clustering beyond MCut, including works like LSC by Li and Chen (CVPR 2015) and the analysis of limitations of these spectral-based segmentation methods by Boaz Nadler (NIPS 2006).

Q: Whether other detection methods could be integrated into the MOCL
A: Yes, our CUTER framework is designed with modularity in mind, allowing for integration of various detection methods beyond MCut. However, we want to emphasize that in the online continual learning scenario, MCut's advantage of requiring no training is significant. While other detection methods like LOST, TokenCut, or attention-based approaches could theoretically be integrated, they would need to overcome these online learning constraints.

Additionally, methods like SAM could be a viable choice, but without additional prompts, SAM typically produces many more segmentation masks than required for MOCL objectives. Its segmentations tend to be relatively fragmented because SAM's results are not inherently class-oriented or semantically guided. This would introduce additional challenges in establishing the correct correspondence between regions and labels.

We greatly appreciate the reviewer's suggestions and will carefully discuss these considerations in a future version of our work.

Q: Additional explanation is needed to clarify why the proposed regularization terms work and outperform conventional approaches like smoothing or sparse regularizations.
A: To clarify why nuclear norm regularization outperforms conventional approaches: While all regularization methods in Table 3 are established in graph structure learning, they differ fundamentally in their effects on representation learning for our task.

Nuclear norm regularization promotes low-rank structure in the graph Laplacian, better preserving the intrinsic manifold structure of ViT features while removing noise $\epsilon$ as established in Theorem 2.3. In contrast, sparse regularization, though effective for denoising, penalizes hypernode patches with naturally high cross-node similarity, disrupting inherent structural properties of ViT parameters and compromising classification capacity (Table 3). Similarly, smooth regularization forces excessive feature similarity across nodes, impairing the spectral clustering processes essential for effective localization.

Q: The model's performance in offline setting should be included.
A:Despite the offline setting not being the primary goal of our proposed method, we make a simple comparison with KRT (method designed for offline setting) on MSCOCO:

These results suggest that CUTER's design principles are broadly applicable across different multi-label classification settings.

Respected Reviewer eZ9b, we first thank you for your valuable and insightful feedback, and for recognizing our motivation and theoretical analysis. Below, we address your concerns in a point-by-point manner.

Q: How the the accuracy and reliability of proposed localization process is ensured?
A: Our approach ensures the accuracy and reliability of the proposed localization process through three key mechanisms:

1. Principled Model Selection: We conducted detailed analysis of the localization potentials across different pre-trained models, establishing general selection principles that identify models with optimal localization capabilities for MOCL tasks.

2. Selective Label-Region Matching Strategy: We developed a targeted matching approach that accurately aligns semantic regions with corresponding labels, minimizing false correlations and enhancing localization precision.

3. Spectral-based Regularization: To prevent degradation of localization capacity during MOCL progression, we implemented a specialized regularization term that preserves the model's ability to accurately identify regions of interest over time.

The effectiveness of these mechanisms is substantiated by both qualitative evidence (visualizations in Figures 3 and 5 demonstrating accurate region identification) and quantitative validation (ablation studies in Table 6 showing the benefits of our Cut-out Replay strategy in capturing fine-grained spatial information). Together, these elements form a comprehensive framework that consistently delivers accurate and reliable localization performance throughout the MOCL process.

Q: Performing cut-out operations and regularization terms introduce additional computational overhead.
A: We acknowledge that our approach introduces some additional training time. In fact, as shown in Figure 8 (left) in the Appendix, when not using regularization terms, CUTER achieves relatively comparable model throughput to other popular MOCL methods. Additionally, we recognize that the proposed regularization terms do cause a considerable decrease in model throughput. Finding ways to accelerate this process and achieve a better balance between performance and computational efficiency will be a focus of our future work.

Q: Nuclear norm regularization details spread across appendix and main text, affecting clarity.
A: We appreciate the reviewer's valid point about the nuclear norm regularization presentation. The current separation was primarily due to page constraints. In the revised version, we will restructure Section 2.3 to present a more cohesive narrative by:
(1) Introducing the theoretical motivation behind nuclear norm regularization;
(2) Connecting it directly to the derived graph Laplacians with concise mathematical formulations;
(3) Providing intuitive explanations of how this regularization enhances representation learning;
(4) Briefly highlighting comparative advantages over alternative approaches like smooth regularization or sparse regularization on the graph laplacian before the experimental section;

Q: Typos.
A: We feel sorry for any confusion or inconvenience may have caused. In the revised version, we will carefully proof-read our text to ensure that no grammar mistakes or typos still exist.