Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector

Thanks for the insightful comments.

Q1: Dataset concern.

R1: To show the generalizability of our key findings, we conducted experiments on a wildly used remote sensing detection datasets DIOR. The three key findings still hold with different two-stage IODs. As shown in this link DIOR, all curves are aligned with VOC dataset. Our results show that the detector's RPN struggles with unseen data, as indicated by the red curves. However, the detector performs well on seen data, suggesting the RPN and the RoI Head's regressor remain resilient and do not forget previously learned knowledge on DIOR.

We also conducted experiments on DIOR with our framework, as shown in

Our framework surpass the baseline by 2.9% in 5-5 setting and 1.8% in 10-10 setting, further showing the effectiveness of framework.

Q2: NSGP's novelty concern.

R2: Although NSGP has been explored in incremental classification, it is non-trivial to apply NSGP in IOD. The following table demonstrates the performance applying NSGP from Backbone to RoI Head accumulatively under VOC 5-5 setting.

The catastrophic forgetting in two-stage detectors is mainly caused by the severe classifier instability in the RoI Head. However, directly appling NSGP on these two-stage object detector (i.e., FPN, RPN,RoIHead) shows limited performance improvements and can not well address the classifier instability issue, as illustrated in the above Table.

Instead, the proposed Regional Prototype Replay (RePRE) module addresses this issue by replaying coarse and fine-grained region prototypes in the RoI Head's classification branch. NSGP serves as an assistant component in our framework by mitigating semantic drift caused by parameter updates, thereby preventing toxic replay in RePRE. As shown in the table, our RePRE achieves a +2.4% gain compared with +FPN, underscoring the effectiveness of our proposed framework.

Q3: Architecutre-specific concern.

R3: To show the generalizability of our key findings, we also conducted experiments with two popular two-stage detectors, i.e. Cascade-RCNN and Vanilla Faster RCNN. The three key findings still hold with different two-stage IODs as shown in these links CascadeRCNN and VanillaFasterRCNN. Detailed discussion can also be found in our response to reviewer huSY R3.

Q4: Insufficient ablation concern.

R4: Thanks for the suggestion. We conducted ablation study in VOC 10-10 setting as shown in

The same conclusion can be drawn from this table as in ablation study in 5-5 setting.

Q5: Visualization of the results.

R5: As shown in this link visualization, we visualized images from VOC2007 test set under 10-10 setting. Task 1 represents visualization with model from time step 1 while baseline and NSGP-RePRE represents the visualization results are from time step 2 trained with corresponding strategy. In (a) baseline forgets "boat". In (b) and (c), baseline forgets "cat" and "car" due to the interference of new classes "dog" and "motorbike". Our NSGP-RePRE successfully remembers old classes while learning new classes effectively, suggesting that our methods achieves better stability while retaining comparable plasticity compared with baseline.

Q6: The pseudo labeling concerns

R6: One major problem in IOD is that objects from past tasks can be included in the subsequent tasks yet their labels are not annotated. For example, airplanes will be labeled as "airplane" in the first task but considered as background in the subsequent tasks. Optimizing with wrong label will leads to a drastic performance drop. Pseudo labeling(Mo et al.,2024; Liu et al.,2023;) are wildly adopted to alleviate such performance drop. Thus, we also choose pseudo labeling as our baseline to avoid this problem.

We also conducted experiments under the setting of without pseudo label. As shown in

Our NSGP-RePRE achieves +20% performance gain compared with baseline, demonstrating the effectiveness of our framework.

We sincerely appreciate the reviewer's positvie feedback and insightful comment. The Questions and Responses are as follows.

Q1: More theorical analysis, such as complexity analysis can be provided.

R1: Thanks for the suggestions. We provide a comprehensive analysis of the parameter, computational, and memory complexity of NSGP-RePRE:

Parameter Analysis:

The origional detectors contains a total of 96.89 M trainable parameters. Our approach maintains the same parameter count, ensuring no increase in model complexity.

Computational Complexity:

To assess the computational complexity during training, we present the FLOPs for key components of the model, as shown in the table below:

It is important to note that the computational complexities of RePRE and NSGP do not scale with batch size, whereas the forward and backward passes of the detector do. As shown in the table, the majority of the computational cost during training arises from the detector's forward and backward passes. In contrast, RePRE and NSGP contribute only about 1% of the additional computation compared to the forward and backward operations when scaling batch size to commonly used 8.

We also present the actual training time for different components on 1 RTX 3090 GPUs in the following table:

This table suggests that the additional computational cost of NSGP-RePRE is minimal, adding only ~1% overhead per iteration compared to baseline training, which is aligned with our complexity analysis.

Memory cost of RePRE:

The memory footprint of our RePRE scales linearly with the increase of the number of classes. In our implementation, each class consumes approximately 3.8Mb of memory in float32 with each prototype consuming 0.38Mb. Our method achieves performance comparable to the previous example-based SOTA method, ABR, using only one coarse prototype per class and without relying on NSGP in the 10-10 setting. Additionally, it consumes about 25% of the memory required by ABR. As shown in

Thanks for the insightful comments.

Q0: On the Avg metric.

R0: We show the Avg performance at every step.

All represents the mAP of all seen classes, e.g. 1-15 in 5-5 setting step 3. Our method achieves the best Avg and All mAP at every time step, showing its superior balanced stability-plasticity properties.

Q1 & Q6:On the plasticity-stability trade-off.

R1: The plasticity-stability trade-off is explicitly controlled by the nullity of the uncentered feature covariance. In our experiments, we adopt an adaptive approach VPT-NSP^2(Lu e al.,2024) and achieved better trade-off.

Q2: On the scaling of memory requirement and prototype interference risk.

R2: The memory footprint of our RePRE scales linearly with increasing number of classes. Each class consumes approximately 3.8Mb, and each prototype consumes 0.38Mb. The comparison between our method and previous exampler-based SOTA method ABR is shown below.

We address redundancy by setting a distance between prototypes to capture the whole distribution of the feature space. Our results show consistent gains, suggesting effective handling of similar classes.

Q3: conclusion on other two-stage architectures.

R3: To show the generalizability of our key findings, we also conducted experiments with two popular two-stage detectors, i.e. Cascade-RCNN and Vanilla Faster RCNN without FPN and RoI Align. The three key findings still hold with different two-stage IODs, as shown in these links CascadeRCNN and VanillaFasterRCNN. We conducted the same anotomy to these detectors in our paper, all curves are aligned with Faster RCNN using Pascal VOC 5-5. These two detectors' RPN Recall curve show that seen models even can surpass current model, suggesting that our findings are universal in most two-stage IODs.

We also evaluated our method on these detectors.

Our method achieved noticeable performance improvement.

Q4: Concerns about pseudo-labeling.

R4: One major problem in IOD is that objects from past tasks can be included in the subsequent tasks yet their labels are not annotated. Optimizing with wrong label will lead to a drastic performance drop. Pseudo labeling(Mo et al.,2024; Liu et al.,2023;) are widely adopted to alleviate such performance drop. Thus, we also choose pseudo labeling as our baseline.

We also conducted experiments with pseudo label been ignored(neither foreground nor background). As shown in

Our NSGP-RePRE achieves +20% performance gain compared with baseline under the settings of without pseudo label module, demonstrating the effectiveness of our framework.

Q5: How NSGP affect training time.

R5: Our NSGP-RePRE maintains high efficiency. NSGP introduces additional training time by SVD decomposition and null-space projection. Both SVD and projection's required time only increases with the size of the model. For SVD, it only computes once at every incremental step and only took around 30 seconds in our experiments, much less than the whole training time. As for null-space projection, it also adds only ~1% overhead per iteration compared to baseline training as shown in

NSGP-RePRE does not introduces extra overhead during inference, which makes it practical for real-time application.

We sincerely appreciate the reviewer's positvie feedback and insightful comment. The Questions and Responses are as follows.

Q1: Using NSGP has been explored in the literature for continual learning.

R1:Although NSGP has been explored in incremental classification, it is non-trivial to apply NSGP in IOD. The following table demonstrates the performance applying NSGP from Backbone to RoI Head accumulatively under VOC 5-5 setting.

The catastrophic forgetting in two-stage detectors is mainly caused by the severe classifier instability in the RoI Head. However, directly appling NSGP on these two-stage object detector (i.e., FPN, RPN,RoIHead) shows limited performance improvements and can not well address the classifier instability issue, as illustrated in the above Table.

Instead, the proposed Regional Prototype Replay (RePRE) module addresses this issue by replaying coarse and fine-grained region prototypes in the RoI Head's classification branch. NSGP serves as an assistant component in our framework by mitigating semantic drift caused by parameter updates, thereby preventing toxic replay in RePRE. As shown in the table, our RePRE achieves a +2.4% gain compared with +FPN, underscoring the effectiveness of our proposed framework.

Q2: Although the specific form of replying regional prototype has not been explored, the idea of prototype based continual learning has been proposed in the literature. Online Prototype Learning for Online Continual Learning, ICCV 2023 Prototype-Guided Memory Replay for Continual Learning, IEEE TNNLS 2024

R2: While naively applying prototype-based methods on the classifier does achieves performance gain for the detector, our work makes distinct contributions in the context of IOD as we considered RoI Head's uniqueness in pre-processing MLPs before classifier. Further, we extend our method to fine-grained regional prototype replay to capture the distribution of regional object features, which is crucial for preserving old knowledge in continual learning.

To validate our design, we compare NSGP-RePRE against a baseline ("Classifier") that applies prototype replay only to the classifier (mimicking classification-focused approaches):

As shown in the table, our method outperforms "w/o Prototype" by +2.4%(5-5) and +1.5%(10-10). Though Classifier achieves performance gain compared with w/o Prototype, our method outperforms "Classifier" by +2.3% (5-5) and +0.8% (10-10). These results demonstrating that regulating only the classifier (as in classification-based works) is insufficient for IOD.

In addition, we also find our finding still holds across two extra two-stage detectors and a remote sensing dataset, which further highlights the reliability of our findings and the effectiveness of our framework. The reviewer may find them in our response to Reviewer huSY(Q3, architecture) and Reviewer K17L(Q1,dataset).

We emphasize that our goal is not to propose an overly complex method, but to offer a simple yet effective solution grounded in our analysis of catastrophic forgetting in two-stage detectors. We hope this work contributes to bridging the gap between continual learning for classification and detection.

In the end, we like to thank the reviewer for their valuable suggestions and questions.