Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection

Weekness 1

Reviewer Concern: The key contribution of this paper is RDB and ZiL. However, RDB has been proposed by [1]. Therefore, I would owe the novelty to the differentiated learning rate and ZiL (basically L1 norm). From this point of view, the novelty is somewhat incremental. Importantly, the authors should cite [1] and highlight the distinction.

Response: Thank you for your insightful feedback. We acknowledge the prior work on Reparameterizable Dual Branch (RDB) and the need to clearly distinguish our contributions. We will ensure that [1] is appropriately cited in our manuscript.

The primary novelty of our work lies in the differentiated learning rate and the introduction of Zero-interference Loss (ZiL), which are key enhancements over the existing RDB framework. These innovations address the specific challenges of Incremental Vision-Language Object Detection (IVLOD) by balancing incremental learning performance and zero-shot capabilities more effectively.

Revised Explanation in Manuscript: "Our work builds upon the Reparameterizable Dual Branch (RDB) concept initially proposed by [1]. The key innovations introduced in our approach include the use of differentiated learning rates for the High Learning Rate Branch (HLRB) and Low Learning Rate Branch (LLRB), and the implementation of Zero-interference Loss (ZiL). These enhancements allow for a more effective balance between learning new tasks and preserving previously acquired knowledge, distinguishing our method from prior work."

Weekness 2

Reviewer Concern: The paper only evaluates the general zero-shot performance after incremental learning. Is zero-shot specific-domain inference possible after learning on few examples in this specific domain?

Response: Thank you for your question. In our approach, we do not directly evaluate zero-shot inference in specific domains after learning on a few examples from those domains. Instead, our methodology involves tuning a pre-trained model on the ODinW dataset, which comprises 13 datasets, and then immediately evaluating the model on COCO to assess its zero-shot performance. This evaluation is conducted without exposing the model to any images from COCO during the incremental training phase.

Weekness 3

Reviewer Concern: It's suggested to include more comprehensive incremental object detection literature, including but not limited to [2-3].

Response: Thank you for your valuable suggestion. We acknowledge the importance of providing a comprehensive review of the literature on incremental object detection. In the revised manuscript, we will include a detailed survey of relevant work, including but not limited to the references [2-3] you mentioned.

Question1

Reviewer Concern: Is TFA also pre-trained on Objects365 [30], GoldG [17], and Cap4M? I have some doubts about the TFA ZCOCO performance in Table 1 if it's pretrained on COCO. To my best knowledge, TFA only finetunes the last classifier layer for the incremental steps and has demonstrated strong performance on the base classes.

Response: For consistency and fair comparison, all methods, including TFA, were re-implemented and evaluated using the same pre-training datasets (Objects365, GoldG, and Cap4M) and the same pre-trained Grounding DINO model.

Question 2

Reviewer Concern: How is general IOD (e.g. TFA) tested on Zero-shot COCO? General IOD generally does not support zero-shot inference. Is there any adaptation to achieve it?

Response: Thank you for your question. We acknowledge that general Incremental Object Detection (IOD) methods, such as TFA, do not inherently support zero-shot inference. To address this, we re-implemented these general IOD methods based on the Grounding DINO framework. By leveraging Grounding DINO, which supports zero-shot inference, we adapted these methods to enable zero-shot capabilities. Since all the compared baselines support a DETR-like architecture, re-implementing them based on Grounding DINO was feasible and straightforward. This approach ensures that all methods can be evaluated consistently within the same experimental setup.

Qusetion 3

Reviewer Concern: What learning rate is applied to w/ Rep+ (the first line of Table 2)? Is it the high or low one?

Response: Thank you for your question. We used a high learning rate applied to w/ Rep+ (the first line of Table 2). We also tested using a low learning rate and found that the results were similar to those obtained with the high learning rate. We infer that the absolute value of the learning rate does not significantly impact the performance of the RDB. Instead, the differential learning rates play a crucial role, allowing the High Learning Rate Branch (HLRB) and Low Learning Rate Branch (LLRB) to share different learning burdens effectively.

Here is the performance (w/ Rep+) using the low learning rate:

Weakness 1

Response: Thank you for this valuable feedback. We acknowledge the importance of clearly stating the need for this work and situating it within the context of existing research. In the revised manuscript, we will enhance the introduction and related work sections to better articulate the motivation for our study and to delineate the similarities and differences between our approach and previous methods.

Weakness 2

Response: Thank you for your feedback. We understand the importance of interpretability in demonstrating the effectiveness of our proposed approach. The core of our solution lies in constraining the Reparameterizable Dual Branch (RDB) with Zero-interference Loss (ZiL). ZiL serves as a regularization mechanism that minimizes interference from newly learned tasks on previously acquired tasks, ensuring that the model maintains its zero-shot capabilities while adapting to new data.

Weakness 3

Response: Thank you for highlighting the need for clearer descriptions of the metrics used in our experiments. The "ZCOCO" metric refers to the mean Average Precision (mAP) on all classes in the COCO dataset. Specifically, it represents the mAP@0.5:0.95 (the average of mAP at IoU thresholds from 0.5 to 0.95). This measure provides a comprehensive evaluation of the model's zero-shot performance across all classes on the COCO dataset.

In the revised manuscript, we will update the description of the metrics in Table 1 to explicitly state that "ZCOCO" refers to the mAP on all classes in the COCO dataset, measured at IoU thresholds from 0.5 to 0.95. This clarification will ensure that readers have a precise understanding of the evaluation criteria used in our experiments.

Questions 1

Response: Thank you for pointing out this important aspect. We recognize the need to clarify the relationship between Vision-Language Object Detection (VLOD) and open vocabulary/open set object detection. The VLOD task is indeed consistent with the principles of open vocabulary and open set object detection. VLOD aims to extend the capabilities of Vision-Language Models (VLMs) to detect objects from both known and unknown categories using natural language prompts. The primary reason for using the term VLOD is to emphasize the integration of vision and language in object detection tasks and to highlight the incremental learning aspect, which is a key focus of our work.

Questions 2

Response: Thank you for pointing out the need for clarification regarding zero-shot generalization and the handling of pre-training data. We acknowledge the importance of ensuring that the test set remains unseen during the pre-training phase to validate the zero-shot capabilities of our model.

To address this, we will clarify that we use the pre-trained weights of Grounding DINO, which are trained on O365, GoldG, and Cap4M datasets. These pre-training datasets are the same as those used for GLIP-T. This ensures that our model has not seen any images from the COCO dataset during pre-training, maintaining the integrity of the zero-shot generalization task.

Questions 3

Response: Thank you for your insightful question. Our work combines both incremental learning and zero-shot learning to address the challenges of open-world object detection (OWOD). Both OWOD and IVLOD need to detect seen and unseen objects. However, in IVLOD, models are pre-trained to detect unknown objects before the incremental learning process begins. In contrast, existing OWOD methods train the model's ability to detect unknown objects concurrently with incremental learning. In addition, OWOD does not further classify unknown objects. In contrast, IVLOD leverages VLODMs to classify unknown objects through language prompts, allowing for more precise identification and categorization of previously unseen objects. Given the superior zero-shot generalization capabilities provided by current vision-language models, we believe that starting with a pre-trained vision-language model for open-world detection represents a more effective approach compared to the traditional OWOD paradigm (which involves incrementally learning new objects while simultaneously learning open-set detection).

Questions 4

Response: We simultaneously insert RDB in both the vision and language components for learning. We believe that the parameters of RDB in different modalities require different update speeds. Therefore, we introduce a learnable scaling factor "s" to adjust the parameter update speeds for different modalities. In fact, rather than adjusting the scaling factor directly, we ensure HLRB is dominant by adjusting the learning rates of HLRB and LLRB. We use ZiL to constrain HLRB, mitigating the occurrence of forgetting.

Questions 5

Response: When we refer to "small-norm" random noise, we define it based on the standard deviation of the noise added to the input. Specifically, our experiments show that when the norm of the random noise is small (with a standard deviation (std) within 2), the model’s zero-shot AP on COCO remains above 45. Therefore, we claim that the model’s performance is not significantly affected by the addition of small-norm random noise because, within this range, the decrease in performance is minimal and the model retains a high level of accuracy. The observed drop in average AP by more than 30% occurs when the norm of the random noise exceeds this small-norm arrange. The ZiL can ensure that the RDB's output is within the small-norm arrange, therefore protects the the model’s zero-shot AP on COCO. In the revised manuscript, we will ensure that this definition and explanation are clearly stated to avoid any confusion.

Questions 6

Response: We use the same text template as Grounding DINO.

1. The new task of IVLOD is not well described and compared with existing ones

Response: Thank you for your insightful feedback. We recognize the need to clearly define and differentiate Incremental Vision-Language Object Detection (IVLOD) from existing methods. IVLOD is distinct in that it combines the incremental learning of new tasks with the preservation of zero-shot generalization capabilities. Unlike traditional incremental object detection methods such as CL-DETR and iDETR, which focus on learning new tasks incrementally without maintaining zero-shot abilities, IVLOD ensures that models can continue to generalize to unseen categories even after adaptation. Furthermore, while Grounding DINO utilizes natural language prompts for object detection, it does not address incremental task adaptation. In IVLOD, language is used as prompts for the objects to be detected, the same as how it is utilized in Grounding DINO. To enhance clarity, we will include a dedicated subsection comparing IVLOD with these existing methods both verbally (with Descriptive Subsection) and visually (with enhanced Figure 1), thereby providing a comprehensive understanding of its unique contributions.

2. The main contribution is the different learning rates

Response: We acknowledge the importance of clearly presenting the novel aspects of our work and distinguishing it from existing methods. The key innovation in our approach lies in the use of Zero-interference Loss (ZiL) to constrain the Reparameterizable Dual Branch (RDB), which effectively balances incremental learning performance and zero-shot performance of the VLODM. While methods like LoRA and Adapter introduce additional parameters for efficient fine-tuning, they do not employ a mechanism like ZiL. Our approach uniquely combines RDB with ZiL, addressing the specific challenges of incremental vision-language object detection. We will expand our literature review to include a comprehensive survey of adaptation-based methods for incremental learning, highlighting how our approach builds upon and differentiates itself from these methods.

3. Experimental section has flaws

Response:

1. The selection of the 13 datasets from the ODinW benchmark was based on the same subset used in the GLIP evaluation, ensuring consistency with prior work. We acknowledge that the full ODinW benchmark comprises 35 datasets. Due to computational resource constraints, we limited our evaluation to 13 datasets. However, we plan to extend our evaluation to include all 35 datasets in future work to provide a more comprehensive assessment.

2. CL-DETR and OW-DETR are not specifically designed for lower shot scenarios and do not perform well under low shot settings. These methods are more suited for settings where there is a larger amount of training data available. Comparing them with lower shots evaluation would not provide meaningful insights due to their design and intended use cases. We will provide the results of AT under full evaluation. The results are as follows:

3. Grounding DINO requires all the prompts to be combined into a single sentence, with periods separating each prompt. For instance, if the prompts are "dog," "cat," and "car," the combined prompt fed into the model would be "dog. cat. car."

4. When adapting to new tasks, the zero-shot performance of the original Vision-Language Object Detection Model (VLODM) typically degrades. This is a common issue in incremental learning scenarios and is the primary motivation for proposing the Incremental Vision-Language Object Detection (IVLOD) task. While our method shows a 1.31% decrease in zero-shot COCO performance compared to the Vanilla model, this reduction is significantly less than that observed with other methods. Our approach effectively minimizes the decline in zero-shot capability, demonstrating its efficacy.

5. Thank you for highlighting the importance of hyper-parameters in our method. We acknowledge that the presence of multiple hyper-parameters is a limitation of our approach. However, this does not detract from the main contributions of our paper.

6. We need to incorporate a norm into the final loss function to ensure effective training and optimization. Without a norm, it would be difficult to train and optimize the model properly.

4. Other minors

Response: Thank you for pointing out these minor issues. We will make the necessary revisions to address them.

Rebuttal for Weaknesses

1. Differences in Baseline Pre-Training

Response: Thank you for this important observation. We want to clarify that all compared methods, including TFA and iDETR, were indeed reimplemented using the pre-trained GroundingDINO model. This approach ensures a fair "apples-to-apples" comparison, as all methods were evaluated on the same scale of pre-training data. We will revise the manuscript to more clearly convey this setup and ensure there is no confusion regarding the comparability of the methods.

2. Demonstrate ZiRa with Other Architectures

Response: We appreciate the suggestion to evaluate ZiRa with additional architectures. To address this, we have demonstrated ZiRa with OV-DINO [1]. All the method are implemented with the same OV-DINO (swin-T) pre-trained on O365, GoldG, and Cap1M datasets. The results are as follows:

Table: Performance of Various Methods based on OV-DINO

This evaluation demonstrates that ZiRa performs effectively across different DETR-based architectures, validating its versatility. By incorporating ZiRa with OV-DINO, we show that our approach is not limited to GroundingDINO and can be broadly applied to other models.

[1] OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion. Hao Wang, Pengzhen Ren, Zequn Jie, Xiao Dong, Chengjian Feng, Yinlong Qian, Lin Ma, Dongmei Jiang, Yaowei Wang, Xiangyuan Lan, Xiaodan Liang. arXiv, 2024.

Rebuttal for Questions

3. Impact of Pre-Training Data on IVLOD Performance

Response:

The specific pre-training data used can significantly impact the effectiveness of incremental learning in Vision-Language Models (VLMs). Models pre-trained on diverse and extensive datasets tend to have better generalization capabilities, which can facilitate more effective incremental learning. Conversely, models pre-trained on limited or less diverse datasets might struggle with generalization, leading to more pronounced performance degradation when learning new tasks incrementally. We will include an analysis in the revised manuscript discussing the potential impacts of different pre-training datasets on the effectiveness of incremental learning, reinforcing the importance of diverse and comprehensive pre-training data for robust IVLOD performance.

4. Adapting to a Large Number of Tasks

Response: Thank you for this insightful question. Upon examining the evolution curves in Figure 5 more closely, we observe that in the initial stages, ZiL plays a relatively minor role in curbing the growth of the RDB's output norm. However, as the process of accumulating new knowledge progresses, the influence of ZiL becomes more significant, leading to stronger constraint effects. This increase in ZiL's impact leads to a dynamic balance: the interference caused by integrating new knowledge is counteracted by the interference reduction achieved by ZiL. As a result, while the L1 norm of the RDB's output does increase with the number of tasks, it does not rise unboundedly but rather stabilizes after learning a certain number of tasks. Therefore, our proposed approach does manage to scale to a large number of tasks to a certain extent, achieving a balance between the accumulation of new knowledge and the mitigation of interference.

5. Adapting to More Than One Class at a Time

Response: In our experimental setup, each "task" corresponds to learning an entire dataset, each of which contains more than one class. Thus, our proposed approach inherently handles the learning of multiple classes simultaneously. Our results demonstrate that the proposed method can effectively manage the incremental learning of multiple classes within each dataset, preserving zero-shot generalization capabilities while adapting to new tasks. We will clarify this aspect in the revised manuscript to ensure that it is clear that our approach supports learning multiple classes at a time.

6. Runtime and Memory Usage

Response:

Thank you for highlighting this crucial aspect. We agree that quantifying the marginal increase in runtime and memory usage when adapting to new tasks would provide valuable insights for future benchmarks. We will conduct additional experiments to measure and report the runtime and memory usage associated with our approach. These results will be included in the revised manuscript, providing clear benchmarks that can be used for comparison in future work.

7. Writing Quality

Response: Thank you for your feedback regarding the writing quality. We apologize for any confusion caused by grammatical errors and unclear phrases. We will thoroughly review and polish the manuscript to improve clarity and readability. Additionally, all dataset names (not class names) (e.g., Ae, Aq, Co) in Tables 1 and 2 are fully provided in the Subsection " Datasets" of Section "Experiments Setup".