DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection

We sincerely thank Reviewer Xs7M for their constructive feedback and positive assessment of our work. We particularly appreciate their acknowledgment of our straightforward and effective warm-up strategy, the thoroughness of our experimental section, the realism of our novel ODinW-O benchmark, and the value of our provided codebase—an acknowledgment we deeply value. We will now address each of their concerns in detail and incorporate all resulting adjustments into the final manuscript.

Rearrangement of the method section

I think the method section should be better arranged. 4.1 focuses on two-stage training [...]

We thank the reviewer for this constructive feedback. Accordingly, we will adopt the recommended rearrangement of Sections 4.1 and 4.2 and adjust the positioning of Figure 3 in the final version of the paper.

On the memory cost

I think training a LoRA matrix for each class is not a significant contribution and is still far from efficient and scalable.

We respectfully clarify that the use of per-class LoRA modules is not presented as our primary contribution, but rather as a necessary architectural choice to enable our main research goals: investigating the fine-grained compositional properties of fine-tuning modules—a direction previously unexplored in the context of open-vocabulary object detection. These properties include not only module addition, which is instrumental for incremental learning, but also module subtraction, enabling targeted unlearning. The warm-up phase, which the reviewer kindly acknowledged, is crucial to this design. It serves not only to extract general knowledge but also to initialize the class-specific modules in a shared representation space. Indeed, as suggested by prior work [1, 2], a common starting point for adaptation greatly enhances model compositionality, which is a central theme of our investigation.

To address the valid concerns about efficiency and scalability, we refer the reviewer to our response to Reviewer voji (section "On the memory cost"), where we introduce a more parameter-efficient variant. This variant trains semantic-specific modules rather than class-specific ones, a strategy that significantly reduces memory overhead by grouping related classes together. Furthermore, we invite the reviewer to proceed to the following section, where we have devised additional experiments and considerations to specifically address this point.

Following weakness 3, what is DitHub's memory usage compared to baselines?

Prompted by the reviewer's question, we conducted additional experiments to analyze the trade-off between performance and memory usage in our class-specific modules. Specifically, we evaluated how the average AP on ODinW-13 varies with the rank $r$ of our matrices. The results are presented in the table below.

As shown, performance degrades gracefully even as the rank substantially decreases. Notably, a rank of $r=2$ achieves a result less than 2 AP points below our best-performing model ($r=16$), while representing an 8-fold reduction in per-class parameters and memory usage. We also test the extreme case where $r=1$, which collapses the adaptation matrix into a vector, and we find the performance remains remarkably strong. We attribute this robustness to the granularity of our framework; since each module only needs to encode the knowledge of a single class, it can do so effectively with fewer parameters compared to approaches that require information aggregation from multiple classes into a single module.

Below, we provide the direct memory usage comparison (in MB) against baselines, as requested by the reviewer.

The results show that our method with $r=2$ employs approximately the same memory as ZiRA while performing significantly better (+2.28 AP). Furthermore, our $r=1$ variant delivers surprisingly competitive results, scoring less than 1 AP point below ZiRa while using only half the memory.

Additional datasets

The method is only tested on one benchmark ODinW-13. [...]

Our focus on ODinW-13 was initially motivated by its suitability for testing incremental learning across domain gaps and because it is the primary benchmark used by ZiRa, the method that introduced our experimental setting.

However, we agree with the reviewer's point and have now performed additional experiments on two diverse datasets. We evaluated our method on the large-vocabulary LVIS benchmark, as well as on AODRaw, a recent and challenging benchmark encompassing different meteorological and environmental conditions. We kindly refer the reviewer to our detailed discussion and results in the response to Reviewer voji (section Additional datasets).

Other questions

If $B$ is not shared (class-specific), what is the difference between $A$ and $B$?

Indeed, as the reviewer suggested, when matrix $B$ is not shared, it is also class-specific. In this configuration, we do not enforce any distinct roles upon $A$ and $B$. Their purpose becomes a collective one: to jointly learn a low-rank matrix ($\Delta W=BA$) that optimally adapts the model's behavior for that class. The empirical results for this configuration are reported in Appendix A.2.

Can this method apply to other open-vocabulary tasks, like classification?

Yes, we believe the core principles of our method are indeed applicable to other open-vocabulary tasks, such as classification. For instance, one could maintain a library of modules to incrementally adapt the vision or text encoder of a model like CLIP [3] for specialized zero-shot classification tasks (e.g., adapting it to a fine-grained domain like "species of birds" or "models of cars").

However, we chose to focus on Object Detection as we believe it is a more challenging task and, therefore, better suited to studying the compositional properties of specialized modules (e.g., composing modules to detect multiple, distinct classes in a single image).

[1] Mirzadeh, Seyed Iman, et al. "Linear Mode Connectivity in Multitask and Continual Learning." ICLR 2021.

[2] Qin, Yujia, et al. "Exploring Mode Connectivity for Pre-trained Language Models." Conference on Empirical Methods in Natural Language Processing 2022.

[3] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." ICML 2021.

We sincerely thank Reviewer i82i for their constructive feedback and for acknowledging the strengths of our work. We are particularly grateful for the reviewer's recognition that our methodology successfully addresses the critical need for selective, composable, and revisable adaptation—our primary objective—and for their acknowledgment of our model's memory efficiency over naive approaches.

We will now address each of the reviewer's concerns in detail, and all corresponding revisions will be integrated into the final manuscript.

Additional baselines on ODinW-O

In Table 2, comparison on ODinW-O is limited to ZiRa and G-Dino only. [...]

As requested, we have expanded our comparison on the ODinW-O benchmark to include the results for iDETR and AT. For completeness and ease of comparison, the updated table also retains G-Dino and ZiRa and includes our method, DitHub. This revised table will replace the original Table 2 in the final manuscript.

The expanded results in the table are consistent with our findings in Table 1. While iDETR outperforms ZiRa on the average metric (Avg), it demonstrates weaker performance in the zero-shot scenario (ZCOCO). Conversely, AT achieves a lower average score but is more effective at preserving zero-shot capabilities, yielding results comparable to ZiRa. Our method, DitHub, confirms its state-of-the-art performance by a consistent margin.

On the memory cost

[...] DitHub's memory profile could become a bottleneck [...]

We thank the reviewer for acknowledging both the existing discussion on potential memory bottlenecks within the original manuscript and our efforts to enhance memory efficiency over naive class-specificity.

For a more comprehensive analysis, including additional experimental studies on this topic, we kindly refer the reviewer to our detailed responses to Reviewer voji and Reviewer Xs7M ("On the memory cost" sections).

Analysis on divergent domains

[...] deeper analysis of potential information loss/conflict when merging across heavily divergent domains [...]

We agree that investigating the effects of merging modules from heavily divergent or semantically conflicting domains is a valuable research direction, which we also highlighted in the conclusions of our original manuscript. Prompted by the reviewer's feedback, we designed a new experiment to shed further light on this matter.

The experiment focuses on the class "Person" and compares two merging scenarios. In the first scenario ("Similar Domains"), we merge two class-specific modules ($A_{class}$) that were both trained on standard RGB image datasets (e.g., Pascal VOC, Pv, and Hardhat Workers, Hw). In the second scenario ("Divergent Domains"), we merge one module trained on a standard RGB dataset (Pv) with another trained on a heavily divergent dataset—the Thermal (Th) dataset. We then evaluate the zero-shot performance of the merged modules from both scenarios on the COCO validation set. The results are presented below.

We repeat the same experiment for the class "Car," using the Aerial (Ae) dataset as the divergent domain and the Vehicles (Ve) dataset as the similar one. The results are presented in the following table.

The results show that merging modules from divergent domains is, in general, more challenging than merging modules from similar ones. As the results for the "Person" class demonstrate, merging modules from similar domains yields a higher ZCOCO score (59.89) compared to merging from divergent domains (57.64). This modest drop suggests that a minor degree of negative interference may occur when merging modules from significantly different data distributions, as the reviewer hypothesized. For the "Car" class, the drop in performance is less pronounced, suggesting that the Aerial domain conflicts less with standard RGB images than the Thermal domain does.

We sincerely thank Reviewer voji for their valuable feedback, for finding our manuscript to be well-written, and for recognizing our method's state-of-the-art performance as a substantial contribution. We also appreciate their acknowledgment of our detailed and analytically reasonable ablation studies. We will now address the reviewer's concerns in detail and incorporate all resulting adjustments into the final manuscript.

On the memory cost

In some scenarios, maintaining a separate matrix for each category can be expensive. [...]

While the memory cost can be a concern in some scenarios, our work adopts the exact experimental setting from [1] to ensure fair comparison and build upon established benchmarks. This framework focuses on enhancing a large, pre-trained model with a smaller set of specialized or rare categories (e.g., "cottontail-rabbit" or "pothole" from the ODinW-13 benchmark) or handling significant domain shifts (e.g., RGB to thermal), rather than on adding thousands of new classes.

Within this established setting, our use of class-specific modules is a deliberate design choice, as it is fundamental to our core contributions: exploring the composability of learned concepts and fine-grained unlearning. An architecture without this modularity would undermine our ability to investigate these properties.

To ensure this modularity remains efficient, we implement each module with only a single low-rank projection matrix ($A$, with dimensions $r \times d$, where the rank $r \ll d$). To further address the reviewer's valid concern, we have conducted new experiments specifically optimizing the memory consumption of our approach. These are detailed below and in our answer to Reviewer Xs7M (in the section "On the memory cost").

[...] how can the $A$ matrices be efficiently managed, particularly in scenarios where classes exhibit clear hierarchical or semantic dependencies?

To address the reviewer's insightful question, we designed a new experiment to evaluate the management of $A$ matrices based on semantic hierarchies. Following the DitHub pipeline, instead of training a specific $A$ component for each class, we define three macro-categories: animals, vehicles, and objects. When a class from one of these groups is encountered during training, the corresponding group matrix (e.g., $A_{animals}$) is updated.

This methodology allows us to study the semantic relationships within our learned modules and to analyze how individual class concepts can coexist within a single, hierarchically superior representation. The class-to-group mappings are as follows:

The results, comparing the standard DitHub (per-class $A$) with the semantic aggregation approach, are reported below.

The results are highly encouraging. The experiment confirms that a single group matrix can learn a robust, shared representation for an entire semantic category. We found that groups with high semantic relation (e.g., animals) are more composable than more disparate ones (e.g., objects), which we attribute to the semantic similarity between their constituent classes (e.g., cat and dog vs. pistol and couch).

Crucially, this approach demonstrates a significant gain in efficiency. By reducing the number of trainable $A$ matrices from 24 to just 3, we observe only a minor impact on accuracy. This confirms that our modules can effectively leverage semantic dependencies, as the reviewer correctly hypothesized.

Additional metrics

In Fig 8, paper only reports the average performance across the tasks of ODinW-13. The ZCOCO metric is also essential [...]

We thank the reviewer for this suggestion. As requested, we now report the Zero-Shot COCO (ZCOCO) metric along with the Avg AP scores across tasks.

The results show a progressive decrease in the ZCOCO metric as the average AP across tasks rises, which is an expected trade-off as specialization degrades zero-shot capabilities.

Notably, the Base fine-tuning baseline scores higher on ZCOCO than the original Grounding DINO zero-shot model. We attribute this counter-intuitive result to catastrophic forgetting. Our full method, DitHub, is trained to be robust across the varied domains of ODinW. In contrast, the Base procedure completely forgets such broad knowledge and overfits to the new tasks. This process of forgetting diverse, out-of-distribution data results in a model that is more aligned with the data distribution of the COCO benchmark, leading to an artificially inflated ZCOCO score.

Backbone diversity

The paper evaluates the proposed method in only a single setting (Grounding DINO), without validation across different models or scales. [...]

To address the concern regarding model scales, we have conducted new experiments using the larger, publicly available pre-trained version of Grounding DINO with a Swin-B backbone. The table below presents these new results for our method, DitHub, and the primary competitor, ZiRa.

These results confirm that our method, DitHub, scales effectively and maintains its strong performance when integrated with a larger and more powerful backbone model.

Regarding model diversity, our decision to focus on Grounding DINO was twofold. Firstly, while other Detector with large-scale pretraining on both Detection and Grounding data, like GLIP [2] exist, Grounding DINO [3] is its state-of-the-art successor, and a full adaptation of our method to GLIP would be infeasible within the rebuttal period. Secondly, and crucially, Grounding DINO is the foundational model used in the work that established our experimental setting [1], making it the most appropriate choice for a direct and fair comparison. For these reasons, we believe our in-depth evaluation on this single, highly challenging model provides a strong and valid test of our method's contributions.

Additional datasets

[...] Additional experiments on other open-vocabulary or long-tailed benchmarks [...]

To address the reviewer's concern and better demonstrate the generalizability of our approach, we have conducted new experiments on two additional, diverse benchmarks. We excluded datasets like Object365, as it was part of the pre-training corpus for the Grounding DINO backbone and would not be a valid test set. Furthermore, our original benchmark choice was guided by our experimental setting, which specifically requires class re-occurrence across semantically distinct domains. This property is not typically found in standard long-tailed benchmarks, as they are generally not structured with overlapping classes across divergent domains.

First, we present results on the large-vocabulary dataset LVIS. To make this evaluation computationally feasible while retaining the complete set of 1203 categories, we adopt an established procedure from the literature [4] that uses a 10% subset of the annotations.

Additionally, we evaluated our method on AODRaw [5], a challenging benchmark recently introduced at CVPR 2025. This dataset is particularly well-suited to our setting as it features 9 distinct domains with significant class overlap, spanning varied meteorological conditions as well as indoor and outdoor scenes. The results are reported in the following table.

These additional results on both LVIS and the challenging AODRaw benchmark provide substantial new evidence for the robustness and generalizability of our proposed method beyond the primary ODinW-13 setting.

[1] Deng, Jieren, et al. "Zero-shot generalizable incremental learning for vision-language object detection." NeurIPS 2024.

[2] Li, Liunian Harold, et al. "Grounded language-image pre-training." CVPR 2022.

[3] Liu, Shilong, et al. "Grounding dino: Marrying dino with grounded pre-training for open-set object detection." ECCV 2024.

[4] Kamath, Aishwarya, et al. "Mdetr-modulated detection for end-to-end multi-modal understanding." ICCV 2021.

[5] Li, Zhong-Yu, et al. "Towards raw object detection in diverse conditions." CVPR 2025.