Question 1: Conceptual Gap Between Medical and Natural Domains

We appreciate the reviewers' feedback and would like to clarify the differences between the medical and natural domains, as well as the limitations of existing methods in medical object detection. Previous studies, such as those by Qin et al. (2022), Ma et al. (2024), and Zhu et al. (2024), highlight key challenges in the medical domain, including data scarcity, modal diversity (e.g., CT, MRI, X-rays), and annotation complexity, as medical annotations require expert input. Existing methods face issues like foreground-background coupling, where background areas may confuse classifiers, and prompt-label attention coupling, which can dilute prompt information and reduce sensitivity to subtle features, such as small lesions. The iDPA framework overcomes these challenges through instance-level prompt generation and decoupling attention, significantly improving medical detection. While designed for medical applications, our method shows strong generalization capabilities, suggesting it can also perform effectively in natural domains due to the greater complexity of the medical domain. We hope this clarifies the necessity and relevance of our method in the medical field, and we appreciate the reviewer’s suggestions, which have helped refine the paper.

Question 2: Value of Incremental Medical Object Detection

Incremental medical object detection is crucial for addressing real-world challenges in clinical deployment, offering significant advantages over zero-shot models like Medical SAM. It enables adaptation to new tasks without full retraining, thus overcoming regulatory constraints (e.g., HIPAA, GDPR), and supports the dynamic evolution of medical knowledge, allowing models to adjust to new diseases and imaging technologies without losing prior learning. Incremental learning effectively handles the diversity of medical image modalities (CT, MRI) and annotation inconsistencies, ensuring generalization across different types while reducing the computational and storage demands of retraining large models. While models like Medical SAM excel in zero-shot scenarios, they struggle with rare, out-of-distribution concepts and multitask compatibility. Incremental learning methods like iDPA are better suited for these challenges, particularly in detection tasks. Our method complements models like MedSAM by generating bounding boxes as prompts for precise predictions and enabling continuous learning in out-of-distribution domains. In summary, incremental medical object detection addresses regulatory, data, modality, and learning challenges in dynamic medical settings.

• Qin, Ziyuan, et al. "Medical image understanding with pretrained vision language models: A comprehensive study." arXiv preprint arXiv:2209.15517 (2022).
• Ma, Jun, et al. "Segment anything in medical images." Nature Communications 15.1 (2024): 654.
• Zhu, Jiayuan, et al. "Medical sam 2: Segment medical images as video via segment anything model 2." arXiv preprint arXiv:2408.00874 (2024).

Question 3: Comparison with Dense Prediction Continual Learning Works

The paper focuses on incremental object detection (bounding box regression and classification) in medical images, with an emphasis on dense prediction approaches like DenseBox, which predict relative object positions. This differs from tasks like panoptic or open-vocabulary segmentation. iDPA contrasts with methods such as Eclipse (CVPR 2024) and PanopticCLIP (TPAMI 2024): Eclipse targets panoptic segmentation with dense pixel masks, while iDPA detects discrete objects through bounding boxes. Both methods use prompts, but iDPA’s instance-level prompt generation isolates fine-grained features, which could inspire dense prediction methods, though direct comparison is challenging due to task differences. PanopticCLIP focuses on zero-shot open-vocabulary segmentation, while iDPA handles incremental class learning. Technically, Eclipse uses spatial-semantic cues for segmentation, whereas iDPA decouples instance-level knowledge from background clutter. Architecturally, Eclipse modifies segmentation heads like Mask2Former, while iDPA extends VLOD models like GLIP. Future work includes using iDPA’s decoupled prompt attention to enhance dense prediction and combining it with panoptic prompts for joint detection-segmentation continual learning. We also plan to evaluate iDPA on dense prediction datasets (e.g., Retina, ISIC) to assess its generalizability.

Question 4: Others

1. We will add an efficiency comparison of the method in Table 1 in the revised manuscript, where the high efficiency of our method can already be observed in the current table.

2. During training, the ground truth (gt) boxes are used to generate instance features, and during testing, only the prompts saved in the prompt pool are used.

3. The sentence in Section 4.1 is in blue color to indicate emphasis.

We appreciate your detailed feedback and suggestions for improvement. We treasure the opportunity to address your concerns and improve our work.

Weakness 1: Domain Shift Robustness with Bounding Box-only Features

Thank you for this important observation. The paper focuses emphasizes instance-level knowledge decoupling to specifically focus on discriminative foreground regions (e.g., lesions or organs), thereby minimizing interference from irrelevant background information. The inherent cross-modal alignment capabilities of VLOD models such as GLIP also help in capturing contextual relationships, which further reduces dependency on background features. Empirical results in Table 1 demonstrate that our proposed iDPA consistently outperforms global prompt-based methods (e.g., L2P, DualPrompt) in full-data settings, indicating that localized feature extraction improves detection accuracy without compromising robustness to domain shifts. Moreover, Figure 5(a) empirically validates that instance-level knowledge surpasses image-level knowledge in our scenario. Nevertheless, we agree with the reviewer that selectively incorporating relevant background context could further enhance the model’s robustness. Thus, in future work, we plan to explore hybrid attention mechanisms that effectively integrate valuable background information during the feature-to-prompt knowledge transfer process.

Weakness 2: Lack of Ablation on Scaling Factor α in Eq. (6)

We would like to thank the reviewer for the suggestion. We have conducted new experiments with respect to $\gamma$, as shown in the table below:

Weakness 3: Ambiguity in W_k and W_v in Eq. (8)

W_k and W_v are linear projection layers mapping instance features (v_c) into query/key/value spaces for cross-attention. They enable adaptive feature alignment between instance representations and task-specific prompts. These will be clearly defined in the revised manuscript.

Weakness 4: Limited Initialization Scope in CCPKI

Incremental initialization (from task $i-1$) balances the stability-plasticity trade-off: recent tasks are given higher priority to mitigate forgetting, while older tasks are retained through the prompt pool.

Compared to using "all tasks before the $i$-th one," this approach reduces the number of additional model parameters. By using the $i-1$-th task for initialization, model training parameters remain consistent, requiring only $1 \times \theta^{CCPKI}_{i-1}$.

In contrast, using all tasks prior to the $i$-th task would require a total of $i \times \theta^{CCPKI}_{i-1}$ additional parameters.

Weakness 5: Unclear Contribution of DPA Components

To clarify whether DPA's gains stem from removing the $[\overline{p_t}]$ learning (Eq. 10) or introducing $\lambda$ (Eq. 13), we present this in Fig. 5(b). Compared to the naive method, removing $[\overline{p_t}]$ (i.e., when the scale type is 1.0) results in a +2.79% FAP, while our final approach, which introduces $\lambda$ (dim), leads to a +5.29% FAP.

Weakness 6: Unfair Comparison of Naïve Baseline in Table 3

We appreciate the reviewer’s concern. However, due to hyperparameter differences, the Naïve baseline in Table 3 operates on $\Phi_{f}$, which requires fewer parameters compared to $(\Phi_{v} + \Phi_{t})$, resulting in lower memory usage and reduced training time. In contrast, the methods in Table 1 operate on $(\Phi_{v} + \Phi_{t})$. To further clarify, we have added the Naïve method for $(\Phi_{v} + \Phi_{t})$, which achieves 41.72 FAP, 47.30 CAP, and 8.57 FFP. By introducing the iDPA method on top of the Naïve approach for $(\Phi_{v} + \Phi_{t})$, we significantly improve these metrics, with FAP increasing by 6.69, CAP by 6.26, and FFP decreasing by 4.19. These improvements, compared to the methods in Table 1, highlight that the gains are due to the effectiveness of our approach, rather than hyperparameter tuning discrepancies.

Thank you for the detailed and constructive feedback! We treasure the opportunity to address your concerns and improve our work.

1. Mathematical Justification for DPA superiority

We appreciate the reviewer’s feedback. DPA enhances prompt learning by separating the attention mechanism into distinct prompt-token interactions, minimizing interference from token embeddings. Its key advantage is the re-normalization of attention weights via the scaling factor $\lambda(f_t)$, as derived in Eq. (11-13). This balances the influence of prompts and pretrained tokens, boosting prompt effectiveness when token embeddings dominate due to length. We agree that adding formal theoretical analysis or empirical complexity comparisons would strengthen this and plan to include them in future revisions.

Overall Analysis

$$f_1 = \text{Concat}[ (1-\lambda(p_t)) \text{Attn}_{v \rightarrow t}(f_v, p_t) + \lambda(p_t) \text{Attn}_{v \rightarrow t}(p_v, p_t); (1-\lambda(f_t)) \text{Attn}_{v \rightarrow t}(f_v, f_t) + \lambda(f_t) \text{Attn}_{v \rightarrow t}(p_v, f_t)] = \text{Concat}[A, B]$$

where $p_{\{v, t\}} \in \mathbb{R}^{l \times d}$ represent the vision and text prompts, and $f_v, f_t$ are the visual and textual features before being fed into $\text{Attn}_{v \rightarrow t}$.

$$f_2 = (1- \lambda(f_t)) \text{Attn}_{v \rightarrow t}(f_v , f_t) + \lambda(f_t) \text{Attn}_{v \rightarrow t}( p_v, f_t) = B.$$

If $\text{Attn}_{v\to t}(\cdot,\cdot) \in \mathbb{R}^{L \times d}$, then $A(p_t)$ and $A(f_t)$ belong to $\mathbb{R}^{L \times d}$ and $f_1 \in \mathbb{R}^{(L+l) \times d}$, while $f_2 \in \mathbb{R}^{L \times d}$.

$$f_2 = \text{Attn}_{v\to t}(f_v,f_t) + \lambda(f_t) \Delta,$$

where $\Delta = \text{Attn}_{v\to t}(p_v,f_t) - \text{Attn}_{v\to t}(f_v,f_t)$.

$$\frac{\partial f_2}{\partial \theta} = \frac{\partial \text{Attn}_{v\to t}(f_v,f_t)}{\partial \theta} + \lambda(f_t) \frac{\partial \Delta}{\partial \theta}.$$

This indicates that $f_2$ has a lower-dimensional structure, residual components, and a more direct gradient flow.

Computation Cost

Lemma 1: Computation Cost

Lemma:
$f_2$ is computationally lighter than $f_1$.

Proof:
The overall $f_1$ is $f_1 = \text{Concat}[A;B]$. However, $f_2$ uses only the $B$ branch, $f_2 = B$. The computation cost of branch $A$ is roughly $O_A = O(f_v, p_t) + O(p_v, p_t) =O(A + B)$, while for $B$, $O_{f_2} =O(B)$. Thus, $O_{f_1} >O_{f_2}$.

Convergence Benefit Analysis

Lemma 2: Convergence Behavior

Lemma:
Let $f_1$ and $f_2$ be models that have converged to local minima, with an optimal representation $f^* = B$. If $f_1$'s output is locally linear around the optimum, then $f_2$ achieves the same performance as $f_1$ at convergence.

Proof:
Consider the loss function $\mathcal{L}(f_{\text{out}})$, where $f_1$ outputs $y_1 = h(\text{Concat}[A;B])$ and $f_2$ outputs $y_2 = h(B)$. At convergence, $\nabla \mathcal{L}(f_1) = \mathbf{0}$ and $\nabla \mathcal{L}(f_2) = \mathbf{0}$, with $f^* = B$ as the optimal representation.

Since $h$ is locally linear at the optima, there exists a matrix $M$ such that:

$$h(\text{Concat}[A;B]) = M \begin{pmatrix} A \\ B \end{pmatrix} = M_1A + M_2B.$$

At convergence, $M_1A = \mathbf{0}$, so:

$$h(\text{Concat}[A;B]) = M_2B = h(B).$$

Thus:

$$y_1 = h(\text{Concat}[A;B]) = h(B) = y_2.$$

Hence, $f_2$ performs as well as $f_1$ at the local minima.

2. Lacking evidence in generality beyond ODinM-13

We thank the reviewer for the suggestion. ODinM-13 already includes tasks across multiple modalities and organs, providing a degree of generalization. To further support generalizability, we conducted additional experiments on polyp datasets from different centers. Results are summarized below:

3. Failure modes and limitations

In our experiments, we observed that the IPG module improves learning in low-resource and hard cases. The DPA module further enhances the IPG's performance and helps reduce forgetting. For example, in the CPM-17 dataset (with only 30 training samples), the naive method achieves an FAP of 1.09. Adding only the IPG module improves performance to 15.64, adding only the DPA module achieves 1.80, and combining both modules increases FAP to 36.54.

Other notes:

We will address the missing definitions in Equation (8). Specifically, $W_k$ and $W_v$ represent the linear projection matrices for keys and values in the attention mechanism. These will be clearly defined in the revised manuscript.

Question 1: Significance of Incremental Learning in Full-Data Settings

Incremental learning methods like iDPA offer significant advantages in medical settings where models must be deployed incrementally due to regulatory, ethical, or resource constraints. Retraining large models on full datasets is often too costly, but iDPA only updates prompt vectors (1.4% of trainable parameters), which reduces training time. While Medical SAM performs well in zero-shot segmentation, it struggles with long-tail classes and domain shifts. On the other hand, iDPA enables incremental class learning and preserves prior knowledge, effectively handling challenges such as adding new classes over time. It also supports task-specific adaptation, such as joint detection and classification, which SAM is not designed for.

Question 2: Stability of Instance-Level Prompt Generation in Few-Shot Settings

The IPG module ensures stability in few-shot settings by aggregating features from all instances within a class using cross-attention, reducing reliance on noisy instances. It combines local features with global ones, preventing overfitting to sparse annotations. Empirical results show that iDPA outperforms other methods in 1-shot settings and provides a 3.66% FAP improvement compared to naive prompts. IPG also enhances generalization by using a scaling factor (γ) to expand regions of interest (RoIs), capturing contextual information beyond tight bounding boxes. By leveraging pretrained GLIP model knowledge, IPG prevents overfitting even with few samples. To further improve stability, semi-supervised initialization, hard negative mining, and hyperparameter tuning are suggested.

Question 3: Model Complexity and Writing Clarity

Thank you for the reviewer’s valuable feedback. We appreciate the concerns regarding model complexity and writing clarity. iDPA’s modular design was intentionally chosen for flexibility, with IPG handling instance-level knowledge extraction and DPA optimizing attention. To manage complexity, we have designed hyperparameters like K and λ to be learned adaptively and are exploring simplifications such as hyperparameter-free designs. We also recognize the importance of clear and consistent notation and will standardize symbols and provide clearer explanations for equations, ensuring the content remains both accessible and rigorous.

Question 4: Technical Details of DPA Module

1. Table 5 Interpretation:

   • The "X-Attn Number" represents the number of cross-attention layers in the fusion encoder where DPA is applied. For example, "6 (all)" indicates that DPA is enabled in all 6 layers.

2. Figure 2 Redesign:

   • It should be clarified that DPA is applied to all three encoders (visual, text, and fusion), with emphasis on its role in the fusion stage. The visual and text encoders are optional, as shown in the ablation experiments in Table 4. When all three are used together, continual learning performs best. However, incorporating DPA into the fusion encoder provides a good balance of performance, additional parameter load, memory usage, and training time. Therefore, in subsequent experiments, we default to using the fusion encoder.

3. Computational Efficiency:

   • Memory Access: DPA requires three parallel attention computations (vision→text, text→vision, and original PA). However, this is mitigated by:
     • Reduced Feature Length: The prompt length (l=10) is much shorter than image/text tokens (L=10,000+), which minimizes memory overhead.
     • Computation Merge: During testing, we merge the three parallel attention computations, reducing the overall computational cost compared to the original PA. This is further demonstrated in Reviewer rTU6 under the section "Mathematical Justification for DPA Superiority in Computation Cost."