Componential Prompt-Knowledge Alignment for Domain Incremental Learning

Thank you for your constructive feedback and recognition. Below are our responses, which we hope effectively address your concerns.

Q1-1: Requirement of maintaining a prompt pool.

(1) Maintaining a prompt pool is not a specific requirement of KA-Prompt but is already present in the baseline method.

(2) A prompt pool is fundamental to prompt-based continual learning methods, enabling the retention of historical knowledge with negligible storage overhead. All compared state-of-the-art prompt-based approaches, including L2P, S-Prompts, DualPrompt, CODA-Prompt, CPrompt, and C-Prompt, incorporate a prompt pool.

Q1-2: Additional training overhead

(1) Maintaining a prompt pool does not introduce additional training overhead since stored prompt sets are frozen after being added and are not further optimized.

(2) In our KA-Prompt, frozen prompts from the pool are selected to guide cross-domain componential knowledge alignment in the Historical Prompt Online Aligning (HPOA) branch. This branch shares nearly the same computational cost as the New Prompt Training branch, introducing only an additional forward pass. However, the HPOA branch contributes a 2.28% improvement when applied to the baseline model on the ImageNet-R benchmark, as shown in Fig. 5 of our main paper.

Q2: Aligned prompt update

(1) The aligned prompts are updated during training to encode knowledge from new domains.

(2) The prompt parameters in the Reusable Prompt Memory remain frozen during the aligned prompt update, ensuring they are unaffected during training.

Q3: Purpose of Historical Prompt Online Aligning module

The Historical Prompt Online Aligning (HPOA) module aims to align the new prompts to parts of historical prompts in the prompt pool. Specifically, as new prompts are updated, HPOA dynamically matches them to the closest historical prompts, assigns fusion weights, and fuses them. The fused prompts are then fed into the ViT model, ensuring cross-domain componential alignment throughout training.

Q4: Sblation study

(1) By default, module ablation studies in the main paper are conducted on ImageNet-R. This has been explicitly stated in the revised version.

(2) Additional ablation studies on ImageNet-Mix and DomainNet are provided in Fig. G of https://anonymous.4open.science/r/ICML-31/FigG-Ablation.png. Specifically, the results show that:

(a) Our prompt initialization strategy (Reusable Knowledge Mining, $\boldsymbol{f}_R + \boldsymbol{f}_G$) significantly outperforms the existing methods (Wang et al., 2023a), due to the improved historical knowledge utilization capacity.

(b) Our online alignment design ($\boldsymbol{f}_A$) consistently improves model performance by strengthening knowledge alignment during training, thereby enhancing cross-domain fusion compatibility at test time.

(c) When all our modules are used together, the model performance is further improved since the historical knowledge utilization is improved during both training and testing.

Q5: Heat map visualization

Thanks for your suggestion! We have visualized the attention maps of prompt tokens at different stages, along with the fused tokens, in Fig. A of https://anonymous.4open.science/r/ICML-31/FigA-Heatmap.png. The results show that each prompt token (component) captures a semantic part of objects.

(1) Due to semantic misalignment among tokens, the fused prompt tokens in C-Prompt fail to precisely capture object-specific information, preventing the model from fully leveraging discriminative features.

(2) Our method improves component-wise alignment, enabling the fused prompt token to effectively capture object features. This results in a 4.73% Average accuracy improvement in Tab. 1 of our main paper.

We sincerely appreciate the reviewer’s constructive feedback and recognition. We hope the following responses effectively address your concerns.

W1: Performance analysis

(1) Both KA-Prompt and the C-Prompt baseline exhibit lower performance on the initial domain compared to other prompt-based methods due to the BEMA design proposed in C-Prompt. BEMA is a batch-wise antiforgetting strategy that prevents the forgetting of cross-domain shared parameters (e.g., classifier). Since BEMA constrains new knowledge learning, and the randomly initialized shared parameters lack semantic knowledge, it limits the initial performance of both KA-Prompt and C-Prompt, particularly on small-scale datasets like ImageNet-R.

(2) To further analyze this effect, we removed BEMA during the first domain learning, and the results are presented in Fig. E of https://anonymous.4open.science/r/ICML-31/FigE-ImageNet-R.png. The findings show that: (a) Without BEMA, our model achieves performance comparable to state-of-the-art methods on the first domain, confirming that BEMA is the primary factor contributing to the degraded initial performance. (b) After learning from 15 domains, our KA-Prompt (w/o initial BEMA) achieves 66.54±0.68, which is 0.03% higher than our results in the paper (w/ initial BEMA). This marginal performance improvement occurs because some discriminative knowledge is shared across domains. Thus, even if the initial domain is insufficiently trained, knowledge from later domains can still enhance its performance within our framework. (c) Overall, KA-Prompt achieves 4.08% and 4.11% improvements on ImageNet-R with and without initial BEMA, respectively, verifying its effectiveness in consolidating long-term knowledge. These improvements stem from our knowledge alignment design, which enhances cross-domain prompt compatibility, enabling more efficient utilization of learned knowledge and facilitating both positive forward and backward transfer.

W2: Methodological details

Your understanding is correct.

(1) At the beginning of Greedy Prompt Search, the Reusable Prompt Memory is empty, and the Memory Matrix $S^M\in \mathbb{R}^{0\times N_t}$ is an empty matrix. Then, the column–wise max process is conducted to obtain a $1\times N_t$ null matrix. Consequently, the Assignment Matrix $S^*$ become a ($[(t−1)×N_p]×N_t$) null matrix.

(2) We have incorporated these methodological details into our paper for improved clarity.

W3: Unify the component shapes

Thank you for this valuable suggestion. We have modified the shape of the ViT model in Fig. 1(b) to match Fig. 3, ensuring overall consistency. The revised version is illustrated in Fig. F in https://anonymous.4open.science/r/ICML-31/FigF-Modify.png.

Comments: Moving Figure 7 to main paper

Thanks for the valuable suggestions and appreciation of our experimental designs. We have moved Fig. 7 to our main paper. We have moved Fig. 7 to the main paper. Additionally, the following experimental settings and quantitative analyses are included:

To demonstrate that componential prompt-knowledge alignment mitigates knowledge conflicts, we conducted an ablation study by shuffling prompt components under different conditions. The experiments were performed on the final domain of the DomainNet benchmark, where all learned domain prompts were used for prompt matching. As shown in Fig. 7 (a), C-Prompt (Liu et al., 2024a) experiences performance improvement in Shuffle-B/C/D/E due to its randomly learned componential structure, which tends to be suboptimal in its original form. In contrast, KA-Prompt consistently degrades when prompt components are perturbed, indicating its intrinsic alignment structure. Despite the misalignment noise introduced by random shuffling, the worst performance of KA-Prompt (Fig. 7 (b) Shuffle-E) remains 6.4% higher than the best performance of C-Prompt (Fig. 7 (a) Shuffle-B). This demonstrates that our greedy prompt search algorithm effectively extracts generalizable knowledge across domains, significantly enhancing adaptation to new domains.

Thanks for the valuable feedback and comments. We hope the following responses address your concerns.

W1: Contributions on framework

The C-Prompt baseline corresponds to our New Prompt Training branch. We have made two key designs to form a brand-new DIL framework:

(1) Reusable Knowledge Mining (RKM) mechanism. Unlike existing methods that randomly initialize new prompts, RKM actively searches for old prompts containing shared knowledge between the new and all old domains, significantly improving new domain adaptation and cross-stage knowledge alignment.

(2) Historical Prompt Online Aligning (HPOA) branch. HPOA introduces an online search and re-weighting based prompt fusion strategy to mitigate cross-stage knowledge drift during training, effectively improving the utilization of multi-domain knowledge.

W2: Quantization of misalignment

To quantify misalignment, we have traversed the cross-stage prompt token orders to obtain the performance of the optimal alignment. As shown in Fig. 7, C-Prompt exhibits a 0.64% degradation compared to the optimal alignment, indicating suboptimal order learning. In contrast, our method achieves 0.57-0.79% higher performance than alternative orderings, verifying that it successfully attains optimal alignment.

W3, Q4: Complexity evaluation and theoretical analyses

The objective of RKM module at stage $t$ is formulated as follows: given $m$ training samples and $n=(t-1)×L_s$ old prompts, obtain $k=L_s$ old prompts that minimize $\sum_{i=1}^{m}\{\max_{1≤j≤n} s(x_i,p_j)\}$ where $s(x_i,p_j)$ denotes the similarity between a training sample $x_i$ and a prmpt $p_j$.

(1) The complexity of our greedy search algorithm is $mnk=O(nk)$, more efficient than exhaustive grid search, which requires $mC_n^k=O(n^k)$ operations.

(2) The optimization of RKM can be approximated as a k-medoids problem [1] by considering old prompts as special training samples (since m>>n, this approximation does not affect the theoretical conclusion). According to Theorem 4.4 of [1], in a one-way search setting, the error bound of our solution $E(θ^*)$ relative to the optimal solution $E(θ)$ satisfies: $E(θ^*)≤(1+\frac{2k}{n+m})E(θ)$. Since m+n>>k, our greedy search ensures globally optimal solutions under different conditions and leads to stable performance.

[1] PAMAE: Parallel k-Medoids clustering with high accuracy and efficiency. SIGKDD, 2017.

(3) We have conducted experiments on ImageNet-R to compare our method with existing approaches:

[a] initializes new prompts using those from the previous domain.

[b] selects prompts with the highest similarity scores in a single search step for initialization.

The results show that our greedy search-based reusable knowledge mining strategy (Greedy) consistently outperforms [a] and [b] with improvements of 1.81% and 1.04%, respectively.

Both [a] and [b] suffer from insufficient utilization of old knowledge due to limited knowledge relevance within adjacent domains and overlooking of knowledge in lower-similarity prompts, respectively.

In contrast, our approach evaluates the unique knowledge that has not been included in the selected prompts iteratively, effectively improving the utilization of old knowledge.

Q1: Definition of components and semantic consistency

(1) The components refer to the tokens of each prompt. As shown in Fig. 2, each prompt, e.g., $p_t^1$, contains 4 tokens (i.e., components), each encoding a distinct aspect of object knowledge.

(2) In DIL, the categories of different domains are identical, thus the object parts are semantically consistent across domains.

Q2, Q3: Do alignment alleviate conflicts

We have visualized the attention maps of prompt tokens at different stages, along with the fused tokens, in Fig. A of https://anonymous.4open.science/r/ICML-31/FigA-Heatmap.png. The results show that each prompt token (component) captures a semantic part of objects.

(1) Due to semantic misalignment among cross-domain prompt tokens, the fused prompt tokens in C-Prompt fail to precisely capture object-specific information.

(2) Our method improves component-wise alignment, enabling the fused prompt token to effectively encode discriminative features of objects. This results in a 4.73% increase in Average accuracy, as reported in Tab. 1.

Q5: Influence of prompt fusion

Fig. D in https://anonymous.4open.science/r/ICML-31/FigD-Pos-Transfer.png shows the testing performance trends for each domain. The results indicate that 2 out of 14 old domains exhibit performance reduction after continual training. This arises due to data imbalance, where some domains contain a few training samples, leading to biased knowledge during fusion. Nevertheless, our method achieves significant improvements on 12 out of 14 old domains, indicating that it effectively achieves positive knowledge transfer.

Thanks for the valuable feedback. We hope our responses address your concerns.

W1: Misalignment's occurring and definition

(1) Misalignment occurs during cross-stage prompt fusion in DIL. In Fig. 2, each prompt (e.g., $p_t^1$) consists of 4 tokens, each encoding distinct partial knowledge of objects. Misalignment refers to the disorder of partial knowledge within prompt tokens of different stages. When fusing prompts from multiple stages, this cross-stage token-level misalignment introduces semantic conflicts in the fused prompt. Then, the sub-optimal fused prompt is fed to the Attention layer, leading to degraded performance.

(2) Misalignment does not occur in VPT because it is designed for static training data, where all prompts are optimized jointly rather than incrementally.

(3) Our method explicitly enhances cross-stage prompt alignment, ensuring that the fused prompt retains a coherent representation of semantics. These high-quality fused prompts thereby lead to improved test performance.

Q1: Measuring alignment

(1) To measure alignment, we have traversed different token orders across stages to identify the configuration that yields the highest model performance, which we define as the optimal alignment, as shown in Fig. 7.

(2) The results show that the learned prompt token order in C-Prompt baseline exhibits a performance degradation of 0.64% compared to the optimal alignment, verifying the presence of misalignment.

(3) In contrast, the prompt token order learned by us consistently outperforms alternative orders by 0.57–0.79%, demonstrating that our approach effectively achieves optimal alignment.

W2: Validation of misalignment

We have visualized the attention maps of prompt tokens at different stages, along with the fused tokens, in Fig. A of https://anonymous.4open.science/r/ICML-31/FigA-Heatmap.png. The results show that each domain-specific prompt token learns a semantic part of objects. Compared to C-Prompt which introduces semantic misalignment in prompt tokens from distinct domains, the fused prompt tokens in our method effectively preserve part-level information, enabling the model to fully exploit discriminative features.

W3: Necessity of the Greedy Prompt Search (GPS)

(1) GPS is necessary because (a) the shared knowledge between new and old domains is distributed across various old prompts, and (b) the knowledge of some old prompts overlaps significantly. A naive selection based solely on high similarity scores in a single search would often lead to redundant prompt selection while overlooking relevant knowledge present in lower-similarity prompts. This results in insufficient utilization of old knowledge. Please refer to Reviewer 8nTg-W3, Q4 for more experimental analyses.

(2) The computational cost of GPS is significantly less than performing a single forward pass on the training set since it primarily involves simple matrix addition and subtraction.

W4,Q3,Q4: Catastrophic forgetting

Our method inherits the anti-forgetting capacity of prompt learning and does not significantly risk forgetting:

(1) In our Historical Prompt Online Aligning (HPOA) module, old prompts are frozen, ensuring that knowledge from previous tasks is not overwritten or forgotten.

(2) The prompt selection for old tasks is minimally influenced by HPOA. This is because prompt selection relies on prompt keys, and new prompt keys are only trained by minimizing their distance to new data features.

(3) In Tab. 2, domains are trained sequentially from left to right. Our KA-Prompt performs 1.04% inferiorly to the C-Prompt baseline, while we outperform the C-Prompt from the second domain and obtain 4.25% Average accuracy improvement across all domains. These results show that our approach achieves a better balance between acquisition and forgetting.

W5: Metrics setup

$a_{T,i}$ is evaluated on the test data of the i-th domain. Eq. 12 measures the final model’s performance across all previous domains by computing the average performance over them. We have carefully clarified these details in the revised version.

Q2: Classifier setup

The final classifier is shared across all domains, consistent with most prompt-based methods. We have clarified this in the revised version.

Q5: Visualization of S-Prompt

(1) To maintain consistency in the color-method pairing relations across Fig. 4 (a)(b), Fig. 8, and Fig. 9, only top-8 methods ranked by Average accuracy are chosen for performance visualization, where S-Prompt is not included.

(2) In Fig. B-1, and Fig. B-2 of https://anonymous.4open.science/r/ICML-31/FigB-S-Prompts.png, we have added the performance curves of S-Prompt. The results show that our KA-Prompt effectively outperforms existing methods during long-term learning.

Comments: Simplify Fig. 2

To highlight the misalignment phenomenon, a simplified illustration of Fig. 2 is shown in Fig. C of https://anonymous.4open.science/r/ICML-31/FigC-Simplify.png.