Hybrid Re-matching for Continual Learning with Parameter-Efficient Tuning

Thank you for your insightful comments and questions.

Q1: Clarity

A1: For PET-based CL methods, the model first predicts task identity for each sample $x$. Then, the corresponding PET parameters p_\{\hat{t}_{f}} are retrieved from the parameter pool $P$, and are used for subsequent class prediction. When the predicted task identity equals the true task identity, we refer as matching. When they differ, it is referred to as mismatching. In our method, we correct the wrong task identity, which we term re-matching. We will add the this clarity in the revision.

---

Q2&Q3: Quality and Originality

A2&A3: Our HRM-PET is simple, yet effective and stable. The main contributions are notable:

Most importantly, we explore the core and long-existing problem in PET-based CL, the potential inaccuracy in the task-parameter matching process during the testing phase.

We propose a hybrid re-matching framework to improve the accuracy of task identity prediction. CTIRD alleviates the reliance on matching, facilitating shared knowledge without compromising task-specific knowledge.

Moreover, empirical results serve as evidence for the rationality and effectiveness of our method.

---

Q4: Explanation of Fig. 5

A4: Fig. 5 compares the attention regions of the model with (Ours) and without CTIRD (W/o). The first column displays attention maps generated using the correct task identity, while columns 2–10 correspond to attention maps produced using incorrect task identities. Two main observations can be drawn:

1. For the first column, with CTIRD, the model still focuses on the key regions, indicating that CTIRD does not compromise the plasticity.

2. For columns 2–10, even when incorrect task parameters are used, the model with CTIRD still attends to the critical features, e.g., the lizard’s head and limbs, the bird’s beak. Hence. CTIRD enhances knowledge sharing and reduces the model’s reliance on task matching, demonstrating the superiority of CTIRD.

---

Q5: Explanation of Fig. 4

A5: Fig. 4 illustrates the ratio of incorrectly and correctly matched samples that satisfy $\tau$, relative to the total number of actual mismatched and matched samples, respectively, across each task in the incremental sequence. Specifically, the blue curve shows that over 70% of all mismatched samples are detected by $\tau$, demonstrating that CRM can identify most mismatched samples during the incremental process. Additionally, the green curve indicates that 10–30% of correctly matched samples are misidentified as mismatches, highlighting the necessity of using Eq. 5 to determine whether task identity replacement is required.

---

Q6: Instance relationship distillation in Fig. 2

A6: The module of instance relationship distillation is in the section with training at the bottom of Fig. 2. The loss of instance relationship distillation is between the old feature space and the new feature space. We will add the module name in the revision to improve clarity.

---

Q7: Theory of re-matching improving the experiment performance for continual learning

A7: The accuracy of task identity prediction has been shown to be critical for improving CL performance, as demonstrated in HiDe-Prompt [77]. Assuming that in the class incremental learning (CIL) scenario, a total of $T$ tasks are defined as $D = \{D_1, D_2, \ldots, D_T\}$. In $D_t$ of task $t$, $\mathcal{X}\_{t}$ and $\mathcal{Y}\_t$ are the domain and label of task $t$. Let $\mathcal{X}\_t=\bigcup\_{j}\mathcal{X}_{t,j}$ and $\mathcal{Y}\_t=\{\mathcal{Y}\_{t,j}\}$, where $j \in \{1,...,|\mathcal{Y}\_t|\}$ indicates the $j$-th class in task $t$. Given a pre-trained model $f\_\theta$, CIL aims to learn $P(\boldsymbol{x} \in \mathcal{X}\_{i,j}|D,\theta)$ for the sample $\boldsymbol{x} \in \bigcup\_{k=1}^{t}\mathcal{X}\_{k}$, where $j \in \{1,...,|\mathcal{Y}\_t|\}$ indicates the $j$-th class in task $t$. Based on the theorem of Bayes, the goal can be decomposed as:

$P(\boldsymbol{x} \in \mathcal{X}\_{i,j}|D,\theta) = P(\boldsymbol{x} \in \mathcal{X}\_{i,j}|\boldsymbol{x} \in \mathcal{X}\_{i},D,\theta) P(\boldsymbol{x} \in \mathcal{X}\_{i}|D,\theta)$

where $P(\boldsymbol{x} \in \mathcal{X}\_{i}|D,\theta)$ is the task identity inference probability, $P(\boldsymbol{x} \in \mathcal{X}\_{i,j}|\boldsymbol{x} \in \mathcal{X}\_{i},\mathcal{D},\theta)$ is the within-task prediction probability. Hence, when the performance of task identity inference is improved, the performance of CIL will be enhanced. For more proof details, we recommend referring to HiDe-Prompt.

---

Q8: More reasonable framework

A8: We sincerely thank for the insightful suggestion. The idea of framing re-matching within a probabilistic framework is indeed inspiring. Assume the input is $\boldsymbol{x}$. The posterior probability that it belongs to class $y = c$ is $P(y=c | \boldsymbol{x}, D, \theta)$. We can further decompose it into task identity inference and within-task prediction:

$\begin{aligned} P(y = (i,j) \mid \boldsymbol{x}, D, \theta) = P(y = (i,j) \mid \boldsymbol{x}, \text{task}=i, D, \theta) \cdot P(\text{task}=i \mid \boldsymbol{x}, D, \theta) \end{aligned}$

where $y = (i,j)$ is the $j$-th class in task $t$. After training with PET, we obtain a parameter pool $\boldsymbol{\mathcal{P}} = \{p_1, p_2, \dots, p_T\}$, where $p_i$ denotes the learned parameters specific to task $i$. Rematching involves inference with multiple $p$. To aggregate these predictions in a principled manner, we draw upon Bayesian Model Averaging (BMA), which supports combining models under uncertainty about model selection. Applying BMA, the final posterior is expressed as:

$P(y=(i,j) | \boldsymbol{x}, D, \theta, \boldsymbol{\mathcal{P}}) = \sum_{i=1}^T P(y=j | \boldsymbol{x}, \text{task}=i, \theta, p_i) \cdot P(\text{task}=i | \boldsymbol{x}, D, \theta)$

However, not all task parameters can be weighted equally. We hope that the correct task identity has a higher weight. Therefore, we calculate the weights $\phi_t$ by the conditions in our strategy, such as the confidence of CRM:

$P(y=(i,j) | \boldsymbol{x}, D, \theta, \boldsymbol{\mathcal{P}}) = \sum_{i=1}^T P(y=j | \boldsymbol{x}, \text{task}=i, \theta, p_i) \cdot P(\text{task}=i | \boldsymbol{x}, D, \theta) \cdot \phi_t$

In our paper, $\phi_i$ acts as a binary gate (i.e., $\phi_i \in \{0,1\}$). Our rematching is uniformly implemented in the above equation. We will include this insight in the revisions.

---

Thank you for your insightful comments and questions.

Q1: Explanations for subscript notations and distribution

A1: Clarification of subscript: $\hat{d}(x)$ and $\tilde{d}(x)$ are all class-level distribution. $\mathcal{T}$ is used to convert class in $\hat{d}(x)$ into task identity in equations 5 and 7.

Clarification of distribution: $j$ in equation 3, $c_j$ in equations 5 and 7 all represent the index of the category. We will unify the expression in revision.

---

Q2: A clear statement specifying the parameter source for this prediction distribution

A2: In fact, we utilize $\hat{t}\_{s}$ in DRM to obtain $\tilde{d}(x)$ in equation 3. DRM has corrected part of the task identities, using $\hat{t}_{s}$ can reduce subsequent repeated calculations.

---

Q3: A comprehensive analysis of training time

A3: During training, for each new task $t$, we compute the feature embedding $e_k(x)$ for each sample under task $t$ using the top-$K$ ($K=5$) previous task-specific parameters. Specifically, we first utilize the pre-trained task classifier $g_{\omega}$ to obtain the top-$K$ old task identities with the highest confidence scores for all samples. The time cost of performing a forward through the classifier is negligible.

Subsequently, before training, each sample undergoes $K$ inferences to obtain $K$ features for knowledge distillation. Compared to training, the inference-only forward introduces minimal additional time overhead.

Experimentally, we measured the training time of baseline and HRM-PET on ImageNet-R on an RTX 3090 with a batch size of 64. As shown in the table, our method incurs only an additional 4.4% (i.e., 0.30 hours) of time overhead, which is acceptable. We will add the discussion in the revision.

---

Q4: The independent contributions and necessity of each re-matching module in the ablation study.

A4: As shown in the table, we conduct more detailed ablations on ImageNet-R. We have the following observations:

1. Compared with the baseline, each re-matching module brings improvements, which demonstrates independence.

2. Combining CRM and DRM achieves better performance compared with either CRM or DRM, which shows the necessity of each re-matching module.

---

Q5: The rationale for performing DRM prior to CRM

A5: As shown in the table, we conduct ablations on the relationship between CRM and DRM. We make the following validation:

1. Combining CRM and DRM achieves better performance than using either DRM or CRM alone, demonstrating their complementarity. In Figure 1 b, there is a partial intersection between correct and incorrect matching in the confidence-based distribution. DRM compensates for the limitations on these samples, thereby combining CRM and DRM further improves the matching.

2. When CRM and DRM are used together, the individual performance gains relative to the baseline are partially offset.
   This suggests a degree of overlap in the sets of samples selected for re-matching by the two methods. To minimize redundant computation, we apply CRM after DRM in our pipeline.

3. The model's performance is not sensitive to the execution order of CRM and DRM, further probing their complementarity and necessity.

---

Q6: The technical and typographical errors

A6: Thanks for the helpful comment. Line 193 states: For fairness, both our method and HiDe-Prompt use LoRA as the PET, and therefore HiDe-Prompt is referred to as HiDe-LoRA in the table. We promise to correct grammar, formulas, unclear expressions, and other technical and typographical errors in the revision.

---

Thank you for your insightful and timely feedback.

Q1: Difference from MOS and the mechanism for maintaining valid prediction distributions for past tasks

A1:

Difference from MOS: Although MOS [1] also addresses the problem of matching inaccuracy through Self-Refined Retrieval Mechanism, their strategies and assumptions are fundamentally different from ours: MOS assumes that the sample is mismatched when the task identity $\hat{t}\_{s}$ of the final predicted class is not same as the task identity $\hat{t}\_{f}$ corresponding to the selected PTE parameters. Specifically, MOS utilizes different task identities for inference until $\hat{t}\_{s} = \hat{t}\_{f}$. However, not all samples satisfy this assumption. In the incremental process, each task-specific PET's parameter and the final class classifier are trained jointly. When an input is inferred by a PET parameter with an incorrect task ID, the final classifier is likely to activate classes belonging to that same (erroneous) task. Therefore, although $\hat{t}\_{f}$ and $\hat{t}\_{s}$ are the same, they may both have wrong task identities. For example, in all mismatched samples of ImageNet-R, 55.71% of them are $\hat{t}\_{f} \neq \hat{t}\_{s}$. In contrast, our HRM-PET adopts a confidence-based rematching, under a different assumption for detecting and correcting mismatching based on prediction confidence rather than task identity consistency.

The mechanism for maintaining valid prediction distributions for past tasks: We follow the advanced PET-based method HiDe-Prompt to maintain valid prediction distributions for past tasks by modeling the categorical features as a Gaussian distribution and storing them. We will add these details and cite [1] and [2] as related works in the revision.

---

Q2:The motivation behind the CRM.

A2: Thank you for your helpful comments. We clarify the motivation of CRM as follows:

1. CRM is divided into two stages, i.e., detecting mismatched samples and finding the correct task identity for the mismatched samples. During the detection phase, the sample with $E(\tilde{d}(x))$ meeting the threshold is determined as mismatching. This process only depends on whether the task identity used in inference is correct, regardless of the high or low of $E(\hat{d}(x))$.

2. During the finding the correct task identity, we obtain the top-N candidate task identities from $\hat{d}(x)$ based on the predicted probability, so $E(\hat{d}(x))$ potentially affects the recall of the correct task identity. On the one hand, we can increase N to increase the candidate range. On the other hand, in Equation 5, we compare the distribution confidence after inference with all candidate task identities to avoid selecting the wrong candidate identity.

---

Q3:Discussion of scalability

A3: To discuss the resource consumption on longer tasks, we conduct extra experiments on 20 tasks on ImageNet-R as shown in the table below. Since $N$ in CRM is set to 2, the inference cost of our method only depends on the number of mismatched samples, e.g., the number of samples that meet the confidence threshold. When the number of tasks increases to 20, the matching error rate is higher, which inevitably leads to an increase in inference time. However, we improve the utilization of inference resources. Our method achieves a higher gain in accuracy per millisecond of increased inference time. The inference time per image only increases by 0.57 ms, but the accuracy has increased by 3.1%. Certainly, further reducing absolute inference cost remains a promising direction.

---

Thank you for your insightful comments and questions.

Q1: Clarification of inference time

A1: The minimal additional inference time overhead is mainly attributed to the following three factors:

1. Not all samples require additional inference in the re-matching. For DRM, we perform extra inference only on samples where $\hat{t}\_{f}$ is not equal to $\hat{t}_{s}$. For CRM, re-matching is conducted on samples whose confidence scores satisfy $\tau$. Hence, the average inference time per image increases marginally. In the table below, we provide the Ratio of samples undergoing re-matching under each strategy for clearer interpretation. | Method | $A_{N}\uparrow$ | Time$\downarrow$ | Ratio | |------------------|-----------------|------------------|-------| | Baseline | 71.60 | 2.81 | - | | Baseline+DRM | 72.60 | 2.91 | 12.2 | | Baseline+CRM+DRM | 73.86 | 3.24 | 30.3 |
2. In CRM, $N$ is set to 2, only one additional forward pass through the ViT is required.
3. In the baseline, each sample requires two forward passes through the ViT: one for task identity prediction and another for class prediction. Re-matching process involves only a single forward pass through the ViT, so the inference time does not double that of the baseline.

---

Q2: Explanation of Eq. (5)

A2: Eq.(5) describes how to find a more appropriate task identity when the sample $x$ is mismatched. Specifically, we first obtain the top-$N$ classes with the highest probabilities from the initial matching prediction distribution $\hat{d}(x)$:

\text{\Gamma} = \underset{\{c\_j\}\_{j=1}^N}{\text{argmax}}\ \hat{d}\_{c_j}(x)

Where \text{\Gamma} is the set of the top-$N$ classes. Candidate task identities are obtained by converting all classes to task identity in \text{\Gamma} with $\mathcal{T}$. Then, we compare the confidence scores $E$ of the final prediction distributions generated by inference with the parameters $p$ corresponding to all task identities. The task identity with the highest confidence is selected as the corrected task identity $\hat{t}_{s}$:

\hat{t}\_{s} = \underset{i \in \text{\Gamma}}{\mathrm{arg\,max}}\ E(g(h(x;p\_{\mathcal{T}(i)},\theta\_{ptm});\theta\_{g}))

The final class prediction ${\hat{y}}\_{CRM}$ is derived from the prediction distribution corresponding to $\hat{t}\_{s}$:

${\hat{y}}\_{CRM} = \underset{i}{\text{argmax}} \, g(h(x;p\_{\hat{t}\_{s}},\theta\_{ptm});\theta\_{g})$

---

Q3: Missing training details

A3: In fact, for each new task $t$, we compute the feature embedding $e_k(x)$ for each sample under task $t$ using the top-$K$ ($K=5$) previous task-specific parameters, rather than all old parameters. Specifically, we first utilize the pre-trained task classifier $g_{\omega}$ to obtain the top-$K$ old task identities with the highest confidence scores for all samples. The time cost of performing a forward through the classifier is negligible. Subsequently, before training, each sample undergoes $K$ inferences to obtain $K$ features for knowledge distillation. Compared to training, the inference-only forward introduces minimal additional time overhead.

Experimentally, we measure the training time of baseline and HRM-PET on ImageNet-R on an RTX 3090 with a batch size of 64. As shown in the table, our method incurs only an additional 4.4% (i.e., 0.30 hours) of time overhead, which is acceptable. We will add the discussion in the revision.

---

Q4: The expressions of some detailed steps are confusing

A4: Thanks for the helpful suggestion. We promise to fix the confusing expression in the revision.

---

Thank you for your insightful comments and questions.

Q1: A systematic study on the effect of the confidence measure

A1: To explore the effect of the confidence measure, we conduct ablation experiments on different post-hoc confidence measures in the table.

1. Overall, our method is not sensitive to different confidence calculation methods.

2. Generalized entropy (GEN) performs best across most settings. This is attributed to its design for semantic shift scenarios, i.e., detecting inputs with semantic categories that are absent from the training set [48]. In CL, the PET parameters for task $t$ are only trained on the semantic categories of task $t$. When predicted task identity $\hat{t}$ does not match the true task $t$ for sample $x$ from task $t$, the semantic class of $x$ is absent from the training set in task $\hat{t}$. Therefore, GEN is more advantageous in detecting mismatched samples.

3. On CIFAR-100, performance is near the upper bound with few mismatched samples. MaxLogit with a high threshold effectively filters mismatches, so we adopt it for better performance.

---

Q2: Increasing inference time

A2: Although the inference time is relatively increased, we achieve state-of-the-art performance across multiple datasets and pretraining settings. We acknowledge its practical significance and investigate the accuracy-efficiency trade-off in Section 4 of the supplementary material. In the future, further improving efficiency remains a promising direction.

---

Q3&Q7: Selection of hyperparameters

A3&A7: Our method is robust to most hyperparameters, and their selection is generally straightforward:

1. $\gamma$ and $M$ are from generalized entropy (GEN). According to GEN paper, we set $\gamma$ in (0, 1). $M$ represents the top-$M$ highest probability in the distribution. Given that the minimum number of classes encountered during the incremental process is 20 (in the second task) for all datasets, we set it to 20.
2. Given the inference cost, $N$ is set to 2 for all datasets. As shown in Section 4 of the supplementary material, $N$ governs the trade-off between accuracy and efficiency. $N$ can be adjusted based on available deployment resources.
3. $K$ and $\lambda$ in CTIRD are 5 and 0.2 for all datasets. We use common settings for $K$, i.e., top-5. In Eq. 8, $\mathcal{L}_{\mathrm{CTIRD}}$ is the sum of $K$ KL divergence losses in knowledge distillation. Drawing from prior work such as IRD [7], where the coefficient for a single distillation step is typically set to 1, we scale the overall loss by $1/K$, resulting in $\lambda = 1/K = 0.2$. The experiment in Fig. 6 verifies the effectiveness of our choice.
4. The $\tau$ affects the accuracy of mismatch detection. We search for the optimal value on the validation set. Overall, our method is robust to $\tau$. As shown in Fig. 6 (c), treating all samples as mismatches with $\tau=10^3$ still yields an improvement of 0.77 compared to selecting none ($\tau=-10^3$).
5. For the baseline methods, we employ the same selection strategy with identical hyperparameters to ensure a fair comparison.

---

Q4: The standard deviation in the ablation studies

A4: As shown in the three tables below, we supplement the standard deviations of three random seeds for some experiments in Tab. 3, 4, and Fig. 6 (a). The conclusions drawn remain consistent with those in the initial draft. We will include all standard deviations in the revision.

Tab. 3

Tab. 4

Fig. 6 (a)

---

Q5: The hyperparameters of generalized entropy

A5: Considering advanced performance and advantages in detecting semantic drift, we choose GEN as the main confidence function. According to GEN paper, we set $\gamma$ in (0, 1). $M$ represents the top-M highest probability in the distribution. Given that the minimum number of classes encountered during the incremental process is 20 (in the second task), we set it to 20. GEN’s hyperparameters influence the range and distribution of confidence scores, which in turn affect the optimal threshold. By fixing $\gamma$ and $M$, we ensure consistency in the confidence scale, enabling a more stable threshold.

---

Q6: Results of different PETs

A6: As shown in the table, using prompt as PET, we show the $A_N$ on two different pre-trained weights on the ImageNet-R dataset compared to the state-of-the-art prompt-based method. Our method HRM-PET still achieves state-of-the-art performance.

---

We sincerely appreciate your insightful comments and will add the following discussion to the revision.

Q1:​​Blurry Class-Incremental Learning​​

A1: Our algorithm can be applied to a blurry class-incremental learning scenario with some minor modifications:

1. In the blurry class-incremental learning scenario, some classes are associated with multiple task identities. However, in the initial task identity prediction of the baseline we follow, the model first predicts the class and then maps the class to a task identity with $\mathcal{T}$, which conflicts with blurry class-incremental learning setting. To address this issue, we revise the initial task identity prediction by adopting the key-value mechanism from DualPrompt. Specifically, during training, a learnable key is maintained for each task, and the input image features are used as a query to retrieve the corresponding task identity through key-value matching.

2. Based on key-value mechanism, we make the following modifications to our algorithm: For DRM, we use the final predicted features as query to retrieve $\hat{t}\_{s}$, and then determine whether $\hat{t}\_{s}$ is equal to $\hat{t}\_{f}$. For CRM and CTIRD, key-value similarity is utilized to obtain the top-N or top-K task identities in Equation 5 and Equation 7.

As shown in the table, we present a $A_N$ comparison of our method with the adapted version of HiDe-LoRA under blurry class-incremental learning scenario. We adopt the dataset split from [1] and iBOT-21K pretraining setting on ImageNet-R. * indicates that the key-value mechanism has been incorporated. As can be observed, our method also achieves the best performance.

[1] Moon, Jun-Yeong, et al. "Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning." Proceedings of the IEEE/CVF international conference on computer vision. 2023.

---

Q2:​​Label Noise Sensitivity

A2: As shown in the table below, we conduct experiments on Imagenet-R with iBOT-21K by injecting noise at different ratios. The label noise assigns {0%, 10%, 20%} samples of the dataset to other labels by a uniform probability. Our method still achieves the best performance under different noise ratios. This benefits from our approach of comparing confidence scores in CRM to determine whether detected mismatched samples require task identity replacement, which alleviates the interference of noise on mismatching detection with threshold filtering. Moreover, Instance relation distillation in CTIRD is more robust to noise than logits or feature distillation, effectively learning valuable knowledge from previous tasks.

---

Q3: Mention of PET-based CL works that do not maintain task-specific parameters

A3: Thank you for your helpful suggestion. We will add the necessary details in the revision.

---