C$^2$Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning

Thank you for your constructive feedback and recognition. Below are our responses, which we hope effectively address your concerns.

W1: Improvements in class coherence

We appreciate your insightful suggestion. As visualizations are not supported during the rebuttal phase, we instead provide quantitative evidence to demonstrate how our method enhances inter-prompt class-wise relevance and intra-class distribution alignment.

(1) Mean Inter-Prompt Class-wise Relevance Score (mPR).

The mPR measures the overall class-wise relevance among prompts during fusion. It is defined as: $mPR=1/N_p\sum_{i=1}^{N_p}(\sum_{j=1}^C\sum_{k=1}^{N_p}s_{i,j}W_{i,k}s_{k,j})$ where $N_p$ and $C$ are the prompt number and class number in a task, respectively. $s_{i,j}$ is the proportion of class $j$'s matched instance for prompt $i$. $W_{i,k}$ denotes the weight of prompt $k$ when generating the fusion prompt corresponding to prompt $i$.

The mPR comparison of our method and the state-of-the-art Powder is shown as follows:

Our method achieves a 0.036 absolute gain in mPR, corresponding to a 29.0% relative improvement over Powder. This result verifies that our Class-aware Prompt Aggregation design effectively filters and fuses class-relevant prompts, leading to better semantic alignment. As a result, our method achieves superior new knowledge acquisition and anti-forgetting capacity, as shown in Figure 1(a) of the main paper.

(2) Mean Intra-Class Distribution Distance (mCD).

mCD assesses the distributional alignment between local and global class representations. It is calculated as:

$mCD=\frac{1}{KC}\sum_{k=1}^{K}\sum_{j=1}^CKL(d_j, d_{k,j})$ where $K$ is the client number, $d_i$ is the ground truth distribution of class $j$, which is a Gaussian distribution obtained by calculating the mean and variance of the instance features of class $j$. $d_{k,j}$ is the local distribution of class $j$ on client $k$. $d_{k,j}$ is also a Gaussian distribution. For existing methods, $d_{k,j}$ is obtained by calculating the mean and variance of class $j$ using data of client $k$. In our method, $d_{k,j}$ is a fused distribution of the class-wise local data and transferred data.

The quantitative comparison of our method and Powder under the mCD metric is reported as follows:

Our method reduces mCD by 0.69 compared to Powder, which is a 9% improvement, demonstrating that the proposed Local Class Distribution Compensation module effectively narrows the gap between local and global distributions. This promotes inter-client knowledge consistency and enhances the model's overall knowledge accumulation, as shown in Figure 1(b) of our main paper.

W2: Ablation on the KD loss

(1) The KD loss is an inherent component of the Powder baseline. Therefore, the performance improvement reported in our experiments is solely attributed to our proposed LCDC and CPA modules. Specifically, as illustrated in Figure 4 of our main paper, LCDC and CPA individually contribute 1.88% and 1.33% improvements, respectively. When combined, a total gain of 2.51% is observed.

(2) To further validate the effectiveness of our proposed modules, we conduct experiments by removing the KD loss used in Powder. The results are presented below:

Even without KD, our method consistently outperforms the Powder baseline, achieving 1.59% on Avg and 0.61% on AIA. These results further confirm the robustness of our design, independent of KD.

(3) In fact, KD is a widely adopted fundamental technique in federated continual learning methods, such as Podnet [a], CFeD [b], GLFC [c], FedSpace [d], and Powder [e]. It is typically employed to align the aggregated global model with local client models, especially under non-IID task distributions. Without KD, local models tend to overfit to new data and forget previously learned global knowledge. Therefore, the use of KD is orthogonal and complementary to our contributions.

[a] Podnet: Pooled Outputs Distillation for Small-Tasks Incremental Learning. ECCV 2020

[b] Continual Federated Learning based on Knowledge Distillation. IJCAI 2022

[c] Federated Class-Incremental Learning. CVPR 2022

[d] Asynchronous Federated Continual Learning. CVPR 2023

[e] Federated Continual Learning via Prompt-based Dual Knowledge Transfer. ICML 2024

W3: The design choice of local class distribution compensation prompt

(1) We retrieve the local class distribution compensation prompt from the pool using the label $y$ to avoid inter-class interference during prompt training.

(i) Specifically, local distributions for each class typically exhibit independent and random shifts from the global distribution. As a result, there are no unified semantic patterns across classes for guiding distribution-shift modeling. Using a shared prompt pool for all classes, followed by weighted fusion, introduces optimization conflicts across classes and leads to sub-optimal learning.

(ii) In contrast, our method explicitly utilizes the class label $y$ to segregate prompts for different classes, enabling focused and optimal learning of class-specific prompts.

(2) We further compare our method with a weighted-sum prompt strategy. The results are as follows:

Our approach surpasses the weighted-sum baseline by 0.90% on Avg and 0.21% on AIA, confirming the effectiveness of our class-specific compensation prompt design in capturing localized distribution shifts in federated scenarios.

W4: Typo

Thank you for identifying this issue. We have corrected "the ocal class distribution" to "the local class distribution" in the revised version.

We sincerely appreciate your thoughtful and constructive feedback, which has significantly contributed to strengthening our work. We hope our response sufficiently addresses the concerns raised, and we would be grateful if the reviewer considers increasing the rating. Please feel free to raise any additional questions.

We sincerely appreciate the reviewer's constructive feedback and recognition. Please find our point-by-point responses below.

W1: More experimental comparison on FCIL benchmarks

Thank you for the valuable suggestion. We have conducted additional experiments on CIFAR-100 and ImageNet-A under the Federated Continual Incremental Learning (FCIL) setting [a], using the Dirichlet distribution with parameter $\beta$ to control data heterogeneity. The results are presented below:

Our method outperforms the state-of-the-art FCIL method LoRM by 2.98%/5.91%/1.22% at $\beta=0.5 / 0.1 / 0.05$, respectively. Compared to the prompt-based FCL baseline Powder, we achieve 2.47%/2.34%/0.49% improvements under the same settings.

Similarly, our method surpasses LoRM by 3.25%/3.42%/4.10%, and Powder by 1.22%/2.65%/1.59% at $\beta=0.5 / 0.1 / 0.05$, respectively.

In summary, these results validate the robustness and adaptability of our method under varying levels of data heterogeneity. In particular, the growing performance gains under smaller $\beta$ values can be attributed to our inter-client, intra-class distribution compensation mechanism, which enhances the model's ability to acquire and aggregate discriminative knowledge while mitigating inter-client conflicts.

[a] Closed-form merging of parameter-efficient modules for federated continual learning. ICLR 2025

W2: Definition of FAA

Thanks for pointing this out.

(1) Final Average Accuracy (FAA) is a standard metric used in FCIL to measure knowledge retention and accumulation. Let $a^t$ denote the test accuracy on the $t$-th task after the final incremental step. FAA is defined as: $\mathrm{FAA}=\frac{1}{T}\sum_{t=1}^Ta^t$ where $T$ is the total number of tasks. A higher FAA indicates better overall performance across all tasks and stronger continual learning ability.

(2) We have incorporated the definition of FAA into our appendix to improve the clarity and comprehensiveness of our paper.

We hope our responses adequately address your concerns. Please feel free to reach out with any further questions or suggestions.

Thank you for your valuable feedback and insightful comments. We address your concerns as follows:

W1: More datasets for comparison

(1) In the main paper, we report experimental results on DomainNet and ImageNet-R in Table 1, following the settings in Powder [a].

(2) In Table 4 of our appendix, we evaluate our method on the CIFAR-100 dataset. Our approach outperforms the state-of-the-art method Powder by 1.54% on the Avg metric and 0.86% on the AIA metric.

(3) We further conduct experiments on the ImageNet-A dataset. The results are summarized below:

Our method surpasses Powder by 1.99% on Avg and 2.20% on AIA, further confirming its robustness on challenging real-world benchmarks.

(4) These consistent improvements across multiple benchmarks verify the adaptability and generalizability of our approach. The performance gain primarily stems from our two core designs:

(i) Local Class Distribution Compensation (LCDC) actively models the global distribution of each class, thereby improving local discriminative knowledge acquisition and enhancing inter-client knowledge compatibility. This results in a more robust and informative prompt aggregation process.

(ii) Class-aware Prompt Aggregation (CPA) selectively filters and preserves class-specific prompts, effectively mitigating inter-class knowledge interference during aggregation. This design effectively consolidates the discriminative capacity of the learned features across clients.

Together, these modules enable effective distribution modeling at both global and local levels, enabling our model to boost performance on diverse benchmarks.

[a] Federated Continual Learning via Prompt-based Dual Knowledge Transfer. ICML 2024

W2: Efficiency and communication costs

We report communication and parameter overhead in Table 2 of the main paper. The results show our method maintains strong efficiency:

(1) Communication Overhead:

Our method requires 496.01MB for communication. Compared with prior prompt-based methods (FED-L2P, FED-DUAL, Fed-CODAP, Fed-CPrompt), this is 125.77MB to 319.62MB lower, yielding a 25.4% to 64.4% reduction. Compared to Powder, our method incurs only 2.93MB (0.6%) additional overhead, mainly due to the exchange of lightweight class distribution information. Since the number of classes is small, this additional cost is negligible.

(2) Training Overhead:

Our method introduces 2.82MB of learnable parameters. Compared with other baselines, this is 1.14MB to 8.61MB fewer, leading to 40.4% to 305.3% better training efficiency. Compared with Powder, our method adds only 0.18MB (6.8%), resulting from the lightweight distribution prompts introduced by the LCDC module.

(3) Inference Overhead:

At inference time, our method requires only 2.64MB of additional parameters, which is the same as Powder and 50% to 333.0% more efficient than other prompt learning baselines. This is because only discriminative prompts are retained during inference, incurring no additional cost compared to Powder.

W3: Experiments with severe data heterogeneity

(1) In Table 3 of our appendix, we report experimental results on the ImageNet-R dataset under varying levels of data heterogeneity using the Dirichlet distribution parameter $\beta$ ranging from 0.5 to 0.05. Notably, $\beta=0.05$ is typically regarded as a setting with severe data heterogeneity [b]. Under this condition, our method achieves a 2.29% performance gain over the state-of-the-art LoRM [b], and a substantial 6.48% improvement compared to the prompt-based baseline Powder.

[b] Closed-form merging of parameter-efficient modules for federated continual learning. ICLR 2025

(2) We further evaluate our method on the CIFAR-100 and ImageNet-A datasets under various levels of data heterogeneity. The results are summarized below:

The results demonstrate that our method surpasses the state-of-the-art FCIL method LoRM, achieving improvements of 2.98%/5.91%/1.22% at $\beta = 0.5/0.1/0.05$, respectively. Furthermore, compared to the state-of-the-art prompt-based FCL method Powder, our approach achieves 2.47%/2.34%/0.49% improvements at $\beta = 0.5/0.1/0.05$, respectively.

On ImageNet-A, our method surpasses LoRM by 3.25%/3.42%/4.10% and Powder by 1.22%/2.65%/1.59% at $\beta = 0.5/0.1/0.05$, respectively.

(3) These results clearly demonstrate that our method maintains robust performance across varying degrees of data heterogeneity, particularly under severe heterogeneity conditions (i.e., small $\beta$ values). The advantages over the Powder baseline in lower $\beta$ regimes can be attributed to our inter-client, intra-class distribution compensation mechanism, which effectively aligns client-specific distributions, enhances the model's knowledge acquisition capability, and reduces inter-client knowledge conflicts.

We sincerely appreciate the reviewer's thoughtful and constructive feedback, which has significantly contributed to strengthening our work. We hope our response sufficiently addresses the concerns raised, and we would be grateful if the reviewer considers increasing the rating. Please feel free to raise any additional questions.

Thanks for the valuable feedback. We hope our responses effectively address your concerns.

W1: Layout correction

Thank you for the suggestion. We have revised the manuscript layout to ensure that all figures appear in the correct order and align properly with the corresponding textual references.

W2: Clerical errors

Thank you for pointing this out. The references to Fig. 2(a) and Fig. 2(c) in Lines 305–306 should have been Fig. 2(b) and Fig. 2(d), respectively. We have corrected this typo in the revised version.

W3-1: Larger or smaller model architectures

(1) Larger Architectures:

We have adopted ViT-L as the backbone and compared our method with the state-of-the-art Powder. The results are as follows:

Our method achieves 1.80%/0.61% improvements on DomainNet and 0.42%/0.01% improvement on ImageNet-R under the Avg/AIA metrics compared to Powder.

The more significant improvement on DomainNet can be attributed to its larger domain gap from the pretraining dataset (ImageNet-21K). In this case, our class-aware client knowledge interaction mechanism improves knowledge acquisition by enhancing new domain distribution adaptation.

In contrast, the relatively modest gain on ImageNet-R is due to the strong alignment between ImageNet-R and the pretraining dataset. ViT-L can already capture a large portion of relevant knowledge during pretraining, leaving limited room for post-training improvement. Nevertheless, the consistent performance gains on both benchmarks verify the effectiveness of our method when using large-scale architectures.

(2) Smaller Architectures:

We have also evaluated our method using ViT-S as the backbone. The results are shown below:

Our method outperforms Powder by 1.28/1.67 on DomainNet and 1.51/1.22 on ImageNet-R under Avg/AIA metrics, demonstrating its adaptability to small-scale architectures.

(3) Summary:

These results confirm that our approach is robust across different model scales and consistently outperforms the state-of-the-art Powder. This is mainly because our class-aware client knowledge interaction mechanism leverages distributional information, which is largely independent of the backbone scale.

W3-2: Out-of-distribution pre-training datasets

(1) We would like to clarify that all reported benchmarks, DomainNet, ImageNet-R, and CIFAR-100, are out-of-distribution datasets relative to the pre-training dataset ImageNet-21K [a]. This is because part (ImageNet-R) or all (DomainNet, CIFAR-100) of their images do not appear in ImageNet-21K, and they typically exhibit significant domain shifts from the pre-training distribution.

[a] What Variables Affect Out-of-Distribution Generalization in Pretrained Models? NeurIPS 2024

(2) To further address the reviewer's concern, we additionally evaluate our method in another out-of-distribution setting by adopting CLIP as the backbone and conducting post-training on unseen benchmarks [b].

[b] SCAP: Transductive Test-Time Adaptation via Supportive Clique-based Attribute Prompting. CVPR 2025

The results are presented below:

Our method achieves 1.89%/1.28% Avg/AIA improvements on DomainNet and 1.04%/0.48% on ImageNet-R, respectively. These results demonstrate that our method is not dependent on the pre-training distribution and generalizes well under out-of-distribution scenarios.

This robustness is primarily attributed to the fact that many high-level visual semantics (such as shape, structure, and layout) are transferable across domains. Our Local Distribution Compensation and Class-Weighted Prompt Aggregation modules are capable of ** exploiting such domain-invariant cues**. By modeling the global distribution and adjusting local distributions on clients, they enable effective adaptation from pre-training models to unseen target domains in the federated continual learning scenarios.

W4: Equation 13

Thank you for pointing this out. The notation $\mathcal{L}_{ce}$ in Equation 13 is indeed a typo. It should be corrected to $\hat{y}$, which denotes the predicted class probabilities generated by the model during inference. We have carefully corrected this issue in the revised version.

W5: Cross-dataset sampling

(1) We have conducted additional experiments by alternately sampling task classes from ImageNet-R and CIFAR-100, as suggested. The results are summarized below:

Our method achieves 93.18% and 93.31% on the Avg and AIA metrics, respectively, outperforming the state-of-the-art Powder by 2.38% and 0.63%. These results demonstrate that our approach generalizes effectively under the more challenging cross-dataset federated incremental learning scenario. This robustness partially stems from the prompt learning scheme, which helps disentangle knowledge across datasets and mitigates mutual interference. In addition, the performance gain over Powder is primarily attributed to our Local Class Distribution Compensation and Class-Weighted Prompt Aggregation designs, which enhance the model's ability to learn and accumulate knowledge from clients at each stage.

Q1: Privacy leakage

The uploaded local class-aware feature distributions do not cause privacy leakage

(1) The uploaded distributions only contain the mean and variance of each local class. No instance-level information or raw data is transmitted to the server, ensuring that no individual-specific information is exposed.

(2) These uploaded statistics are equivalent to distributional prototypes, a widely adopted strategy in privacy-preserving learning. Similar approaches have been employed in existing privacy-friendly methods such as [c,d].

[c] Fecam: Exploiting the heterogeneity of class distributions in nonexemplar continual learning. NeurIPS 2024

[d] Distribution-aware knowledge prototyping for non-exemplar lifelong person re-identification. CVPR 2024

Q2: Indiscriminative attention

(1) The presence of indiscriminative attention patterns similar to those in Powder is expected, as Powder serves as our baseline. Our method inherits the local discriminative prompt from Powder, which may include suboptimal attention behaviors. However, our approach mitigates the impact of indiscriminative knowledge by enhancing the acquisition and accumulation of discriminative information as discussed below.

(2) We further observe that the locations of indiscriminative attention tend to be shared across different images, e.g., the two hotspots above the first, second, and sixth images. This phenomenon stems from dataset bias, where certain non-informative regions frequently appear in the same spatial positions across images, thus leading to spuriously high attention scores.

(3) Our method significantly reduces such indiscriminative attention through the following designs:

(i) The Local Class Distribution Compensation module actively models the global distribution of each class. This enhances both intra-client discriminative knowledge learning and inter-client knowledge compatibility, thereby facilitating more discriminative knowledge aggregation.

(ii) The Class-aware Prompt Aggregation mechanism selectively retains class-specific prompts while avoiding inter-class knowledge conflicts during aggregation. This strengthens discriminative feature extraction and suppresses noisy or irrelevant information.

In summary, our proposed method primarily overcomes the influence of indiscriminative attention inherited from the baseline by significantly improving the model's capacity to acquire and accumulate discriminative knowledge.

We sincerely appreciate your thoughtful feedback and hope the above clarifications are helpful. We welcome further discussion and are grateful for the opportunity to improve our work through your insights. Thank you again for your time and consideration.