Class-wise Balancing Data Replay for Federated Class-Incremental Learning

We sincerely appreciate the professional comments provided by Reviewer i4C7

W1: Privacy analysis

A1: For any feature vector $x \in \mathbb{R}^d$, a random orthogonal matrix $P \in \mathbb{R}^{d \times d}$ satisfies:

$$P^T P = P P^T = I.$$

Hence, the orthogonal transformation $x' = P x$ preserves the L2-norm of the vector:

$$\|x'\|_2 = \|P x\|_2 = \sqrt{x^{T} P^{T} P x} = \|x\|_2.$$

This implies that all points on the unit sphere are merely rotated by $P$, with their original coordinate meaning destroyed. $P$ is sampled uniformly from the Haar distribution on $O(d)$, then $y = P x$ is effectively the representation of $x$ under an unknown random rotation basis.
Without knowledge of $P$, an attacker observing $y$ cannot deduce the unique $x$, because:

$$\forall x' \text{ with } \|x'\|_2 = \|x\|_2, \ \exists P' \in O(d), \ y = P' x'.$$

In other words, infinitely many $x'$ on the sphere could lead to the same $y$ with an appropriate rotation $P'$.

W2: Experiments with GLFC and the simple replay baseline

A2: We conducted 5-task class-incremental experiments under different data heterogeneity settings and 5 clients. As shown in the Table, FedCBDR achieves the best performance across all cases.

W3: SVD/ISVD Transformation and Privacy

A3: We replaced the term “encryption” with “orthogonal obfuscation,” and clearly defined the transformation:

$$X_k^{(i)'} = P_k^{(i)} X_k^{(i)} Q^{(i)},$$

where $P_k^{(i)}$ and $Q^{(i)}$ are orthogonal matrices preserving norms but rotating the feature space. And $P_k^{(i)}$ is generated locally on each client (via QR decomposition of random Gaussian matrices) and never transmitted to the server. The global $Q^{(i)}$ is initialized by the server and publicly shared but does not leak sensitive information.

W4: Dimensions of $\Sigma^{(i)'}$

A4: In the original manuscript, we wrote the full SVD as:

$$X^{(i)'} = U^{(i)'} \Sigma^{(i)'} V^{(i)'T},$$

with $U^{(i)'} \in \mathbb{R}^{n \times n}$, $\Sigma^{(i)'} \in \mathbb{R}^{n \times d}$, and $V^{(i)'} \in \mathbb{R}^{d \times d}$. While this is correct for the full SVD, in practice we only need the reduced SVD form with

$$U^{(i)'} \in \mathbb{R}^{n \times r}, \ \Sigma^{(i)'} \in \mathbb{R}^{r \times r}, \ V^{(i)'} \in \mathbb{R}^{d \times r},$$

where $r = \min(n, d)$. We will revise Section 4.1 to clarify this notation and explicitly define $r$ as the effective rank used for pseudo-feature construction.

W5: Figure captions

A5: We have revised the caption of Figure 7 to: “Performance comparison of all methods across varying settings on CIFAR10 (3 tasks, $\beta=\{0.1,0.5\}$), CIFAR100 (5 tasks, $\beta=\{0.1,0.5,1.0\}$), and TinyImageNet (10 tasks, $\beta=\{0.1,0.5,1.0\}$).”

W6: Scalability, Complexity, and Communication Efficiency concerns

A6: We have conducted additional large-scale experiments by extending the number of clients to 50/100 while simulating asynchronous participation, and the client sampling rate is set to 0.2. The dataset is divided into five tasks, and in Re-Fed and FedCBDR, the number of replay samples is set to 150/300 for CIFAR-10 and 500/1000 for CIFAR-100.The following results can be summarized:

Moreover, we have further clarified the computational complexity of the SVD step, Client-side ISVD: $𝑂(𝑛_𝑘𝑑^2)$ per client. Server-side SVD: $𝑂(𝑁𝑑^2)$, where $𝑁=\sum_kn_k$ is the total number of pseudo-features.

And we provide a summary of the additional training cost on clients, communication overhead, and computation cost on the server incurred by our proposed method in comparison to FedAvg. Overall, the additional cost of performing SVD and computing leverage scores is relatively small.

The table shows that the communication cost scales linearly with the feature dimension $d$ and the number of clients $K$, meaning that higher feature dimensions or more clients lead to proportionally increased upload costs.

W7: TTS Hyperparameter Sensitivity

A7: (i) The temperature parameters in the TTS module are chosen based on empirical insights and grid search. A lower temperature for old classes ($\tau_{\mathrm{old}} < 1$) sharpens logits to reduce forgetting, while a higher temperature for new classes ($\tau_{\mathrm{new}} > 1$) smooths logits to avoid overfitting to new classes. (ii) Alternatively, the temperatures and weights ($\tau_{\mathrm{old}}, \tau_{\mathrm{new}}, w_{\mathrm{old}}, w_{\mathrm{new}}$) can be treated as learnable parameters, optimized during training, similar to recent approaches like FedETF [Li et al., ICCV 2023].

W8: Compare with GLFC

A8: Unlike GLFC, which relies on gradient compensation and prototype sharing for global coordination, FedCBDR uses SVD-based pseudo-feature construction and leverage score sampling to build a class-balanced replay buffer. Moreover, the TTS module dynamically assigns temperatures ($\tau_{\mathrm{old}} < 1, \tau_{\mathrm{new}} > 1$) to mitigate prediction bias. Experiments on CIFAR10 and CIFAR100 show that FedCBDR significantly outperforms GLFC, demonstrating its effectiveness.

W9: Temperature Scaling

A9: Compared to existing temperature scaling, our TTS module is specifically designed to address the imbalance between old and new classes in incremental learning. By applying different temperatures ($\tau_{\mathrm{old}} < 1, \tau_{\mathrm{new}} > 1$) and weights ($w_{\mathrm{old}}, w_{\mathrm{new}}$), TTS balances predictions and mitigates new-class overconfidence. Ablation studies confirm its crucial role, as removing TTS significantly degrades performance.

W10: Pseudo-Feature Construction

A10: For each client's local feature matrix $X_k \in \mathbb{R}^{n_k \times d}$, we perform SVD $X_k = U_k \Sigma_k V_k^{\top}$ and replace $U_k$ with a random orthogonal matrix $P_k$ to generate pseudo-features $\tilde{X}_k = P_k \Sigma_k V_k^{\top}$, which perturbs the local feature space while preserving spectral information. The server aggregates $\tilde{X}_k$ and applies SVD with leverage scores to select representative samples for constructing a class-balanced replay buffer. For privacy, $P_k$ remains local, and its randomness ensures that $X_k$ cannot be reconstructed from $\tilde{X}_k$.

W11: Generality of Datasets

A11: Our experiments are conducted on CIFAR10, CIFAR100, and TinyImageNet, which are widely used FCIL benchmarks and also adopted by methods such as GLFC, LANDER, and Re-Fed to ensure fair comparisons. The core modules of FedCBDR, GDR and TTS, are resolution-agnostic and can, in principle, be applied to larger datasets like ImageNet-1k, with feature dimensionality reduction helping to lower computation and communication costs. Due to time limitations, we plan to extend FedCBDR to such large-scale benchmarks in future work.

We sincerely appreciate the professional comments provided by Reviewer Csv6

W1: The meaning of n_k^i in lines 136 and 139 may need further clarification. Is it related to defined in line 138? Some descriptions may need correction. For instance, the term “Global-perspective Active Data Replay” in line 208 should be revised to “Global-perspective Data Replay”

A1: $n_k^i$ denotes the number of samples of client $k$ in the $i$-th task. And we have also corrected the full name of the GDR module to ``Global-perspective Data Replay.'

W2: It is recommended to unify the notation throughout the paper. For example, in Section 3, t is used to denote tasks, while in Section 4, i is used for the same purpose.

A2: Upon reviewing all symbol definitions and usages, we have standardized the task notation to $t$.

W3: The paper could better explain the motivation for using leverage scores and how it compares to random or uniform sampling strategies.

A3: Compared to random or uniform sampling, which ignores data geometry and may select redundant or less informative samples, leverage-score-based sampling prioritizes samples that span the critical directions of variation. This leads to a more balanced and informative replay buffer, particularly under class-imbalance scenarios. And we have compared our method with uniform sampling in the response to Reviewer i4C7's W2, where the results consistently demonstrate the superiority of our approach due to its ability to select more representative and informative samples.

W4: Will uploading features pose a risk of privacy leakage?

A4: Our method does not upload raw data but instead transmits pseudo-features $\tilde{X}_k = P_k \Sigma_k V_k^{\top}$, which are protected by random orthogonal transformations, where $P_k$ is generated locally on the client and kept secret. Given any $\tilde{X}_k$, there exist infinitely many pairs $(X_k', P_k')$ satisfying the same decomposition, making it impossible to uniquely reconstruct the original samples. This random rotation mechanism functions as a randomized privacy protection approach.

W5: How can the additional communication cost caused by uploading features be reduced?

A5: To reduce the communication overhead caused by feature uploading, we combine feature dimension reduction with sparsification and quantization. Specifically, PCA, random projection, or bottleneck layers can be used to map high-dimensional features (e.g., $d = 512$) to a lower-dimensional space (e.g., $d = 64$), reducing the transmitted data volume by more than 80% with minimal impact on model performance. In addition, applying sparsification to retain only the most informative elements of each feature vector, or using 8-bit/16-bit quantization instead of floating-point encoding, can further compress the communication cost.

W6: Is the performance of the proposed method sensitive to the order in which tasks are presented?

A6: The order of tasks is implicitly determined by the random seed, and changing the seed will alter both data splits and task order. To evaluate robustness, we report the mean and standard deviation across multiple random seeds in Tables 1, 2, 6, and 7. The results show that our method consistently achieves better performance in both mean accuracy and variance compared to baselines, indicating that FedCBDR is not sensitive to task order.

We sincerely appreciate the professional comments provided by Reviewer scc5

W1: The feature uploading process appears to introduce additional communication overhead.

A1: The additional communication overhead primarily comes from uploading the pseudo-features ($\tilde{X}_k \in \mathbb{R}^{n_k \times d}$) generated by each client during the GDR module. The exact size of this overhead, including per-client and total communication per round, is provided in the response to Reviewer i4C7's W6. To further reduce this cost, we can employ feature compression techniques (e.g., projecting features from 512 to 128 dimensions, reducing data by 75%) and client sampling strategies, where only a subset of clients upload pseudo-features in each round.

W2: The authors claim in line 199 that training a generator incurs higher costs than local training.

A2: For the LANDER method, each round of local training on a client requires 15.6657 seconds and 1582 MB of memory, whereas the sample generation process takes 753.9422 seconds and 2430 MB of memory.

W3: The bolded content mentioned in the appendix appears to be missing in the actual tables.

A3: We will address this issue in the revised manuscript by highlighting the best performance values in the tables using boldface for better clarity.

W4: The notation used in Figure 2 appears to be inconsistent with that in the main text.

A4: We reviewed the symbols in Figure 2 and the corresponding text, and found an inconsistency in the task sequence notation (using both $t$ and $i$). We have corrected this issue and unified the notation by using $t$ to represent the task index.

W5: The meaning of Concat() in Equations (7) and (9) has not been clearly defined.

A5: In Equations (7) and (9), Concat() denotes the row-wise concatenation of pseudo-features from all participating clients into a single matrix. Specifically, if client $k$ provides pseudo-features $\tilde{X}_k \in \mathbb{R}^{n_k \times d}$, then:

$\text{Concat}(\tilde{X}_1, \tilde{X}_2, \ldots, \tilde{X}_K) = \begin{bmatrix} \tilde{X}_1 \\\\ \tilde{X}_2 \\\\ \vdots \\\\ \tilde{X}_K \end{bmatrix} \in \mathbb{R}^{N \times d},$

where $N = \sum_k n_k$. We have added this definition and explanation to Section 4.1 in the revised manuscript.

We sincerely appreciate the professional comments provided by Reviewer 3EQU

W1: Some symbols are not clearly defined; it seems that there exists an inconsistent use of a single notation.

A1: We reviewed all symbol definitions and usages, identified that task indices were inconsistently represented by $i$ and $t$, and have corrected this inconsistency by unifying the notation to $t$.

W2: Some implementation details are not directly provided, such as the hyperparameters used in the work, the replay buffer size, etc.

A2: We have further clarified some experimental details to ensure reproducibility. For example, we specify that LANDER and TARGET generate 10,240 samples per task. Moreover, based on the source code, we report that the momentum of Batch Normalization in LANDER is set to bn_mmt = 0.9, the temperature parameter for knowledge distillation is $\tau = 2$, the temperature coefficient for KL divergence distillation loss is $T = 20.0$, and the number of training steps with synthetic data is student_train_step = 50. For Re-Fed, the number of replay samples is kept consistent with our FedCBDR, as referenced in Section~5.1.

W3: Some designs require more detailed explanation. For example, the process of Pseudo Feature Construction uses global model.

A3: The pseudo-feature construction leverages the global model’s feature extractor to ensure that features from all clients are aligned in a common latent space, which is crucial for meaningful aggregation. Each client first uses the current global model $f_{\theta}^{(i)}$ to map its local samples into a feature matrix $X_k \in \mathbb{R}^{n_k \times d}$. Then, we perform an ISVD transformation: $X_k = U_k \Sigma_k V_k^T$, replace $U_k$ with a random orthogonal matrix $P_k$, and generate pseudo-features $\tilde{X}_k = P_k \Sigma_k V_k^T$, which are subsequently uploaded to the server. The server concatenates all $\tilde{X}_k$, applies SVD, and selects representative samples using leverage scores. We will revise Section 4.1 to provide a clearer explanation of this process.