W1. Discussion on Indicator Function.

Response: We appreciate the reviewer's insightful comments. Below is a comprehensive explanation:

(1) Motivation of the Indicator Function. The Indicator Function $I_{\lambda}(l_i, \sigma_i)$ is designed to dynamically adjust sample weighting based on loss values and uncertainties, inspired by self-paced learning [1]. It follows curriculum learning principles, prioritizing simpler tasks before more complex ones, to improve training convergence efficiency. The proof of this assertion can be found in global rebuttal.

(2) Derivation of the Indicator Function. Our design is heuristic, similar to confidence-aware cross-entropy [2]. It consists of a loss-amplifying term and a regularization term, enhancing high-loss sample contributions while regularizing uncertainty estimation. To further discussion, see the convergence speed discussion of CRFed with FedAvg in the global rebuttal.

(3) The Relationship Between the Indicator Function and the Optimal Data Distribution. Please refer to the response to Q1 for detailed explanation.

W2. Necessity of diffusion model

Response: Thanks for your comment. In CRFed, the diffusion-based data harmonization process is integrated into each global aggregation round. This means the harmonization update occurs every time the global model is aggregated and broadcasted to the clients, specifically set to once per global round. The model is trained online during the federated learning process.

To address the reviewer's concern, we conducted comparative experiments with alternative methods to estimate the optimal data distribution based on the indicator function:

• Simple Weighted Sampling (SWS): Directly adjusts sampling probabilities based on indicator function values without additional transformation.
• Kernel Density Estimation (KDE): Uses KDE to smooth and adjust data distribution based on the indicator function.

The results, shown in Table 8 on the one-page pdf, indicate the diffusion model achieves the best performance. The diffusion model's iterative denoising process effectively harmonizes data distributions, ensuring robust and consistent alignment, crucial for improving convergence and performance.

From a practical perspective, CRFed harmonizes these data distributions, ensuring consistent updates and improved convergence. CRFed outperforms state-of-the-art methods and remains effective, as its training can be performed during the local phase(although running on server), ensuring real-time operation of the FL system.

W3. Issue about Eq. (1) and Eq. (16)

Response: : We appreciate the reviewer's observation. The indicator function is defined at the client level but computed from the sample level. Specifically, Eq. (1) calculates the indicator function for each sample, while Eq. (16) aggregates it at the client level. There is no inconsistency between Eq. (1) and Eq. (16). We will improve clarity in future version.

Q1. Why is the optimal uncertainty obtained by minimizing the indicator function?

Response: Thank you for your comment. Minimizing the indicator function is crucial for:

• Balancing Loss and Uncertainty: Minimizing $I_{\lambda}(l_i, \sigma_i)$ ensures higher importance for samples with higher loss values while controlling uncertainty.
• Preventing Overfitting: The regularization term $\lambda (\log \sigma_i)^2$ discourages high uncertainties, preventing overfitting to unreliable samples.

Additionally, minimizing the indicator function helps control the parameter update magnitude, as shown by:

where $\eta$ is the learning rate, and $C$ is a constant that bounds the term $\left( \sigma_i^* + (l_i - \tau) k + 2\lambda \frac{\log \sigma_i^*}{\sigma_i^*} k \right)$. This inequality indicates that by choosing $\sigma_i^*$ to minimize the indicator function, we can control the magnitude of the parameter updates.

Q2. Discussion on the distribution decoder

Response: Thank you for your comment. The role of the Distribution Decoder is essentially to map the latent representation $z_t$ to the distribution space, so the training is conducted offline on a known ground truth distribution. For a specific task, using the task's test set (or validation set) is a good choice. Other details can be found in section A.2.2 of the original text. We hope these explanations can address your concern.

Q3. Details of model training

Response: Thank you for your comment. The diffusion model, the model encoder, and the distribution decoder are all trained globally. Specifically, the model encoder and the distribution decoder are trained offline.

To quantify the training overhead introduced by our approach, we have measured the wall-clock time per round and the total convergence time. The results are shown in Table 2 on the one-page pdf. We observe that the wall-clock time per round for CRFed is slightly higher compared to other methods due to the additional noise/denoise operations, but the total convergence time is comparable to other methods. The slight increase in computational cost is balanced by the improved model performance, making it a worthwhile trade-off.

Reference

[1] Self-paced learning: An implicit regularization perspective. AAAI, 2017.

[2] Data parameters: A new family of parameters for learning a differentiable curriculum. Neurips, 2019.

Thank you for your timely feedback and for raising the score. We’ve provided the following response to address the remaining questions you raised:

1. Motivation for Diffusion Model

Previous importance sampling methods typically require prior analysis of the data relevance at each client-side [3,4] or necessitate deriving optimal sampling weights based on assumptions such as the convexity of the loss function [1,2]. While these methods offer strong theoretical guarantees, they are somewhat limited in their adaptability to real-world FL scenarios. For instance, both FedIR[3] and Harmony[4] assume that the server has knowledge of the local distributions of all clients. Although this assumption does not violate the privacy-preserving principles of FL, it can be challenging to obtain in real-world applications.

In contrast, the diffusion model based method we proposed does not depend on these assumptions. Instead, it iteratively adjusts the data distributions during the FL process itself. This enables the model to dynamically harmonize the diverse, non-IID data across clients without requiring explicit distributional assumptions or centralized access to all client data distributions. Guided by the indicator function, our CRFed can derive the optimal sampling strategy for each local node.

Moreover, as shown in the table below, empirical experiments demonstrate that the diffusion model achieves superior performance, outperforming other benchmark methods.

The performance of different importance sampling methods on CIFAR-100 under various β values.

It’s worth noting that this comparison isn’t entirely fair, as each importance sampling method operates under different assumptions. For example, ISFL requires a validation set to update the empirical gradient Lipschitz constants for each local model, while FedIR requires all clients to upload the conditional distribution of images given class labels that matches the target distribution. Nevertheless, our CRFed outperforms the others even under less restrictive conditions—unlike ISFedAvg and ISFL, it doesn’t require assumptions about the loss function or gradient variance, and unlike FedIR and Harmony, it doesn’t require centralized access to all client data distributions before calculating the importance sampling weights.

2. Additional Computation

We acknowledge that CRFed requires additional computational resources, primarily on the server side. However, in FL, the main computational burden typically lies in the local model training and communication. In practical FL systems, increasing server-side computation to achieve performance gains is often desirable, as it does not compromise the real-time operation of the FL system and tends to offer stronger economic benefits. For example, in CLIP2FL[5], to mitigate data heterogeneity and class imbalance, the server generates federated features based on client-uploaded gradients and uses CLIP's text encoder for prototype contrastive learning. Similarly, in FedMRUR[6], the server must compute compressed data received from clients, calculate the corresponding logits, and perform global knowledge matching, which involves substantial computational intensity.

Lastly, we would like to express our sincere gratitude to Reviewer CpsC. Your feedback has inspired us to delve deeper into discussions regarding CRFed, and the additional experiments have further strengthened its contributions. We plan to include these results and discussions in future versions. We hope this response addresses some of your concerns.^_^

References

[1] ISFedAvg. Federated learning under importance sampling. IEEE Transactions on Signal Processing. 2022.

[2] ISFL. Federated Learning for Non-iid Data with Local Importance Sampling. IEEE Internet of Things Journal. 2024

[3] FedIR. Federated Visual Classification with Real-World Data Distribution. ECCV. 2020.

[4] Harmony. Heterogeneity-aware hierarchical management for federated learning system. MICRO. 2022.

[5] CLIP2FL. CLIP-Guided Federated Learning on Heterogeneity and Long-Tailed Data. AAAI. 2024.

[6] FedMRUR. Federated learning with manifold regularization and normalized update reaggregation. Neurips. 2023.

W1. Some notations could be more clearly defined.

Response: Thanks for pointing this out. In our framework, $\tau$ represents a confidence threshold that determines the difficulty level of samples based on their loss values. The term $(l_i - \tau) \sigma_i$ in the Indicator Function adjusts the weight of each sample based on its loss value $l_i$ relative to the threshold $\tau$. This adjustment mechanism prioritizes samples with higher loss values (more difficult samples) when $l_i > \tau$, giving them higher weights, while samples with lower loss values (easier samples) are given lower weights. The impact of $\tau$ on the learning process is twofold:

• By setting an appropriate $\tau$, the model can focus on learning from more difficult samples first, thereby implementing a form of curriculum learning. This helps the model to progressively handle more complex patterns in the data, leading to improved generalization and robustness.
• $\tau$ can be dynamically adjusted during the training process to reflect the model's evolving understanding of the data. Initially, $\tau$ can be set to a lower value to focus on easier samples, and as training progresses, $\tau$ can be increased to prioritize more difficult samples. This dynamic adjustment ensures that the model continually challenges itself, preventing stagnation and promoting continuous improvement.

W2. Evaluation the performance and efficiency of the denoising steps.

Response: We appreciate the reviewer's concern regarding the scalability of the iterative denoising process, especially when dealing with very large datasets or a high number of iterations. To address this, we have conducted additional experiments to evaluate the performance and efficiency of the denoising steps under such scenarios.

We conducted experiments on the CIFAR-100 and NIPD datasets, varying the dataset size and the number of iterations in the denoising process. The key parameters evaluated include:

• Dataset sizes: Full CIFAR-100 (50,000 samples) and NIPD (80,000 samples)
• Number of denoising iterations: 10, 50, 100, 200

The results are shown in Table 7 on the one-page pdf. We can find that the performance improvements (in terms of accuracy for CIFAR-100 and mAP for NIPD) saturate as the number of iterations increases. These results demonstrate that while the iterative denoising process is effective, the benefits in performance diminish beyond a certain number of iterations. Therefore, for practical applications, we recommend limiting the number of denoising iterations to around 100, where a good balance between performance and efficiency is achieved. This approach ensures that the method remains scalable even for large datasets.

W3. A brief discussion on future work.

Response: Thanks for the suggestion. Future work could explore more sophisticated noise injection techniques in the diffusion-based data harmonization process. For instance, adaptive noise schemes that consider the specific characteristics of local data distributions could potentially improve the alignment of data across clients. We will include relevant discussions in future version to inspire further work.

W1. Sensitivity analysis.

Response: We appreciate the reviewer's insightful comment regarding the sensitivity analysis of our hyperparameters. To address this, we conducted additional experiments to analyze the sensitivity of the key hyperparameters in our CRFed framework, namely the variance of Gaussian noise ($\beta_t$), the regularization coefficient ($\lambda$), and the dynamically adjusted confidence threshold ($\tau$).

We performed the sensitivity analysis on the CIFAR-10 and CIFAR-100 datasets, which were also used in the original experiments. The default values for the hyperparameters in our original experiments were:

• Variance of Gaussian noise, $\beta_t$: 0.1
• Regularization coefficient, $\lambda$: 0.1
• Dynamically adjusted confidence threshold, $\tau$: dynamically adjusted starting from 0.5

We varied each hyperparameter while keeping the others fixed to their default values and measured the test accuracy. The results are summarized in Table 3,4,5 on the on-page pdf. The results indicate that our CRFed framework is relatively robust to variations in these hyperparameters.

W2. Additional experiments addressing scenarios where data distributions are extremely diverse.

Response: We appreciate the reviewer's concern regarding the effectiveness of our harmonization mechanism in handling extremely diverse data distributions. To address this, we have conducted additional experiment to evaluate and demonstrate the robustness and adaptability of our CRFed framework under such challenging conditions.

To empirically validate the effectiveness of our approach under extreme data heterogeneity, we conducted additional experiments using the CIFAR-100 dataset with even smaller Dirichlet concentration parameters ($\beta$) to simulate highly imbalanced and diverse data distributions. Specifically, we set $\beta$ to 0.01 and 0.05 to create scenarios with extreme non-IID characteristics. The results are shown in Table 6 on the one-page pdf and demonstrate that CRFed framework significantly outperforms the baseline FedAvg method even under extreme data heterogeneity.

W3. Discussion on additional overhead.

Response: Thank you for your comment. We have measured the wall-clock time to evaluate the computational overhead introduced by the proposed approach. Results are shown in Table 2 on the one-page pdf. We observe that the wall-clock time per round for CRFed is slightly higher compared to other methods due to the additional noise/denoise operations. However, the total convergence time is comparable to other methods. Although CRFed involves additional computations per round, the convergence in terms of accuracy and model robustness is achieved efficiently. The slight increase in computational cost is balanced by the improved model performance, making it a worthwhile trade-off.

W4. Discussion on equation (3).

Response: We thank the reviewer for recognizing the value of Theorem 3.1. We acknowledge that the purpose of equation (3) in the derivation process may not have been sufficiently clear. The purpose of this equation is to define the indicator function $I_{\lambda}(l_i, \sigma_i)$, which measures the reliability of each sample based on its loss value $l_i$ and associated uncertainty $\sigma_i$. This function is critical for the self-paced learning mechanism in our framework, where samples are prioritized based on their difficulty and uncertainty.

W5. Formatting issue.

Response: Thanks for the suggestion. In future versions, we will make the corresponding annotations.

W1: Further elaboration on the selection of the methods

Response: Thanks for pointing this out. Below, we offer a detailed explanation and theoretical basis for the key methods employed in CRFed framework.

1. The Indicator Function $I_{\lambda}(l_i, \sigma_i)$ is designed to dynamically adjust sample weighting based on loss values and uncertainties, inspired by self-paced learning [1]. The construction of the indicator function is grounded in the principles of curriculum learning, which prioritizes simpler tasks before progressively addressing more complex ones. The design of the Indicator Function $I_{\lambda}(l_i, \sigma_i)$ is intended to improve training convergence efficiency. The proof of this assertion can be found in global rebuttal.

2. The diffusion-based data harmonization mechanism is the core to our approach, aiming to mitigate data distribution disparities. This method is based on the principles of denoising diffusion probabilistic models (DDPMs), which have demonstrated success in various generative tasks. The iterative process of noise addition and removal aligns data distributions effectively, reducing heterogeneity across clients. Through the diffusion-based data harmonization mechanism, we can ensure that the data distributions of all clients gradually converge during the noise addition and removal processes, thereby reducing the impact of data heterogeneity on model updates.

W2: More information on the update frequency of the harmonization approach and respective curriculum sequence of the clients' training samples.

Response: Thanks for the suggestion. In the CRFed framework, the diffusion-based data harmonization process is integrated into each global aggregation round. This means the harmonization update occurs once per global round, ensuring timely adjustments to data distributions across clients.

• Maximum global rounds ($T_G$): 100
• Local training cycles per global round ($E_l$): 1

Our curriculum learning approach involves dynamically adjusting the sampling weights of clients' training samples based on their difficulty, as measured by the Indicator Function $I_{\lambda}(l_i, \sigma_i)$. This function is recalculated at each local training cycle to reflect the latest state of the global model. The curriculum sequence progresses from easier to more difficult samples, facilitating a self-paced learning paradigm.

W3: Theoretical framework.

Response: Thanks for the suggestion. In the global rebuttal, we have added theoretical analysis.

Q1：More discussion on the Indicator Function.

Response: We appreciate the reviewer's detailed comments. Our indicator function, inspired by confidence-aware cross-entropy [2], includes a loss-amplifying term and a regularization term to amplify high-loss samples' contributions while regularizing uncertainty estimation. The Lambert W function helps find the optimal uncertainty $\sigma_i^*$, ensuring difficult samples are appropriately weighted during training. Subtracting the confidence threshold $\tau$ from the loss value $l_i$ centers the loss values, making it easier to prioritize difficult samples (i.e., those with $l_i > \tau$). The uncertainty $\sigma_i$ ranges from $10^{-3}$ to $10$ and $\tau$ is dynamically adjusted based on the weighted average loss, initially set to 1.0 and updated every 10 rounds to reflect the dataset's evolving difficulty. We will include these details in future versions of the manuscript. Thank you for your valuable feedback.

Q2: Discussion on the adaptive learning rate.

Response: Thank you for your comment. Adjusting the learning rate $\eta_i$ based on $I_{\lambda}(l_i, \sigma_i)$ ensures that clients with more challenging data receive higher learning rates, allowing significant updates. This balances the learning pace among clients, preventing dominance by any single client and ensuring equitable contributions. Higher indicator function values typically correspond to more difficult or uncertain data, leading to slower convergence with a uniform learning rate. Increasing the learning rate for these clients accelerates their learning, speeding up overall convergence.

Additional experiments (see Table 1 on the one-page pdf) show that models with adaptive learning rates based on the indicator function demonstrate faster convergence and higher final accuracy compared to fixed learning rates.

Q3. The principle and details of noise/noise operation.

Response: Thank you for your comment.

The noise/denoise operation in the CRFed framework is performed at every global federation round. This ensures consistent alignment of data distributions across clients, maintaining effective model updates. After each global model aggregation, noise is added and the denoising process adjusts the data distributions before the next round of local training begins.

We have measured the wall-clock time to evaluate the computational overhead introduced by the proposed approach, results are shown in Table 2 on the one-page pdf. Although our system involves additional computations per round, he total convergence time is comparable.

Q4. Discussion on the $\beta$ and experimental details.

Response: Thank you for your comment. In our experiments, we used the Dirichlet distribution to create non-IID data partitions. The $\beta$ value, which controls the degree of data heterogeneity, was set to 0.5 for both domains in the experiments presented in Figure 4. We have considered all available clients at every round of federated learning in our experiments. We hope these clarifications address your concerns.

Q5. Formatting issue.

Response: Thank you for pointing out this. We will fix the citations style in future version.

Reference

[1] Self-paced learning: An implicit regularization perspective. AAAI, 2017.

[2] Data parameters: A new family of parameters for learning a differentiable curriculum. Neurips, 2019.