Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors

We thank the reviewer for his astute observation about the labelling functions.The only mathematical assumption that the theorem makes is that the encoded distributions of the source $P_s$ and the global data domain $P_t$ converge to each other. Practically, this is enforced by the anchor term in the loss function, $\mathcal{L}_{REG}$.

Coming to the interpretation of the bound through the similarity of the labeling functions.

Firstly, it is important to note that the theorem does not require both of the error terms to be small. As long as any single one of the terms is low, we can change the component weights $\alpha$ to rely more on that term , so that we get a tighter generalization guarantee. These component weights alpha are directly tied to the regularization weights $\lambda_{REG}$ and $\lambda_{KD}$. The explanation for either of the labelling functions being close to $f_T$ is given below,

1. The real data and the synthetic data will have similar labeling functions because their distributions are enforced to be similar under an MMD metric during the data generation process. - (Eq. 3) . Therefore $f^{syn}$ will be close to $f_T$

2. The statement about the extended KD data is a bit more subtle. It is crucial to note that $D_{KD}^{syn}$ depends on the client models $M_j$ and is not a static distribution. Towards the end of training, when the empirical KD loss consistently decreases and generalized models are learned, the labeling function of the extended KD data $D_{KD}^{syn}$, which represents the consensus knowledge of all the domains, will correctly classify any example drawn from the global data domain $P_T$. Therefore, we project that the labeling function $f_{KD}^{syn}$ will be close to $f_T$.

We thank the reviewer for the careful review and clarification questions.

Typo on notation: We apologize for the typo. There shouldn’t be a summation over i to N, so the client i’s data is not affected by client j, $\forall i \neq j$. We thank the reviewer for the careful review and have corrected it in our revision, and we sincerely apologize for causing your confusion.

Class-balanced guarantee: We synthesize data based on each class, i.e., for synthesizing data for class c, we will sample real data from P(x|y=c). As a result, we can control the number of synthetic data for each class c, and generate a class-balanced synthetic data set.

Potential privacy leakage: As discussed in [2], using MMD distilled data to train a model can defend against reconstruction attacks and membership inference attacks, which is empirically justified in our Appendix B. For further privacy preservation, we also show we can apply Differential Privacy when distilling synthetic data in Appendix C.

[1] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514–6523, 2023.

[2] Dong T, Zhao B, Lyu L. Privacy for free: How does dataset condensation help privacy?. InInternational Conference on Machine Learning 2022 Jun 28 (pp. 5378-5396). PMLR.

Sorry for the typo, we have corrected it in our revision.

We clarify that the statement “when the local data heterogeneity is severe, the model learning should rely more on the centralized data” is on the assumption that “ synthetic and the extended KD datasets are similar to the global ones” Thus the LHS of Eq(9) will be dominated by $\epsilon_{{P}^T}(f^{Syn})$ and $\epsilon_{{P}^T}(f_{\text{KD}}^{Syn})$. It is intuitive that if synthetic data is ideal aligned with global distribution and we fully rely on synthetic data in local training, the performance gain between training on synthetic data vs on local data is expected to be larger when local data is more heterogeneous. Also, as noted in Theorem 1, the performance is not solely related to hyperparameters $\lambda_{REG}$ and $\lambda_{KD}$. It will be also affected (or dominated) by the fourth term of Eq 8, when we rely on sufficient number of ideal synthetic data (aligned with global distribution).

We thank the reviewer for pointing this out.

The exact mathematical definitions have been added to our revision. Below, we explain their meaning in words.

1. $d_{H \Delta H}$ is a distance metric that relates to the maximum difference in probability assigned to the space of disagreements between any two hypothesis functions drawn from H.
2. $\lambda(P_i)$ refers to the combined generalization loss of the best joint model on both local and client domains.
3. The condition implies that there should exist a function $\psi$ such that the encoded distributions of $P_s$ and $P_t$ converge to each other in probability. This convergence condition is enforced by the loss $\mathcal{L_reg}$, which acts as an anchor to pull all encoded distributions together.

We thank the reviewer for the suggestion. As noted in our Section 3.1, DeSA only requires sharing logits w.r.t. Global synthetic data during model training. Thus it has a relatively low communication overhead compared to baseline methods which require sharing model parameters. For example, if we use ConvNet as our model and set IPC=50, the number of parameters for communication for DeSA will be 10(number of classes) x 50(images/class) x 10(logits/image) = 5k. In comparison, baseline methods need to share 320K parameters (the number of parameters of ConvNet model), which is much larger than DeSA. We also add the discussion in Appendix D in our revision.

We thank the reviewer for the careful review. The $P(\cdot)$ is unused and we have removed $P(\cdot)$ in our revision.

We assume the client nodes form a complete graph in DIGITS and OFFICE experiments. For CIFAR10C experiments, we randomly sample neighboring clients for each round.

The comparison results with FedGen are presented in Homogeneous Model Experiments (Section 5.3). We did not include FedGen in Table 2 as it is not applicable to model heterogeneous scenarios discussed in table 2. FedMD[1] was not considered to be included due to its poor performance, consistent with the observation report in its similar method FedHe [3]. We have corrected the paragraph by removing FedGen and FedMD and adding FedHe in our revision.

[1] Li D, Wang J. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581. 2019 Oct 8.

[2] Zhu Z, Hong J, Zhou J. Data-free knowledge distillation for heterogeneous federated learning. InInternational conference on machine learning 2021 Jul 1 (pp. 12878-12889). PMLR.

[3] Chan YH, Ngai EC. Fedhe: Heterogeneous models and communication-efficient federated learning. In2021 17th International Conference on Mobility, Sensing and Networking (MSN) 2021 Dec 13 (pp. 207-214). IEEE.

Regarding how we merge D^{syn}: We perform simple interpolation (averaging) among clients as it is shown that using this mixup strategy can improve model fairness [1].

Regarding how we define our neighbors: We assume the client nodes form a complete graph in DIGITS and OFFICE experiments. For CIFAR10C experiments, we randomly sample neighboring clients for each round. We have added the clarification in our revision.

[1] Chuang CY, Mroueh Y. Fair Mixup: Fairness via Interpolation. InInternational Conference on Learning Representations 2020 Oct 2.

Finding the minimal amount synthetic data is not the focus of this work. We have replaced “minimal” with “small” in our revision for clarification. Following your suggestion, we have performed a more detailed hyperparameter search on IPC. Specifically, we search IPC from {5, 10, 20, 50, 100, 200} and the experimental results are updated in our revision. Overall, blindly increasing the IPC does not guarantee to obtain optimal intra- and inter-accuracy. It will cause the loss function to be dominated by the last 2 terms of Eq. 8, i.e., by synthetic data rather than minimizing the empirical risk of classification on cross-entropy loss. However, synthesizing larger number of synthetic data may degrade its quality, and the sampled batch for $\mathcal{L}_{REG}$ may fail to capture the distribution.

We thank the reviewer for raising this concern. Let us think about an essential scenario: In 2020, hospitals from different regions want to collaboratively train a COVID-19 classification model. Due to quarantine, people cannot travel outside the local area, so the local hospital can only collect local cases with local strains of Covid viruses. If they train a model that only cares about intra-domain, this personalized model will suffer from a performance drop when the quarantine restriction is revoked and people from one region (with its local virus strains) will travel to other regions and use other hospitals’ models. Thus, we consider intra and inter-domain performances are both important, and allowing each client model to generalize well to other client domains can potentially improve model robustness. Thus, In this work, we consider a different FL setting without sever and define it as decentralized federated mutual learning, which shares the same motivation as the existing work [1].

[1] Huang W, Ye M, Du B. Learn from others and be yourself in heterogeneous federated learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022 (pp. 10143-10153).

We wish to clarify that for this study, our choice of datasets was guided by the objective to align with the widely recognized standards in dataset distillation literature [1,2,3]. The selected datasets, commonly used in FL research, enable direct comparisons with existing methods and model both feature and label heterogeneities, ensuring our findings are relevant and interpretable in current research contexts.

We acknowledge that the distribution-based distillation method employed in our study is optimized for the datasets we selected and may not be directly applicable to high-resolution images, which often require more sophisticated distillation strategies, such as [1,2]. Due to time constraints and the specialized nature of medical image processing for COVID19 classification task, it wasn't feasible to include these in this rebuttal phase.

Notably, obtaining good distilled datasets can be a standalone step to be optimized, but not the focus of our work. Our primary focus was to handle both model and data heterogeneity in a challenging serverless decentralized FL setting and develop a theoretically solid framework to justify the proposed loss functions given being able obtain ideal synthetic data. Our theoretical findings are versatile and can be applied to other distillation strategies, potentially including those suitable for high-dimensional data.

Finally, we would like to reiterate the contributions of our study: we develop a theoretically solid framework to handle both model and data heterogeneity in a serverless decentralized FL scenario, and, empirically, we apply dataset distillation to generate synthetic data that fits our theory. The experimental result on benchmark datasets out-performs existing decentralized FL methods, which validates the effectiveness of DeSA.

We believe our framework lays a solid foundation for future exploration in data and model heterogeneous FL, and we are hopeful that subsequent research will build on our theoretical insights to address a broader range of applications, including those involving high-dimensional real data. We appreciate the reviewer's insights and will highlight potential areas for future exploration.

[1] Cazenavette G, Wang T, Torralba A, Efros AA, Zhu JY. Dataset distillation by matching training trajectories. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022 (pp. 4750-4759).

[2] Yu R, Liu S, Wang X. Dataset distillation: A comprehensive review. arXiv preprint arXiv:2301.07014. 2023 Jan 17.

[3] Zhao B, Bilen H. Dataset condensation with distribution matching. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 2023 (pp. 6514-6523).

What does the assumption of "the encoded distributions of the source $P_s$ and the global data domain $P_t$ converge to each other" mean?

In our paper, we represent the $i$th client model as two parts: the feature encoder $\psi_i$ and classification head $\rho_i$. $\psi_i$ will map the raw data (i.e. the image data in our experiments) to the embedding representation, and $\rho_i$ will output the prediction on top of the embeddings. In our theoretical analysis, the core assumption is that the distributions of embedding representations under the synthetic dataset $P^{Syn}$ and global dataset $P^T$ across clients are similar. We express this mathematically as $\psi_i \circ P^{Syn} \rightarrow \psi_i \circ P^{T}$, where $\psi_i \circ P^{Syn}$ is the distribution of the embeddings over the synthetic dataset and $\psi_i \circ P^{T}$ is that over the global dataset.

What are "encoded distributions of the source" here?

The encoded distributions represent the embeddings derived from the global synthetic dataset. In this context, we use the term 'source data domain' to specifically refer to the synthetic dataset of all the clients.

How can this be enforced by $\mathcal{L}_{REG}$

If we go back to the definition of $L_{REG}$ (i.e., Eq (4) in the paper), it minimizes the per-class distances of the embeddings over the synthetic dataset and the true datasets. When it is approaching zero, it means that the per-class averaged embeddings over the synthetic and true datasets are the same. When we minimize $\mathcal{L}_{REG}$ for every client model, the synthetic data behaves as an anchor, pulling each domain’s encoded representation close to the encoded representation of the synthetic data. Therefore, as the models learn $\psi_i$, they will learn to project the synthetic data distribution $P_s$ into a domain invariant representation $\psi_i \circ P_T$

How can the MMD in Eq. (3) guarantee that $f^{syn}$ is similar to $f_T$?

1. The MMD in Eq. (3) is a label-conditional distance. It requires that the generated local synthetic data and local client data behave similarly under each class.

2. The global synthetic data $D^{syn}$ is the aggregation of all the local synthetic data. Since eq(3) guarantees that the local synthetic data distribution is similar to the local client data distribution, the global synthetic data distribution becomes representative of $P_T$

3. The labelling function from [1] is characteristic to the distribution it is defined upon, therefore, the labelling function $f_{syn}$ is close to $f_T$.

[1] Ben-David S, Blitzer J, Crammer K, Kulesza A, Pereira F, Vaughan JW. A theory of learning from different domains. Machine learning. 2010 May;79:151-75.

We thank the reviewer for raising this concern. Let us think about an essential scenario: In 2020, hospitals from different regions want to collaboratively train a COVID-19 classification model. Due to quarantine, people cannot travel outside the local area, so the local hospital can only collect local cases with local strains of Covid viruses. If they train a model that only cares about intra-domain, this personalized model will suffer from a performance drop when the quarantine restriction is revoked and people from one region (with its local virus strains) will travel to other regions and use other hospitals’ models. Thus, we consider intra and inter-domain performances to be both important, and allowing each client model to generalize well on other client domains can potentially improve model robustness. Thus, In this work, we consider a different FL setting without sever and define it as decentralized federated mutual learning, which shares the same motivation as the existing work [1].

[1] Huang W, Ye M, Du B. Learn from others and be yourself in heterogeneous federated learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022 (pp. 10143-10153).

We thank the reviewer for the careful review and pointing out the typos. We have addressed the typos as follows. We have also done more careful proofreading in our revision.

• There shouldn’t be a summation over i to N, and we have corrected it.
• The L_{CE} description shouldn’t be there and we have removed it.
• We corrected it into DeSA.

We thank the reviewer for the great question. In Eq. 8, if we have limited local data, the bound will be dominated by $\alpha^{Syn}$ since $\alpha$ is small. Thus, the performance depends on the quality of global synthetic data. However, we would also like to point out limited local training data may affect the generation of synthetic data and further decrease its utility.

For DIGITS and OFFICE experiments, we assume each dataset represents one client following [1]. We further clarify the setting in our revision.

[1] Li X, JIANG M, Zhang X, Kamp M, Dou Q. FedBN: Federated Learning on Non-IID Features via Local Batch Normalization. InInternational Conference on Learning Representations 2020 Oct 2.

We thank the reviewer for suggesting that we should split a dataset and assign different models to subsets. In fact, we implement this strategy in our CIFAR10C experiments, where we split Cifar10C into 57 clients (subsets) and randomly assigned different model architectures to them. The objective of Table 2 is to show that our method can obtain competitive inter and intra-domain performance under both data and model heterogeneous scenarios.

Dear Reviewer xb5U,

As the rebuttal deadline is approaching, we would like to know if our rebuttals have addressed your concerns and questions. Also, we have tried our best to reflect your brilliant feedback to our revision. We appreciate the opportunity to discussing during this stage and are delighted to address your further question if there is any. If you are satisfied with our response and revision, we are grateful if you could kindly re-consider the rating for DeSA.

Best Regards,

DeSA authors

We thank the reviewer for the constructive comment. Following your suggestion, we have performed a more fine-grained hyperparameter search on Reg loss and KD loss. Specifically, we search $\lambda_{KD}$ from {0, 0.1, 0.5, 1, 2} and $\lambda_{REG}$ from {0, 0.01, 0.02, 0.05, 0.1}, and the experimental results are updated in our revision. The findings are aligned with our original version that $\lambda_{KD}$ helps improve the inter-accuracy increases, and $\lambda_{REG}$ helps improve the intra-accuracy as well as the inter-accuracy within a certain range (observe the peak inter-accuracy at $\lambda_{REG}=0.01$).

We thank the reviewer for the clarification question. We agree the privacy of the synthetic data is important, and we provide a DP version for synthetic data in our Appendix C. In addition, we have shown our method can defend against Membership Inference Attack (MIA) [1] in Appendix B. Regarding raw data recovery, since we don’t share gradients among clients (we only share logits w.r.t. global synthetic data), the commonly used gradient inversion attacks [2,3] may not be able to recover original raw data.

[1] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. IEEE, 2022a.

[2] Geiping J, Bauermeister H, Dröge H, Moeller M. Inverting gradients-how easy is it to break privacy in federated learning?. Advances in Neural Information Processing Systems. 2020;33:16937-47.

[3] Huang Y, Gupta S, Song Z, Li K, Arora S. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems. 2021 Dec 6;34:7232-41.

We thank the reviewer for his interesting remark. However, we strongly believe that our theorem gives remarkable insights into the empirical success of our algorithm.

• The only assumption that our theorem makes is the convergence in probability of the encoded target and source domains, which is empirically enforced by our anchor loss regularizer $L_{reg}$. This is a novel condition in theorem 1, that inspired our design of the empirical anchor regularization term in Equation (7).

• It is of paramount importance to note that the previous work of Ben-David 2010 et al, whose shoulders we build Theorem 1 upon, has no connections to any practical algorithm at all. It was our novel contribution to develop a bound that incorporated empirical connections like the labelling function of synthetic data $f^{syn}$ and the Extended KD data $f^{syn}_{KD}$. These labelling functions connect deeply with our empirical anchor and KD regularization losses . Theorem 1 helps us predict the global performance of models using the component weights $\alpha$ given to the empirical regularization terms. We have further clarified these connections in the appendix of our revision.

• Our newly added ablation studies clearly show that improving the quality and quantity of synthetic data ($D^{syn}$) boosts both inter and intra accuracy. Remarkably, Theorem 1 accurately predicted this outcome. In simpler terms, the theorem suggests that better synthetic data reduces the generalization error of labeling functions ($f^{syn}$ and $f^{syn}_{KD}$), tightening the upper bound on global generalization error. This alignment between theory and experiment underscores the reliability of Theorem 1 in predicting empirical outcomes.

• The connection between the quality of synthetic data and the upper bound in Theorem 1 is further confirmed during experiments in Section C of the Appendix. The injection of noise reduces the quality of the global synthetic data $D_{syn}$, which thereby worsens the upper bound guarantee in Theorem 1. This conforms with the empirical observation of a drop in inter and intra-client accuracy.

Therefore, we strongly believe that our theory is connected with the empirical side of our problem, and helps in explaining it’s results.

We would like to point out that Ben - David 2010 et al’s work was a general bound for learning on new domains. Our theory’s contribution is not only applying Ben-David et al’s work to our scenario, but also connecting it to our main algorithm through the addition of terms like $f^{syn}$ and $f^{syn}_{KD}$.This addition introduces novel mathematical insights into the connection between labelling functions and the anchor and KD regularization losses.

We also provide a novel stronger version of our theorem in Proposition 2, that holds under the mild conditions enforced by our algorithm, and other novel and useful Lemmas in the appendix that theoretically motivate our algorithm.

Following your valuable suggestion, we have edited the reference for Ben-David et al. in our revision.

We thank the reviewer for the suggestion. Since our main focus is to solve the challenging issue - decentralized federated learning with data and model heterogeneities, we decided to put more focus on how our method can utilize data distillation to solve the problem in our manuscript.

We agree that showing the privacy guarantee is important, so we have discussed the empirical privacy guarantee in Section 3.2 and pointed the readers to our Appendix B and C for our privacy-preservation results. Specifically, we provide a DP version for synthetic data in our Appendix C. In addition, we have shown our method can defend against Membership Inference Attack (MIA) [1] in Appendix B. We share the concern that incorporating a comprehensive discussion into the main text might limit the space available for elaborating on the technical details essential for understanding our innovative method. Our priority is to ensure that the core aspects of our approach are communicated effectively and clearly. Other reviewers seem to like our current presentation. In the revised manuscript, we underscored the sentence in the main text that directs readers to the appendix.

[1] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. IEEE, 2022a.

Clarification of client structure and the objective of Eq. 2

We thank the reviewer for pointing out the typo of Eq. 2. The summation should be from 1 to N(C_i), and we have corrected it in our revision. As shown in Algorithm 1, our method is designed to work only to receive neighbor node information, and DeSA is designed for peer-to-peer decentralized network. Although we are working on a peer-to-peer decentralized network but the loss requires all nodes’ information, we can leverage the FastMix algorithm to aggregate all nodes’ information [1,2]. This method can aggregate all nodes' information via adjacent nodes’ communication at a linear speed. It is very common in fully decentralized optimization. In fact, our method can also work if each node can only receive neighbor nodes’ information, and we empirically show the feasibility in our CIFAR10C experiments by sampling neighboring clients.

Clarification of (x|y)

We thank the reviewer for the clarification question. $\psi(x|y)$ means the comparison is conditional on label y , which allows us to distill synthetic data based on each class c.

Clarification of “randomly sampled feature extractors”

We thank the reviewer for the clarification question. Using randomly sampled feature extractors allows us to perform empirical MMD loss as described in [3]. To implement it, we simply randomly initialize the model parameters for each round with PyTorch.

Clarification of Eq. 4

We appreciate reviewer’s careful review. . We have removed the unused notation $P$ and add description of $K$ in our revision.

[1] Ye H, Zhou Z, Luo L, Zhang T. Decentralized accelerated proximal gradient descent. Advances in Neural Information Processing Systems. 2020;33:18308-17.

[2] Luo L, Ye H. Decentralized Stochastic Variance Reduced Extragradient Method. arXiv preprint arXiv:2202.00509. 2022 Feb 1.

[3] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514–6523, 2023.

Dear Reviewer VrA4,

As the rebuttal deadline is approaching, we would like to know if our rebuttals have addressed your concerns and questions. Also, we have tried our best to reflect your brilliant feedback to our revision. We appreciate the opportunity to discussing during this stage and are delighted to address your further question if there is any. If you are satisfied with our response and revision, we are grateful if you could kindly re-consider the rating for DeSA.

Best Regards,

DeSA authors

We apologize for the typo in Eq. 3 which may lead to misunderstanding. There should not be a summation of clients in Eq. 3, and we have corrected it in our revision.

To answer your question regarding Eq. 3 and Eq. 4: No, Eq. 3 and Eq. 4 are not overlapped. First, we hope to correct the typo in Eq. 3 – there should not be a summation of clients in Eq.3 as we stated it as a local data distillation loss. Sorry for leading the misunderstanding. The objective of Eq. 3 is to distill local synthetic data by minimizing the MMD of synthetic data and real data; thus, it is used to update client i’s local synthetic data ($D^{syn}_i). Differently, Eq. 4 aims to update the feature extractor parameters using Supervised Contrastive Loss. Since the inputs of Eq. 4 is *global synthetic data*($D^{syn}$) (the interpolation of every client’s *local synthetic data*) instead of client i’s *local synthetic data* ($D^{syn}_i), minimizing Eq. 3 does not imply Eq. 4 will be relatively small.

Finding the minimal amount synthetic data is not the focus of this work. We have replaced “minimal” with “small” in our revision for clarification. Following your suggestion, we have performed a more detailed hyperparameter search on IPC. Specifically, we search IPC from {5, 10, 20, 50, 100, 200} and the experimental results are updated in our revision.

Overall, blindly increasing the IPC does not guarantee to obtain optimal intra- and inter-accuracy. It will cause the loss function to be dominated by the last 2 terms of Eq. 8, i.e., by synthetic data rather than minimizing the empirical risk of classification on cross-entropy loss. However, synthesizing larger number of synthetic data may degrade its quality, and the sampled batch for $\mathcal{L}_{REG}$ may fail to capture the distribution.

We thank the reviewer for the constructive comment. Following your suggestion, we have performed a more fine-grained hyperparameter search on Reg loss and KD loss. Specifically, we search $\lambda_{KD}$ from {0, 0.1, 0.5, 1, 2} and $\lambda_{REG}$ from {0, 0.01, 0.02, 0.05, 0.1}, and the experimental results are updated in our revision. The findings are aligned with our original version that $\lambda_{KD}$ helps improve the inter-accuracy increases, and $\lambda_{REG}$ helps improve the intra-accuracy as well as the inter-accuracy within a certain range (observe the peak inter-accuracy at $\lambda_{REG}=0.01$).

Thanks for noticing the results, which is in fact align with our theoretical analysis. As shown in Theroem 1, if we have a large $\lambda_{KD}$, the error bound is dominated by the KD loss, which is for improving the inter accuracy purpose. However, a too large KD loss will hurt intra accuracy.

We thank the reviewer for the constructive question. Let us think about an extreme non-iid scenario where clients lack data in class c, our Eq. 2 intuitively suggests that client i, lacking samples in class c, cannot reasonably generate data samples with label c. In such instances, we adjust the interpolation ratio according to the class prior, thus deweighting client i on synthetic data class c and adding weight to the clients with abundant data in class i. Consequently, this implies client i will have a greater reliance on other clients’ data in class c, thereby addressing heterogeneity and enhancing overall generalizability.

Regarding the quality of synthetic data: We hope to clarify that the purpose of data distillation is not to generate hight-fidelity images that look similar to the original image, but rather the ones keeping the similar level of utilities. Our generated distilled data share similar patterns as the ones presented in the original distribution-based dataset distillation paper [1] (see its Fig. 2).

[1] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514–6523, 2023.

Dear Reviewer R93g,

As the rebuttal deadline is approaching, we would like to know if our rebuttals have addressed your concerns and questions. Also, we have tried our best to reflect your brilliant feedback to our revision. We appreciate the opportunity to discussing during this stage and are delighted to address your further question if there is any. If you are satisfied with our response and revision, we are grateful if you could kindly re-consider the rating for DeSA.

Best Regards,

DeSA authors