A Mutual Information Perspective on Federated Contrastive Learning

We thank you for the insightful comments and for acknowledging the theoretical motivation and soundness of our work. We appreciate that you find our results interesting and, together with the shared response, we will provide more details in support of our work

• Empirical results: we consider FedSimCLR as one instance of a method that optimizes a global pretraining objective, stemming from the following intuition. Since on each client the model contrasts between datapoints of the same distribution, a way to close the gap to the centralized model training is by having the model also distinguishing between users as well, i.e., a user-verification objective. In the case of mutual information this becomes precise due to the user-verification task acting as a lower bound to the gap between the global and local MI. Furthermore, we also demonstrate theoretically at Proposition 3 that closing the gap with such an auxiliary objective is beneficial, as in itself is a lower bound to the, unavailable, label classification objective. Based on these intuitions from our perspective and theory, we do extend both Spectral CL and SimSiam in a similar manner (i.e., Spectral CL + UV and SimSiam +UV) and we do observe that the results generalize similarly. However, Spectral CL does better only when incorporating the results from our theory and is worse than FedSimCLR otherwise (e.g., compare FedSimCLR against Spectral CL without the UV loss). Now as to why Spectral CL + UV does better than FedSimCLR, this could be attributed the fact that in the centralized setting, Spectral CL can be better than SimCLR [2] and, through the modifications stemming from our intuitions and theory in the case of MI, can also do better in specific federated settings. Exploring alternative dependency metrics such as [6] in the federated setting is definitely interesting and something we will explore in future work.

• Relation to pseudo-labelling approaches: while from our perspective there are no pseudo-labels, we do agree that having these results as well will help providing a better overall picture. To this end, we are running experiments with two pseudo-label approaches, SemiFed and CBAFed, and so far we observe similar outcomes: the UV loss motivated by our theory is beneficial in the case of label skew.

• Supervised baseline: you are partially correct, in that the supervised baseline corresponds to variant FedAvg. Specifically, we use the superior server-side averaging FedAdam proposed by [1], as we explain in Appendix A. In [1] the authors show empirically that this approach is strictly superior to SCAFFOLD for the cross-device setting, which is why we have not included SCAFFOLD. We have experimented with FedProx but did not see any effect of $\mu>0$ for our choice of $E=1$ across experiments unless setting $\mu>>1$ to detrimental effect. FedBN is a proven FL algorithm under covariate shift, however it prescribes the existence of BatchNorm layers in the network, which we choose to omit. FedBN further assumes the existence of a data-set for computing statistics on a new client, which is an assumption standard FedAvg (FedAdam) does not require with GroupNorm. In summary, we believe that FedAdam is a strong supervised baseline under the given assumptions.

We hope that with the above we have addressed your concerns, and we are happy to continue the discussion otherwise.

[1] Adaptive Federated Optimization, Reddi et al., ICLR 2021

[2] Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss, HaoChen et al., NeurIPS 2021

We thank you for the thoughtful review and the encouraging comments about our work. We hope that with the shared response and the following we address your concerns:

• Theory: Indeed, Sordoni et al. has also proposed a decomposition of the MI, but we argue that the main goal of Sordoni et al. is different than ours. Sordoni et al. decomposes the MI to get a better estimation of the MI with InfoNCE type of bounds, whereas in our case the decomposition happens due to the partitioning of the data in the federated setting. As a result, we believe that our contribution is orthogonal, since, e.g., the methods from Sordoni et al. could be applied for the SimCLR training on each client. Overall, we believe that the decomposition of the MI itself is not our novel result per se, as it is a trivial consequence of the mutual information properties. We simply highlight this decomposition in order to show that a) simple SimCLR on each client (a common baseline in federated SSL [1, 2]) optimizes one term of the decomposition and b) (which is one of our main results) how through that decomposition and an auxiliary user-verification task we can lower bound the global mutual information. Furthermore, through this (similar) decomposition we were able to propose a novel, from the perspective of MI, extension of SimCLR (and our baselines) to the semi-supervised setting. We will update the paper accordingly.

• Amount of views: we use only two views as we adopt the standard convention for SSL methods.

• Algorithms for federated training: indeed, this is a good point and we updated the paper to include algorithms in the appendix.

• Performance relative to baselines: please also see the response to wQ6s; overall the baselines do sometimes better only after we incorporate the intuitions and theoretical results from our MI and FedSimCLR discussion.

• Computational overhead: overall, we, at most, include just two additional heads to the feature encoder which, compared to the feature encoder itself, do not add have much extra computational overhead. We have revised the paper accordingly.

We hope that with the shared response and the above we address your concerns, and we are happy to continue the discussion otherwise.

[1] Federated Unsupervised Representation Learning, Zhang et al., 2020

[2] Federated Contrastive Learning for Decentralized Unlabeled Medical Images, 2021

Dear 3B5H,

The discussion period is nearing its end, so we would like to ask you whether our rebuttal and revised submission adequately address your concerns. More specifically, per your request, we have done the following changes in the submission

• We have discussed Sordoni et al.
• We have added in the appendix the algorithms for federated training with our methods
• We have discussed about the convergence guarantees of our methods
• We have increased the diversity of our experiments by considering more baselines in both the unsupervised and semi-supervised setting.
• We have updated the discussion to better highlight that the baselines do better only after we incorporate the intuitions from our theory and mutual information perspective.
• We have discussed the computational overhead of our methods.

We are happy to engage further in discussions to clarify any confusion or misunderstanding.

Thank you for reviewing our work and for acknowledging the theoretical soundness of our proposed methods as well as the efficacy of our semi-supervised setting. Regarding your specific points:

• Complexity of the experiments: we chose these settings as they are the most popular settings considered in the federated SSL literature. Furthermore, while the difficulty of the image dataset itself is lower compared to settings typically considered in centralized SSL (i.e., ImageNet), they are still difficult from the perspective of federated learning due to the extreme non-i.i.d.-ness we consider, e.g., class proportions from Dirichlet sampling with $\alpha = 0.01$ together with covariate shift, i.e., our joint shift setting.

• Applying our objective to other centralized pretraining objectives: we have indeed demonstrated how our insights apply to other pretraining objectives, be it contrastive, i.e., SimCLR (which is an instance of InfoNCE) and Spectral CL, or non-contrastive, i.e., SimSiam, by intuitively using the same structure for the loss (see the paragraph above the “Unsupervised Setting” in the experiment section). We will expand more about this in the appendix. Indeed, in doing so we observe similar patterns: In the fully unsupervised case with Spectral CL we have better performance in the presence of label skew when adding the UV loss. We also extended these baselines in the same way to the semi-supervised setting, i.e., applying the pretraining objective between elements that belong to the same class while adding an auxiliary classification head that predicts the correct class for those datapoints where it is available. In this scenario we also observe similar improvements, with our SimSiam modifications and the UV loss we significantly improve performance. Extending to feature contrastive approaches such as Barlow Twins is an interesting direction for future work.

• Proposition 2: it indeed follows from the previous arguments, so we are happy to remove it.

We hope that with the shared response and the above, we convince you about the contributions of our work. Let us know if there is something that we should further elaborate upon.

We would like to thank you for acknowledging the generality of our approach, the usefulness of our contribution to the federated SSL field and the additional feedback. Based on your suggestions, we uploaded a revision to our manuscript that includes some very preliminary (i.e., not fully converged and with only one random seed) results on unsupervised and semi-supervised training on TinyImagenet in the Appendix. The preliminary observations are:

• In the unsupervised setting, the user-verification loss is generally beneficial for some models in the cases where we expect it to be
• In the semi-supervised setting, the user-verification loss is still beneficial for the label skew. Adding the rotation non-i.i.d.-ness in this semi-supervised setting resulted in a very difficult problem where with 10% of the labels the convergence is very slow so we were not able to get meaningful performance with any method during the first 15k rounds.

We will continue running these experiments and will update, once we are able, our submission accordingly. Once again, we thank you for the positive feedback and for helping us to further improve our work.

Following the reviewers’ suggestions, we have implemented and are currently running the following additional experiments:

• Pseudo-labelling baselines based on SemiFed [4] and CBAFed [5] for the semi-supervised setup. Baselines such as the ones mentioned by w6Qs are not compatible with our setup, due to assuming the existence of a publicly available dataset. Learning curves suggest that with SemiFed the performance is similar / a bit worse than supervised federated training on just the labelled subset of the data. However, when adding the UV loss to SemiFed as well, we once again see improvements in performance in the case of label skew, surpassing the baseline supervised training. For CBAFed, we adopted the modification with partially labeled clients (see Appendix F.5 of [5]). As we observe, the training speed of CBAFed in cross-device setting is significantly lower than of our semi-supervised Federated SimCLR.

• FeatARC [6] (without the clustering step, in order to be consistent with our setup) for the purely self-supervised scenario. Learning curves currently suggest that FeatARC (“Align Only”) in our joint-shift cross-client Cifar10 FL setting does not improve upon vanilla local SimCLR. We chose regularization strength $\lambda=0.5$ since the default suggestion of $1$ caused loss divergence.

We have annotated major changes to the first version in blue. Additionally, please find new algorithms in the appendix as well as reorganized Table 1 and 2.

[1] Federated Unsupervised Representation Learning, Zhang et al., 2020

[2] Federated Contrastive Learning for Decentralized Unlabeled Medical Images, 2021

[3] Adaptive Federated Optimization, Reddi et al., ICLR 2021

[4] SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling, Lin et al., 2021

[5] Class Balanced Adaptive Pseudo Labeling for Federated Semi-Supervised Learning, Li et al., CVPR 2023

[6] Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision, Wang, Lirui, et al. ICLR 2022