On the Importance of Language-driven Representation Learning for Heterogeneous Federated Learning

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

Q1: Comparison with prototype FL.

We have compared with FedProto [2] in our experiments, and the results are presented in Tables 1 and 2. FedPCL [1] uses the pre-trained image backbone. For a fair comparison, we conducted additional experiments using the pre-trained image backbone (AlexNet) on the Office-Caltech-10. Specifically, we initialized the image backbone with pre-trained weights from ImageNet and kept all other settings unchanged. As we can see, FedPCL performs significantly worse than our method, demonstrating the superiority of FedGLCL over prototype-based FL methods. The results have been included in the revision (Appendix D.9 Table 18)

Q2: More tasks.

As stated in Line 052, this work primarily uses visual recognition as an example, but it can be extended to more visual understanding tasks, such as image segmentation. Inspired by [3], we can extend the image-text alignment to pixel-text alignment for image segmentation. We conducted experiments on ProstateMRI [4], a widely used federated medical image segmentation benchmark. This benchmark has six clients including BIDMC, HK, I2CVB, BMC, RUNMC, and UCL, each from a different domain. We used U-Net as the segmentation network and aligned the output of the final layer with the text features. Since this benchmark involves a binary segmentation task, we use "foreground" and "background" as the class names. The communication rounds are set to 200, with local rounds set to 1. The results are presented below. It can be observed that FedGLCL achieves certain improvements compared to FedAvg, indicating that our method has the potential to be applied to other tasks to address non-IID data issues.

Q3: Curious about the peformance of Glove.

Thanks for your suggestion. We conducted additional experiments using GloVe to encode global text embeddings. The results, presented below, show that GloVe outperforms FedAvg, but achieves lower performance than Bert and CLIP. This is because Bert and CLIP employ deep neural networks to learn data representations through supervised learning on large-scale datasets, allowing them to capture semantic information more effectively. The results have been included in the revision (Appendix D.4 Table 13)

References

[1] Tan, Yue, et al. "Federated learning from pre-trained models: A contrastive learning approach." Advances in neural information processing systems 35 (2022): 19332-19344.

[2] Tan, Yue, et al. "Fedproto: Federated prototype learning across heterogeneous clients." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 8. 2022.

[3] Li, Boyi, et al. "Language-driven Semantic Segmentation." International Conference on Learning Representations. 2022.

[4] Liu, Quande, et al. "MS-Net: multi-site network for improving prostate segmentation with heterogeneous MRI data." IEEE transactions on medical imaging 39.9 (2020): 2713-2724.

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

Q1: Which pretrained vision language models are used in the experiments?

In our default setting (Lines 363-372), we used a language model, Bert-base, as the text encoder rather than a vision-language model. Besides, the image backbone is trained from scratch without loading any pre-trained models.

Q2: Comparison on different text encoders.

Thanks for your comments. We have already presented the results of this experiment in Appendix D.4 (Table 13). As shown, our method still outperformed the best baseline after using different text encoders. This demonstrates the robustness of FedGLCL to various text encoders.

Q3: Are the datasets used in the experiments part of the pre-training of VLMs?

The datasets used in the experiments are NOT used for pre-training. The used text encoder, Bert-base, is trained on the pure language dataset and does NOT contain any knowledge from four used datasets (image datasets). Besides, the image backbone is trained from scratch without loading any pre-trained models. This indicates that the improvement in our method stems not from the pre-training stage, but from our novel learning mechanism.

Q5: what is "the philosophy of language-driven representation learning"?

Sorry for the confusion. It refers to the conceptual approach or theoretical framework that emphasizes the use of language as supervision in learning meaningful, transferable representations of data. ''the philosophy of'' may cause confusion, so we have removed it in the revision (Line 160).

References

[1] Jia, Chao, et al. "Scaling up visual and vision-language representation learning with noisy text supervision." International conference on machine learning. PMLR, 2021.

[2] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

[3] Li, Qinbin, Bingsheng He, and Dawn Song. "Model-contrastive federated learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[4] https://www.kaggle.com/c/tiny-imagenet/overview.

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

Q1: Contribution to the FL community.

We clarify the contribution of our work as follows.

(1) First attempt in FL: To the best of our knowledge, FedGLCL is the first attempt to introduce language-driven representation learning into FL. Although language-driven representation learning is used in centralized settings [1, 2], we address an entirely different problem i.e., non-IID data in the FL setting.

(2) Label-driven vs. language-driven: Existing FL methods typically adopt the label-driven training paradigm, leading to significant drift in local models when handling non-IID data. In this work, we found that using a global text embedding for supervision can effectively mitigate feature differences among local models and prevent overfitting, thus alleviating the non-IID issues in FL. This provides more insights and opens up a new prospect for addressing this challenge.

(3) Theoretical and empirical analyses: Furthermore, we offer theoretical and empirical analyses to gain a deeper understanding of our approach. Extensive experiments show that our method consistently outperforms various state-of-the-art approaches across two different non-IID scenarios, demonstrating its effectiveness.

Q2: Add larger-scale datasets.

The four datasets used in the submitted paper are popular benchmarks in heterogeneous FL, which ensures a fair comparison with previous methods. To the best of our knowledge, in heterogeneous FL, no results have been reported on ImageNet, making it difficult to compare with competing methods. Following [3], we conduct experiments using another larger-scale dataset, Tiny-ImageNet [4], which has a similar data distribution to ImageNet. Detailed information about Tiny-ImageNet is provided below, which has been included in Table 5.

We use the same experimental setup as for CIFAR-100, and $\beta$ is set to 0.5. The results are presented below. As shown, FedGLCL consistently achieves the best performance compared to all competing methods, further demonstrating the effectiveness of our method. The results have been included in the revision (Appendix D.7 Table 16).

Q3: Emphasize the type of FedGLCL.

Our work is not a type of pre-training. Instead, it leverages textual information to conduct supervised learning on specific visual recognition tasks. Therefore, it is a type of image-text learning. To distinguish it from text-only or image-only, we emphasize it in the abstract accordingly:

Line 017-018: "a novel language-driven FL framework that" to "a novel language-driven FL framework for image-text learning that"

Q4: Difference with Lu et al., 2023, Guo et al., 2023, Shi et al., 2023.

FedGLCL is fundamentally different from them, we explain the differences as follows.

(1) They directly apply the well-trained CLIP model (including both the image and text encoders) as the core components for their tasks. Therefore, their methods are highly dependent on the CLIP model. Instead, FedGLCL does not require any modules from CLIP as it can use arbitrary language models as the global text encoder.

(2) They use labels for supervision, which remains the traditional label-driven FL approach. FedGLCL is a language-driven FL method that aligns the local image feature space with the global text feature space through global text supervision.

(3) Since they use image encoders of CLIP, they are easy to incur domain shift in visual tasks. In contrast, FedGLCL is robust to domain shift as it learns the visual encoder from scratch. For instance, the word "cat" holds the same semantic in both natural images and cartoon image tasks. However, for image backbones, the natural images and the cartoon images represent two entirely different data distributions.

Q4: Add experiments with pre-trained image models.

Thanks for your suggestion. We conducted additional experiments using pre-trained image models on the Office-Caltech-10. Specifically, we initialized the image backbone (AlexNet) with pre-trained weights from ImageNet and kept all other settings unchanged. The results show that incorporating the pre-trained image backbone improved the performance of all methods, and our approach consistently achieved the best results. This confirms that under fair comparisons (where all methods either utilize the pre-trained image backbone or do not), FedGLCL significantly outperforms other methods, further demonstrating its effectiveness and superiority. The results have been included in the revision (Appendix D.9 Table 18).

References

[1] Fang, Alex, et al. "Data determines distributional robustness in contrastive language image pre-training (clip)." International Conference on Machine Learning. PMLR, 2022.

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

Q1: Discussion about the impact of language supervision.

Sorry for the confusion. We clarify it as follows and make it clear in the revision (Line 1069-1079).

(1) Impact of language supervision: Non-IID data leads to significant drift in local models. The key idea of FedGLCL is to align the local image feature space with the global text feature space, which benefits learning a consistent feature representation and prevents overfitting to the majority classes across different local models.

(2) Explanation of experimental results: In FedGLCL, local image encoders are trained from scratch without any prior knowledge. Therefore, the critical factor is the consistency of text embeddings between training and testing. This ensures that the encoded text embeddings can effectively guide the learning process, regardless of whether the class names perfectly correspond to the images. While the proxy class names are not the real classes of images, they are still in the domain covered by the pre-trained text encoder. Consequently, the global text embeddings still provide meaningful semantic supervision for training.

(3) Success of linguistic knowledge: This result highlights that the success of language supervision is not dependent on the precise semantic relationships between class names and images but rather on the overall structure and knowledge encoded in the global text embeddings. This not only offers effective supervision for training local image encoders but, more importantly, ensures the alignment of all local image feature spaces with the global text feature space for mitigating drift in local models.

Q2: Measuring the impact of FedGLCL-style training in centralized setting.

(1) We present the results of FedAvg and FedGLCL on Office-Caltech-10 under both centralized and federated settings. The results show that FedGLCL achieves significantly greater performance improvements over FedAvg in the federated setting compared to the centralized setting. This demonstrates that the improvements of FedGLCL primarily arise from its ability to address the non-IID problem rather than solely from the contrastive learning approach.

(2) This paper investigates the issue of non-IID data across different clients in FL, which leads to significant drift in local models. In contrast, reference [1] explores the distributional differences between pre-training data and downstream task data. They are entirely two different problems. Therefore, our results cannot be directly compared or discussed in relation to the results of reference [1].

Q3: Standard errors for results.

Thanks for recognizing the large improvement of our method. We conduct two additional trials using different random seeds and report the mean and standard deviation (std) across all three trials (mean $\pm$ std). Furthermore, to evaluate the statistical significance of the performance improvements, we perform a paired t-test between the baseline and our method, reporting the corresponding p-value. The above results of FedAvg, FedGLCL, and two best baselines, i.e., FedConcat and FedFA, are presented below. It can be observed that the p-values for all baselines are less than 0.05, indicating the statistical significance of the performance improvements achieved by our method. These results have been included in the revision (Appendix D.8 Table 17).