Encoding Expert Knowledge into Federated Learning using Weak Supervision

The title is clear and gives a good indication of the paper's main focus. The abstract provides a concise overview of the problem, the proposed solution, and the results. The introduction sets the context well, highlighting the importance of learning from on-device data and the challenges associated with it. The motivating example of arrhythmia detection provides a real-world context, making it relatable for readers. This paper introduces a new method, WSHFL, that bridges the gap between federated learning and programmatic weak supervision, addressing a challenge in the field. Addressing the issue of on-device data annotation is timely and relevant, given the increasing importance of privacy and on-device computations.

This paper proposed a novel paradigm, WSHFL, for integrating programmatic weak supervision with on-device federated learning. The key ideas are composed of 2 components: automatic labeling function (heuristics) generations and federated learning with WeaSEL (Weakly Supervised End-to-end Learning). The authors conduct experiments to show that both components work across three benchmark tasks.

The paper discusses the challenges of leveraging on-device data for training intelligent mobile applications. The data is distributed on client devices and it is sensitive data, making it difficult to obtain expert annotations for traditional supervised machine learning. Current federated learning techniques typically use unsupervised approaches and cannot capture expert knowledge through data annotations. To address this issue, the paper introduces a method called Weak Supervision Heuristics for Federated Learning (WSHFL). WSHFL utilizes labeling functions, which are heuristic rules, to annotate on-device data in cross-device federated settings. The paper presents experimental results across two data modalities, text and time-series, demonstrating that WSHFL achieves competitive performance compared to fully supervised methods without the need for direct data annotations.

Dear Reviewers and Area Chair,

We sincerely appreciate your time and for offering valuable feedback.

We are glad that all reviewers found our work to be well motivated, and grounded in real-world impact through our experiments on ECG data. We are also happy that the reviewers appreciated our experiments on time-series data, which has received very little attention in federated learning (reviewer qXVo). Finally, we thank Reviewer W4Yv for pointing out how the needs for on-device expert annotation are not met by existing work.

Below we would like to highlight three general areas of improvement and discussion that all the reviewers have pointed out:

Clarity and organization

Reviewers mKax and W4Yv pointed out areas of improvement in Section 3 and Figure 1. We are actively working on improving these elements to make them easier to follow. As of now, we have included more detail in Figure 1 (reviewer W4Yv), and we have consistently differentiated between labels $y$ and expert labels $u$ (reviewer mKax) in Section 3.

Privacy leakage from unigram labeling functions

We agree with Reviewers qXVo and mKax that some families of LFs can leak private information, and already acknowledge these risks in the Societal Impacts section of our work.

Our methodology already follows the privacy principle of data minimization [16, 18] by choosing unigrams based on TF-IDF, therefore never exchanging infrequent words which can be indicative of individual clients. The risk of private information leakage could be further minimized by showing clusters of unigrams instead of individual keywords as text labeling functions, or we could derive unigrams from a vocabulary that is shared by the server to the clients. To further reduce privacy risks, we can leverage techniques such as SynTF [17] to create private term frequency vectors. Finally, under the appropriate threat model, we can also envision a mechanism in which the expert discards suspicious LFs, as the chosen LFs are inspected by the expert before they are sent to other clients.

We don’t make any claims about ensuring privacy in our work, as we don’t formally work with formalisms such as differential privacy. However, depending on the problem, it is possible to use simple interpretable classifiers as LFs, and use differential privacy in this setting.

We will add these discussions to the appendix in our future revision.

Novelty

We would like to highlight the following novel contributions of our work to federated learning:

Novel problem setting - supervising on-device data

To the best of our knowledge, there is no work which considers the problem of providing expert supervision to on-device data, when clients either do not have time or the expertise to label their own data. One solution would be to utilize techniques in federated semi- or self-supervised learning (FSSL), but these techniques are not directly applicable to our setting for the following reasons:

1. Studies considering the labels-at-client setting cannot be used because they assume that clients have partially labeled data. In the practical setting that we consider, clients either do not have the time or expertise to label their own data [2, 3, 4, 5, 8, 10, 14, 15].
2. Studies which consider the labels-at-server setting are not directly usable since they assume that the server has a limited amount of labeled data [6, 7, 8, 9, 10, 12, 13].
3. Finally, a few studies consider the self-supervised federated learning setting, where the goal is to learn unsupervised representations. These studies still require labeled data for classification [1, 11].
4. All of these studies are only evaluated on image data, and most of them rely on some form of weak or strong data augmentation techniques, which cannot be naively extended to text or time-series data.

Weak supervision in a federated setting

While programmatic weak supervision has been studied in a centralized setting, extending it to the federated setting is not straightforward. We considered various nuances of this problem, such as how to generate candidate LFs at scale in a distributed manner, and how to properly account for the heterogeneity of the federated data when aggregating estimates $\hat{u}_j$. In our “Conclusions” section (Section 7), we also note several under-explored directions in this space, e.g., the investigation of privacy preserving LFs.

Modeling time-series data

Time-series is a prevalent data modality in many domains such as healthcare, yet very few studies have considered the problem of modeling time-series in weakly supervised and federated settings.

We are making several changes to our manuscript. All of these changes are in dark blue.

Dear Reviewers and Area Chair,

We would like to express our sincere gratitude for your time and feedback. We have made several changes over the past few days to improve the clarity and experimentation as pointed out by all the reviewers.

Below we provide a summary of changes. All of these changes are in dark blue.

Comparison with federated semi-supervised learning

Based on the suggestions by Reviewer W4Yv, we compared our method (WSHFL) with two state-of-the-art federated semi-supervised learning techniques with labels at the server: SemiFL [1] and FRGD [2], on the IMDb dataset. We found that WSHFL outperforms the compared semi-supervised learning methods across 5 independent runs on the IMDb dataset (Fig. 16 on Page 26 of the Appendix). The detailed experiment setup is described in Appendix A.7.

We also provided a deeper discussion of federated semi- and self-supervised learning methods, and their limitations in the context of the problem that we are trying to solve. You can find this in Appendix A.7. In addition, we would like to note that ours are the first set of reproducible federated experiments on non-image data in settings where the clients hold unlabeled data.

Clarity and organization

In response to Reviewer mKax’s questions and suggestions, we have made changes to Section 3. We also added a table of symbols and equations in Appendix A.10 to serve as a convenient point of reference for readers, enhancing the overall clarity of our manuscript.

In addition, we added annotations to Figure 1 in response to Reviewer W4Yv’s comments. We have supplemented a copy of Figure 1 (Fig. 9 in Appendix A.1, Page 16) with a comprehensive caption to provide a detailed explanation of WSHFL’s approach to supervising on-device data.

Reproducible science

We thank Reviewer qXVo for recognizing the contributions of our work and offering valuable suggestions to enhance its impact, particularly by advocating for the release of our code as open source. In response, we promptly made our code accessible at https://anonymous.4open.science/r/wshfl_pipeline-A13C/. We anticipate that sharing our code will encourage further research on the significant challenge of learning from unlabeled on-device data in non-image data modalities.

Privacy leakage from unigram labeling functions

We improved the clarity of the Societal Impacts section of our manuscript (under Section 7) to reflect the deeper discussions we have had with reviewers mKax and qXVo on the risk of potential information leakage from some families of LFs.

We hope that our responses and efforts have adequately addressed the reviewers’ concerns and questions. Please let us know if you have any remaining questions or concerns. We are looking forward to your responses.

References

[1] Diao, Enmao, Jie Ding, and Vahid Tarokh. "SemiFL: Semi-supervised federated learning for unlabeled clients with alternate training." Advances in Neural Information Processing Systems 35 (2022): 17871-17884.

[2] Zhang, Zhengming, et al. "Improving semi-supervised federated learning by reducing the gradient diversity of models." 2021 IEEE International Conference on Big Data (Big Data). IEEE, 2021.

After careful consideration of the reviews and the authors' responses, it is clear that while the authors have made efforts to address concerns around clarity, organization, and privacy risks in their WSHFL methodology, the paper still lacks the necessary impact required for acceptance. The reviewers brought up critical issues regarding the novelty of the key ideas, with the adaptation of programmatic weak supervision to a federated setting being more incremental than revolutionary. Moreover, concerns about potential privacy leakage through the labeling functions, despite efforts to address them, remain a significant concern in the context of the federated learning paradigm, which prioritizes data privacy.

The experiments, although conducted across different datasets, fail to compare against a broader range of state-of-the-art federated semi-supervised learning techniques, which leaves the actual impact and performance gain of WSHFL in the field unclear. The readability of the manuscript has improved, according to the reviewers, but there is room for enhancement, particularly in the explanations around the generation of labeling functions and the role of the expert in LF selection.

Overall, despite the authors’ response to the reviews, the paper has not met the threshold for acceptance owing to lingering concerns about originality and depth. It is recommended that the authors further refine their contribution, strengthen the privacy-preserving aspects of their methodology, and provide more extensive comparisons with state-of-the-art methods to establish the significance of their work in future submissions.

Reject