(FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning

We greatly appreciate your thoughtful comments and feedback. In our revised manuscript, we intend to address these points as follows:

Clarification of methodology

About Eq10

• At each communication round, the selected clients individually calculate their adaptive thresholds based on their own unlabeled data using Eq 9. Once the adaptive threshold is determined, each client trains its local model according to Eq 10, which incorporates pseudo-labeling and consistency regularization using strongly augmented samples.

About fixed threshold in SACR

• In Section 5.3, the ablation study reveals that applying an adaptive threshold for SACR can actually degrade performance, as seen in the CAT+SACR (All data) scenario. This occurs because applying SACR to wrongly pseudo-labeled samples leads to the generalization of erroneous samples. To mitigate this, we opted for a high fixed threshold to ensure that only high-confidence data samples, which are more likely to be correct, are utilized. Since we already employ an adaptive thresholding scheme (CAT) to address the limitations of a fixed threshold, we believe it is safe to use a fixed threshold specifically for SACR.

Motivation of LSSA method (Related to questions 1, 2)

• In a centralized SSL, methods like FlexMatch and FreeMatch introduced the use of different thresholds for each class within a dataset and showed their effectiveness. The rationale is that different classes pose varying levels of learning difficulty, so lower thresholds are assigned to more challenging classes.
• In the context of FSSL, the learning difficulty can vary across clients. This variation arises for two main reasons. First, since the server has access to only a small labeled dataset, clients whose data closely resembles the server’s data will face lower learning difficulty, while those with more distinct data will encounter higher difficulty. Second, due to the non-iid distribution of data across clients, the learning difficulty naturally differs among them.
• We propose LSAA to take account of different learning difficulties of clients, where LSAA assigns higher aggregation weights to clients with higher learning difficulty, enabling the global model to learn more effectively from these clients. Figure 1 in the global response PDF highlights the effectiveness of LSAA. Not only does it achieve higher test accuracy compared to the fixed aggregation weights (CAT + SACR), but it also demonstrates higher pseudo-label accuracy. Additionally, LSAA consistently achieves the highest correct label ratio, indicating the percentage of correct pseudo labels among all unlabeled data, throughout the experiment. After 600 training rounds, LSAA also records the lowest wrong label ratio, representing the percentage of incorrect pseudo labels among all unlabeled data. These findings suggest that LSAA effectively reduces incorrect pseudo labels while increasing correct ones, thereby mitigating confirmation bias [1].

Questions

Why does $(FL)^2$ need strongly augmented samples for SACR? What happens if we just use weakly-augmented samples instead?

• Thank you for your question regarding SACR. Building on previous work in centralized SSL and SemiFL, our method trains the client's local model using pseudo-labeling combined with consistency regularization (as detailed in Eq 10). In this approach, the pseudo-label, which is generated from the confidence of weakly augmented data, guides the training of its strongly augmented counterpart. This ensures the model produces consistent predictions across various perturbations of the same unlabeled data. Following this scheme, we aim to provide additional consistency regularization with SACR , thus we used strongly augmented samples to calculate the loss.

A minor suggestion: Figure 2 can benefit from numbers to show the flow of what happens after what. It was difficult to figure out where to start reading the diagram from.

• Thank you for your valuable suggestion to enhance the readability of our manuscript. We will make sure to update Figure 2 accordingly in our camera-ready version

Limitations of $(FL)^2$:

Thank you for the feedback on the limitation. Our limitations are as follows, and we will address them in the camera-ready version.

• CAT generates more pseudo-labels compared to fixed threshold methods, which in turn increases the time required for client training.
• In SACR, additional computation is required due to the need for inference on unlabeled samples using perturbed models.
• Our methodology relies on strong data augmentation, which may not be feasible for datasets such as sensor data.

[1] Arazo, Eric, et al. "Pseudo-labeling and confirmation bias in deep semi-supervised learning." 2020 International joint conference on neural networks (IJCNN). IEEE, 2020.

We greatly appreciate your thoughtful comments and feedback. In our revised manuscript, we intend to address these points as follows:

Effect of $(FL)^2$ on confirmation bias

Thank you for your feedback. Since the wrong pseudo-labels usually lead to confirmation bias[1], we evaluated pseudo-label accuracy, label ratio, correct label ratio, wrong label ratio and C/W ratio in addition to test accuracy. We compared $(FL)^2$ against baseline methods using the SVHN dataset with 40 labels in a balanced IID setting, as illustrated in Figure 1 of the global response PDF.

A high pseudo-label accuracy indicates that the method produces reliable pseudo labels. A high correct label ratio suggests that the method supplies the model with a higher number of accurate labels. Conversely, a low wrong label ratio indicates that the model encounters fewer incorrect labels, which is crucial for minimizing confirmation bias [1]. Lastly, a high C/W ratio signifies that the model is exposed to more correct labels than incorrect ones, further helping to reduce confirmation bias.

We observed that $(FL)^2$ consistently outperforms the baseline SemiFL across all metrics. Notably, while SemiFL generates more incorrect labels (C/W ratio < 1), $(FL)^2$ produces twice as many correct labels compared to incorrect ones. Additionally, the wrong label ratio for $(FL)^2$ is approximately 30%, significantly lower than SemiFL’s 45%. These results suggest that $(FL)^2$ effectively reduces the number of incorrect pseudo-labels while increasing the number of correct ones, thereby mitigating confirmation bias.

Furthermore, we can observe the effectiveness of each component of $(FL)^2$, which are CAT, SACR, LSAA. Using CAT and SACR alone delivers better performance compared to the baseline in terms of all metrics. If we use CAT + SACR, pseudo label accuracy increases, correct label ratio increases, and wrong label ratio decreases, which means we can reduce the confirmation bias. When LSAA is added, which is $(FL)^2$, it achieves the best performance across all metrics. This suggests that the synergistic effect of CAT, SACR, and LSAA is reducing confirmation bias effectively.

Novelty of $(FL)^2$

We appreciate your question regarding the technical novelty of $(FL)^2$. We would like to clarify $(FL)^2$’s novelty as follows:

Use of client-specific threshold (CAT) unlike FreeMatch

• $(FL)^2$ can deal with non-iid settings by calculating specific adaptive thresholds for each client, while FreeMatch calculates a single global learning status. Because clients’ data distributions vary in non-iid settings, each client’s learning status can be different, making it necessary to estimate client-specific thresholds.
• $(FL)^2$ mitigates the risk of overfitting by evaluating the learning status based on the entire dataset at a fixed point in time. Since the local model repeatedly encounters the same data across multiple local epochs, using a running batch as FreeMatch for estimating learning status might reinforce incorrect labels.

Introducing Learning Status-Aware Aggregation (LSAA)

• We newly introduced LSAA, which adjusts aggregation weights of client local models. While CAT gives lower thresholds to the clients with low learning status to learn more from abundant unlabeled data, such information is not effectively reflected in the global model because of using fixed model aggregation weights. LSAA gives more weight to the client with low learning status, thus such information can be reflected more in the global model.

Utilization of SAM under FSSL

• We developed a novel SAM objective specific to FSSL setting. We selectively apply the SAM objective to a small subset of high-confidence pseudo-labeled data, while using an adaptive threshold to incorporate a larger portion of unlabeled data.

Questions:

• Thank you for your thorough review. We have re-implemented the missing components of FedMatch and conducted an evaluation on the updated implementation. The updated performance results, presented in Table 1 of the global response PDF, demonstrate that $(FL)^2$ consistently outperforms FedMatch across all settings.
• Thank you for your thorough review of the experimental results. Our observations indicate that SemiFL is sensitive to the labeled dataset when the number of labeled data is small. For instance, in the case of SVHN with 250 labels, two out of three runs failed after training 700 rounds, resulting in a low accuracy of around 20%. However, the one successful run achieved an accuracy of 90.6%. When given 40 labels on SVHN, we could not observe the training failure and the average accuracy was 53.4%. Similarly, when testing CIFAR10 with just 10 labels, all runs of SemiFL failed completely, yielding an accuracy of only 10%. We also attempted to replicate these experiments using the official SemiFL repository with 10 labels and observed the same outcomes.
• We truly appreciate your thoughtful suggestions. We will certainly update the related work on labels-at-clients in our future manuscript.

Limitations of $(FL)^2$:

• CAT generates more pseudo-labels compared to fixed threshold methods, which in turn increases the time required for client training.
• In SACR, additional computation is required due to the need for inference on unlabeled samples using perturbed models. Our methodology relies on strong data augmentation, which may not be feasible for datasets such as sensor data.
• Our methodology relies on strong data augmentation, which may not be feasible for datasets such as sensor data.

[1] Arazo, Eric, et al. "Pseudo-labeling and confirmation bias in deep semi-supervised learning." 2020 International joint conference on neural networks (IJCNN). IEEE, 2020.

Thank you for your thoughtful review of our rebuttal. We are pleased that our responses have addressed your concerns.

We will further enhance our final manuscript based on your valuable feedback. Thank you once more for your helpful comments and feedback to improve our work.

Best, Authors.

We greatly appreciate your thoughtful comments and feedback. In our revised manuscript, we intend to address these points as follows:

Theoretical justification

Thank you for your feedback. We added more experiments to show that our method is robust in different settings. We plan to prove our methodology in terms of theoratical formulation in future works.

More experiments

Thank you for suggesting additional experiments on other public datasets and settings. In response, we conducted further experiments using the CIFAR-100, Fashion-MNIST, and AGNews datasets, and also introduced non-iid-0.1 settings for the CIFAR-10 and SVHN datasets. For detailed information and experimental results, please refer to the global response.

Questions:

Thank you for your feedback. Our accuracies are 38.9%, 81.5%, 83.2%, and 92.2% for 10, 40, 250, and 4000 labels, respectively. While the centralized FreeMatch method achieves 91.9% accuracy, our results, though still limited, represent a significant improvement. Specifically, our method outperforms the previous best-performing approach by a factor of 2.3X. This demonstrates that our approach narrows the gap between centralized and federated settings.

Limitations of $(FL)^2$:

Thank you for the feedback on the limitation. Our limitations are as follows, and we will address them in the camera-ready version.

• CAT generates more pseudo-labels compared to fixed threshold methods, which in turn increases the time required for client training.
• In SACR, additional computation is required due to the need for inference on unlabeled samples using perturbed models.
• Our methodology relies on strong data augmentation, which may not be feasible for datasets such as sensor data.

We greatly appreciate your thoughtful comments and feedback. In our revised manuscript, we intend to address these points as follows:

Novelty of $(FL)^2$

We appreciate your question regarding the difference between $(FL)^2$ and related works (FreeMatch and FlatMatch). We would like to clarify $(FL)^2$’s novelty as follows:

Different motivation with FlatMatch

• The primary goal of FlatMatch is to bridge the gap between labeled and unlabeled data by addressing the differences in their loss landscapes. However, since $(FL)^2$ is designed for the labels-at-server scenario, we cannot utilize both unlabeled and labeled data at the same moment, FlatMatch’s methodology cannot be applied.

Use of client-specific threshold (CAT) unlike FreeMatch

• $(FL)^2$ can deal with non-iid settings by calculating specific adaptive thresholds for each client, while FreeMatch calculates a single global learning status. Because clients’ data distributions vary in non-iid settings, each client’s learning status can be different, making it necessary to estimate client-specific thresholds.
• $(FL)^2$ mitigates the risk of overfitting by evaluating the learning status based on the entire dataset at a fixed point in time. Since the local model repeatedly encounters the same data across multiple local epochs, using a running batch as FreeMatch for estimating learning status might reinforce incorrect labels.

Introducing Learning Status-Aware Aggregation (LSAA)

• We newly introduced LSAA, which adjusts aggregation weights of client local models.

Utilization of SAM under FSSL

• We developed a novel SAM objective specific to FSSL setting. We selectively apply the SAM objective to a small subset of high-confidence pseudo-labeled data, while using an adaptive threshold to incorporate a larger portion of unlabeled data.

Motivation of LSSA method

• The learning difficulty can vary across clients. First, since the server has access to only a small labeled dataset, clients whose data closely resembles the server’s data will face lower learning difficulty, while those with more distinct data will encounter higher difficulty. Second, due to the non-iid distribution of data across clients.

• We propose LSAA where LSAA assigns higher aggregation weights to clients with higher learning difficulty, enabling the global model to learn more effectively from these clients. Figure 1 in the global response PDF highlights the effectiveness of LSAA. Not only does it achieve higher test accuracy compared to the fixed aggregation weights (CAT + SACR), but it also demonstrates higher pseudo-label accuracy. Additionally, LSAA consistently achieves the highest correct label ratio, indicating the percentage of correct pseudo labels among all unlabeled data, throughout the experiment. After 600 training rounds, LSAA also records the lowest wrong label ratio, representing the percentage of incorrect pseudo labels among all unlabeled data. These findings suggest that LSAA effectively reduces incorrect pseudo labels while increasing correct ones, thereby mitigating confirmation bias [1].

Effect of $(FL)^2$ on confirmation bias

Thank you for your feedback. Since the wrong pseudo-labels usually lead to confirmation bias [1], we evaluated pseudo-label accuracy, label ratio, correct label ratio, wrong label ratio and C/W ratio in addition to test accuracy. We compared $(FL)^2$ against baseline methods using the SVHN dataset with 40 labels in a balanced IID setting, as illustrated in Figure 1 of the global response PDF.

A high pseudo-label accuracy indicates that the method produces reliable pseudo labels. A high correct label ratio suggests that the method supplies the model with a higher number of accurate labels. Conversely, a low wrong label ratio indicates that the model encounters fewer incorrect labels, which is crucial for minimizing confirmation bias [1]. Lastly, a high C/W ratio signifies that the model is exposed to more correct labels than incorrect ones, further helping to reduce confirmation bias.

We observed that $(FL)^2$ consistently outperforms the baseline SemiFL across all metrics. Notably, while SemiFL generates more incorrect labels (C/W ratio < 1), $(FL)^2$ produces twice as many correct labels compared to incorrect ones. Additionally, the wrong label ratio for $(FL)^2$ is approximately 30%, significantly lower than SemiFL’s 45%. These results suggest that $(FL)^2$ effectively reduces the number of incorrect pseudo-labels while increasing the number of correct ones, thereby mitigating confirmation bias.

Furthermore, we can observe the effectiveness of each component of $(FL)^2$, which are CAT, SACR, LSAA. Using CAT and SACR alone delivers better performance compared to the baseline in terms of all metrics. If we use CAT + SACR, pseudo label accuracy increases, correct label ratio increases, and wrong label ratio decreases, which means we can reduce the confirmation bias. When LSAA is added, which is $(FL)^2$, it achieves the best performance across all metrics. This suggests that the synergistic effect of CAT, SACR, and LSAA is reducing confirmation bias effectively.

Questions:

We want to clarify terms used in the manuscript.

• CAT + SACR in Table 3 (Section 5.2): Our proposed method in Section 4.2. We apply SACR to only high-confidence samples.
• CAT + SACR (all data) in Figure 3 (Section 5.3): SACR is applied to all of the pseudo labels generated by CAT. This is an ablation study to show why SACR should not be applied to all of the pseudo labels by CAT.
• CAT + SACR (only correct pseudo-label) in Figure 3 (Section 5.3): SACR is only applied to correctly pseudo-labeled samples. This is for testing upper bound of our proposed method assuming we know the ground truth labels.

[1] Arazo, Eric, et al. "Pseudo-labeling and confirmation bias in deep semi-supervised learning." 2020 International joint conference on neural networks (IJCNN). IEEE, 2020.

Thank you for your comment regarding motivation and innovation. We would like to explain further about $(FL)^2$'s motivation and novelty.

Novelty of $(FL)^2$

We would like to further explain the novelty of $(FL)^2$ in detail.

Use of client-specific threshold (CAT) unlike FreeMatch

• $(FL)^2$ provides a precise measure of the client's learning status rather than relying on an EMA-based estimation like FreeMatch, since $(FL)^2$ directly calculates learning status using all unlabeled data.
• $(FL)^2$ is computationally efficient. Instead of continuously updating the learning status every batch like FreeMatch, $(FL)^2$ only requires one interaction of learning status calculation over unlabeled samples per communication round. Additionally, since we utilize the global pseudo-labeling scheme proposed in SemiFL, we must go through all the unlabeled data anyway to pseudo-label them. Additional computation for CAT fits naturally into the existing workflow.

Utilization of SAM under FSSL

• We developed a novel SAM objective specific to FSSL setting. We first applied the Sharpness-Aware Minimization (SAM) objective in a FSSL setting and discovered that applying it to all pseudo-labeled data degrades performance even though SAM shows strong generalization ability across different tasks. Instead, we introduced a novel approach that selectively applies the SAM objective to a small subset of high-confidence pseudo-labeled data, while using an adaptive threshold to incorporate a larger portion of unlabeled data. Our ablation study (Table 2 of the paper), which emphasizes the importance of each component, demonstrates that this combination of techniques effectively reduces confirmation bias and achieves high performance.

Motivation of LSAA

We would like to further clarify the motivation of LSAA.

• In a centralized SSL, methods like FlexMatch and FreeMatch introduced the use of different thresholds for each class within a dataset and showed their effectiveness. The rationale is that different classes pose varying levels of learning difficulty, so lower thresholds are assigned to more challenging classes—those with a lower learning status—to facilitate more effective learning from these harder classes.
• In the context of FSSL, the learning difficulty can vary across clients. This variation arises for two main reasons. First, since the server has access to only a small labeled dataset, clients whose data closely resembles the server’s data will face lower learning difficulty, while those with more distinct data will encounter higher difficulty. Second, due to the non-iid distribution of data across clients, the learning difficulty naturally differs among them.
• We propose LSAA to take account of different learning difficulties of clients, where LSAA - assigns higher aggregation weights to clients with higher learning difficulty, enabling the global model to learn more effectively from these clients. In contrast, previous FSSL approaches did not account for these variations in learning difficulty and instead relied on fixed aggregation weights.

Thank you for your response to our rebuttal. We sincerely appreciate your effort and time in going through our rebuttals and raising your concern. We want to be more explicit in addressing your remaining concerns as follows:

Regarding the confirmation bias

As highlighted in previous work on Semi-Supervised Learning and others that mentioned confirmation bias [2, 3, 4, 5, 6], one approach to reducing confirmation bias is to filter out noisy pseudo-labels and retain only the high-quality ones. For instance, Fixmatch [2] claimed that it effectively reduced confirmation bias by generating higher-quality pseudo-labels. Similarly, Mean Teacher [3] suggested that improving target quality can help mitigate confirmation bias. Additionally, the Softmatch [4] paper highlighted that incorrect pseudo-labels often contribute to the occurrence of confirmation bias. Based on these previous works, we believe that demonstrating a higher correct label ratio effectively shows how well our method filters out incorrect pseudo-labeled samples. Consequently, since $(FL)^2$ has a significantly lower rate of incorrect pseudo-labels, the influence of these on the unsupervised loss is much less pronounced compared to SemiFL. This suggests that the training signals propagated by these pseudo-labels are more reliable in $(FL)^2$ due to lower contribution of noisy pseudo-labels - our unsupervised loss is dominated mainly by correct pseudo-labeled samples compared to SemiFL.

Nevertheless, to further evaluate this, we designed and conducted an experiment to compare the level of overfitting of $(FL)^2$ and SemiFL to wrongly pseudo-labeled data in a scenario with extremely scarce labeled data. We measured this by inspecting Rsoft, RLoss, and RAccuracy. Rsoft is the average confidence of the wrongly pseudo-labeled data, similar metric measured in [1]. RLoss and RAccuracy means the training loss and accuracy on the wrongly pseudo-labeled data, respectively. High Rsoft and RAccuracy, and low Rloss indicate that the model is overfitting to the incorrectly pseudo-labeled data. The experiment used the CIFAR-10 dataset under a balanced IID setting with only 10 labeled samples at the server, keeping all hyperparameters the same as in the main experiments, except for the number of communication rounds, which we set to 500.

The results clearly show that SemiFL overfits significantly to the wrongly pseudo-labeled data compared to $(FL)^2$. This demonstrates $(FL)^2$’s robustness against incorrect pseudo-labels, particularly in scenarios with extremely scarce labeled data at the server, outperforming SemiFL. Thus, we conclude that $(FL)^2$ is more effective in reducing confirmation bias in low-label scenarios.

Thank you again for your insightful suggestion to explore overfitting with incorrectly pseudo-labeled data. We will ensure that the results of this experiment are included in the camera-ready version.

• [1] Arazo, Eric, et al. "Pseudo-labeling and confirmation bias in deep semi-supervised learning." 2020 International joint conference on neural networks (IJCNN). IEEE, 2020.
• [2] Sohn, Kihyuk, et al. "Fixmatch: Simplifying semi-supervised learning with consistency and confidence." Advances in neural information processing systems 33 (2020): 596-608.
• [3] Tarvainen, Antti, and Harri Valpola. "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results." Advances in neural information processing systems 30 (2017).
• [4] Chen, Hao, et al. "SoftMatch: Addressing the Quantity-Quality Tradeoff in Semi-supervised Learning." The Eleventh International Conference on Learning Representations.
• [5] Nassar, Islam, et al. "All labels are not created equal: Enhancing semi-supervision via label grouping and co-training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
• [6] Zhang, Bowen, et al. "Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling." Advances in Neural Information Processing Systems 34 (2021): 18408-18419.

Thank you again for your critical and constructive feedbacks to our papers and rebuttals. If there is still anything that is not clear about our justification regarding the confirmation bias and the novelty/motivation of our work, we would be happy to provide more clarification.

If you have any further concerns or questions, please do not hesitate to let us know as it will be extremely important for our revised manuscript.

Thank you very much!