BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach

Thanks a lot for the reviewing.

Q1.FedDB[3] also employs Bayesian in FSSL. FedDB explicitly and mitigates class prior bias from imbalanced data. BSemiFL does not address internal class bias. Additionally, BSemiFL’s analysis is confined to its aggregation strategy and lacks a comparative evaluation.

Although [3] also employs Bayesian methods, the scenarios in which the two methods are applicable are entirely different. The scenario considered by [3] is "labels at client," which may not be suitable for our "labels at server" scenario. The core step of [3] involves using a single model to annotate local data and suppress the prediction probabilities of the majority class to correct annotation errors of that single model.

• However, this method requires calculating the prior probability of the majority class, which relies heavily on labeled data from the "labels at client" setting. When there is no labeled data locally, an initial miscalculation of the prior probability will exacerbate errors during the suppression process, and there would be no way to correct these priors based on labeled data later on. In contrast, our proposed method does not depend on local labeled datasets but instead leverages a Bayesian weighting approach between the local and global models to jointly correct prediction probabilities.
• The fundamental idea of [3] is a weighted inference method for addressing class imbalance, which has inherent limitations. For example, in a three-class classification task, assuming class C1 is the majority class and classes C2 and C3 are minority classes with equal sample sizes, the trained model might tend to classify C2 and C3 as C1. Although [3] penalizes the majority class, due to insufficient training on minority classes, it still cannot distinguish between C2 and C3, leading to misclassification. Therefore, widely used methods in the field of class imbalance are oversampling rather than weighting. Our method adaptively assigns the labeling of the majority class to the local model and the labeling of minority classes to the global model, thereby avoiding such errors.

Cifar10

Thus, [3] and our method apply to different scenarios. Indeed, combining the error correction of a single model from [3] with the correction of integrated annotations from two models in our method could potentially lead to further improvements and broader applicability across various scenarios.

Q2. Theories have limitations: the paper assumes a known local distribution. using pseudo-labels to estimate the distribution has noise.

Although the pseudo-label estimation distribution contains some noise, as training progresses, the model's accuracy improves, and this estimation error gradually decreases. Moreover, while the theorem has certain limitations, it still demonstrates the effectiveness of the method to a certain extent. It shows that under approximate conditions, the method outperforms using a single model or a simple average of two models.

Q3：Novelty is somewhat limited. The limitations of data heterogeneity have been recognized in FL. Although this work seeks to extend it to FSSL, the characteristics are not sufficiently reflected.

This paper investigates multiple characteristics of FSSL, including but not limited to:

• The first systematic exploration of the advantages and disadvantages of local-only and global-only labeling methods, as well as the underlying reasons, which has not been addressed in traditional FL.
• The first proposal of an adaptive weighting integration of global and local models tailored to the labeling needs in FSSL, a unique feature in FSSL.
• Our theoretical analysis specifically targets the accuracy of labeling in FSSL. Additionally, we adopt an approximation of pseudo-label distributions and do not require prior knowledge of the local data distribution.

Q4. The paper ignores the cumulative amplification of pseudo-label errors across multiple rounds. The theories focus on a single re-labeling step, overlooking the long-term accumulation of noise.

Our method involves labeling in conjunction with the global model, which can be refined using the global labeled dataset, thereby reducing such noise. This is also reflected in SemiFL, where using only the global model can gradually lead to better performance without amplifying cumulative errors.

Q5.Many big models were used in FL. Their knowledge helps alleviate the imbalance. How to evaluate improvements of the method compared to big models in pseudo-label generation? Can theories incorporate big models?

Our method can be combined with foundational models to a certain extent. For example, we can treat the foundational model as the global model and integrate it with the local model to label local data. Considering the differences between foundational models and global models, new theoretical and experimental analyses may be required.

Thank you very much for your reviewing and valuable suggestions.

Q1：The details of the designed ensemble strategies in Figure 5(a) are not specified. It seems that these methods are designed by this paper itself. However, the details are not clear. For example, what’s the ,meaning of the majority voting? In my opinion, there are only two models, which has the concept of “majority”.

We apologize for the lack of clarity in some of our descriptions:

In Figure 5(a):

• Simple average refers to assigning equal weights (0.5) to the results produced by the global model and the local model, and then computing a weighted sum.
• Random refers to randomly assigning two weights to the results produced by the two models and then computing a weighted sum.
• Majority Vote means that for a given unlabeled data point, we only assign a pseudo-label if both the global model and the local model produce the same pseudo-label for that data point.

Q2：I'm curious as to why your method performs worse than Orchestra when it comes to Cifar100 (100, 2)?

Thank you for your feedback.

First, our method achieves a performance of 39.09±3.12 on CIFAR100 (100, 2), while Orchestra achieves 39.93±2.17. These results are based on multiple experiments, so the performance of our method is not inferior to that of Orchestra. Regarding the slightly lower average value compared to Orchestra, we believe this is because Orchestra is a federated self-supervised learning method. In this specific scenario, its better performance is mainly due to the fact that the client data distribution is highly concentrated in a few categories. This local consistency makes it easier for local clustering to capture clear category patterns.

In contrast, our method does not rely on clustering but instead leverages the collaboration between the global model and the local model to assign pseudo-labels. Given the higher complexity of CIFAR100 images compared to other datasets, this leads to slightly higher fluctuations in our method compared to Orchestra.

Thank you very much for your reviewing and valuable suggestions.

Q1：more recent advanced SSL methods [1] can considered.
[1] BEM: Balanced and Entropy-Based Mix for Long-Tailed Semi-Supervised Learning. CVPR 2024.

BEM [1] mainly proposes a novel hybrid method for rebalancing the class distribution in terms of data quantity and uncertainty. This method can be integrated into our proposed approach.

Q2：The proposed method requires more interaction items except of the models between the clients and the server. The risk of the privacy leakage is not discussed.

Although our method requires more interaction terms, it does not actually increase the risk of privacy leakage. Compared to traditional FL methods, as shown in line 681, our method only needs to additionally transmit the K-dimensional probability distribution vector $Q_s=[Q_s^1,...,Q_s^K]$ of the server-side public dataset categories on the global model. In the SSFL scenario, the data on the server side is generally non-private. Therefore, in practice, our method does not result in additional client privacy leakage compared to traditional FL methods.

Q3：There exists a SSFL method using Bayesian approach [1]. What’s the difference between your proposed method their method?
[1] Estimating before Debiasing: A Bayesian Approach to Detaching Prior Bias in Federated Semi-Supervised Learning. IJCAI 2024.

First, the scenario of our method differs from that of the referenced article:

In our scenario, we assume that clients only have unlabeled data, while the server has a small amount of labeled data. In contrast, in [1], the scenario assumes that clients possess both supervised and unsupervised data, while the server does not have any data.

Second, regarding the application of Bayes' theorem:

Our method leverages Bayes' theorem to assign pseudo-labels to local data based on the knowledge of both the local model and the global model. Specifically: We first fine-tune the global model on the server using the labeled data available on the server:
$Q_s^k = \sum_{i=1}^{N_s} f_{\hat{p}}(k|x_i, w_t)$

and then compute the probability distribution of the server-side data on the global model. This parameter, along with the global model, is distributed to the clients. Subsequently, we calculate:

For each data point in the local client, we compute:

Similarly, in the local model, we compute the empirical distribution (using pseudo-labels from the previous round) and the probability distribution of the local data:

$Q_m^k = \sum_{i=1}^{S_m} f_{\hat{p}_m}(k|x_i,w_t^m)$.

Then, we calculate$A=\sum_{k=1}^K f_{\hat{p}_m}(k|x_i^m, w_t^m)\hat{p}_m(k)$,

and $B=(f_{\hat{p}_m}(k|x_i^m, w_t^m)+Q_m^k)$ to obtain $\hat{p}_m(x_i^m)=A/B$

Next, for each local data point, we obtain the confidence levels of the global model and the local model as:

Finally, by denoting $C=f_{\hat{p}}(x_i^m, w_t)$, and $D=f_{\hat{p}_m}(x_i^m, w_t^m)$, we derive the final probability distribution:

In contrast, in FedDB, the use of Bayes' theorem occurs during local training, where the knowledge of the local model is utilized to optimize pseudo-labeling via APP-U (the Average Prediction Probability of Unlabeled Data), thereby reducing label prior bias. Specifically, it is argued that the imbalance in the local dataset introduces bias when the local model assigns pseudo-labels to unlabeled data, favoring classes with more data. To address this issue, the authors propose applying Bayes' theorem to rewrite:

as:

To mitigate the imbalance issue, a regularization-like term is introduced, yielding:

Thus, our method primarily uses Bayes' formula to combine the knowledge of the global model and the local model through pseudo-labeling, whereas the work in [1] focuses on addressing data imbalance issues.

Thank you very much for your reviewing and valuable suggestions.

Q1: How to get the p^(xi,k) in 5.1. It is better to use a consistent notation.

$p^(x_i,k)=p^(x_i,y_i=k)$ represents the joint probability distribution of the sample $x_i$ and $y_i$, i.e., the probability that the sample takes the value $x_i$ while the label simultaneously takes the value $k$. In fact, our notation is consistent throughout the paper. However, due to the relatively large number of variable types involved, it may give an impression of complexity. We will further optimize the explanation and representation of the variable notations moving forward.

Q2：p and p^ in eq.(20)? It is unclear how to get (20) from Hoeffding.

Eq.(20) involves derivation steps that are commonly used in empirical risk analysis, so we omitted these steps. Since $L_p(f)$ is the expectation of $L_{\hat{p}}(f)$, by applying Hoeffding's inequality, we can derive:

Here, $\epsilon > 0$ is an arbitrarily small positive number. Let:

Solving for $\epsilon$:

This implies that with a probability of at least $1 - \delta$, the following inequality holds:

Decomposing the above absolute value inequality yields formula (20). We will include these in the revised version.

Q3：Experiments of "Impact of Labeled Dataset Size" lack the specific heterogeneity, the number of clients, and other settings.

The total number of clients is 100, with 10% activated in each round. The heterogeneity among clients is set to a shard distribution with $k=2$. The threshold is 0.7.

Q4:α visualization.

We add the following experiment. When each client is selected for training, we record the average value of α for all data points in its local dataset. We here present the corresponding values for five randomly selected clients in the table below. As can be seen, in the initial rounds, the local model achieves higher accuracy on the local data compared to the global model, resulting in a relatively smaller weight α for the global model. In later rounds, the values of α gradually stabilize around 0.5, which aligns with the gradual improvement in the accuracy of the global model.

Q5:Meaning of “global model can progressively...”. The difference between a local model with a high capability and a global model with a progressively higher capability.

• This statement is relative to the local model. In Fig.2b, we observed that when labeling data using only the local model, some labeling errors occur. When these labeled data are used to train the local model, the errors are carried forward, and during the next round of labeling, the same errors persist, making it impossible for the model to self-correct. On the other hand, the global model, which incorporates knowledge from other clients, can correct these errors.

• Since the local model converges quickly while the global model converges more slowly, there is a significant difference in accuracy between the two in the early stages of training, but they gradually converge later on. However, due to the impact of heterogeneity, the error of the global model on the local dataset remains consistently higher than that of the local model.

Q6：Motivation does not clearly explain which model to train and what data are used. How are the global and local model obtained in Fig.2a?

All motivation experiments used the Wide ResNet28*2 model. As indicated in lines 151 and 208, we used CIFAR10. All experiments were conducted to train the global model.

• In all experiments, the global model is obtained on the server using FedAvg, while the local model is obtained after further local training of the global model.

• In all figures, the global model is tested using the combined local data from all clients (i.e., the global data), whereas the local model is tested using the local data corresponding to its respective client. This is explained in the captions.

Thus:

• In Fig.2a, as in lines 150–154, the test data for the global model is the global data, while the test data for the local model is the local data corresponding to its respective client.
• In Fig.2b, the global model is tested using two different annotation methods, and the test data used is the global data.
• In Fig.2c, the local model is tested using its corresponding local data.

Q7: Lacks in some recent citations in “labels at client” [1,2].

Thank you for the reminder. We will include them.