pFedClub: Controllable Heterogeneous Model Aggregation for Personalized Federated Learning

>>> W1

Thank you for the reviewer's valuable feedback. We addressed the computational cost comparison in Section 4.5, using computation time as a metric against pFedHR. The results, presented in Figure 4, show that pFedClub generally requires less time than pFedHR and offers more consistent performance. Additionally, we detailed the system running time comparisons with the baseline in Appendix E. Our findings demonstrate that to achieve the target accuracy, our proposed approach is more time-efficient than pFedHR, further underscoring its efficiency.

>>> W2

Thank you for the question. The anchor block selection process is detailed from line 162 to line 169. For each block, we implement a random substitution, with the first substituted block designated as the anchor. This randomness enhances the diversity of candidate model generation. For a clearer explanation of this process, we have included a detailed algorithm in Appendix A, from lines 3 to 7.

>>> W3

Thank you for the reviewer's question. On the server side, public data is used solely to assist in (1) calculating the representations of the blocks for function identification and clustering, and (2) conducting stitching layer tuning. These operations utilize the data as foundational information without exploiting any specific attributes of the data itself. Consequently, our proposed approach is compatible with both labeled and unlabeled public data.

>>> W1

Thank you for the reviewer’s question. In the main paper, we set K to 4, following the methodology outlined in [1]. To further explore how the value of K affects our results, we conducted a hyperparameter study on K. The results of this study are presented in Table 5, with detailed analysis provided in Appendix G due to page constraints. We observed that performance slightly improves with a larger value of K within a certain range. This improvement may be due to the fact that a larger K can differentiate functions more specifically.

[1] Wang et al. "Towards personalized federated learning via heterogeneous model reassembly." Advances in Neural Information Processing Systems 36 (2024).

>>> W2

Thanks for the reviewer’s question. Our controllability is demonstrated in sec 3.2.3 controllable block-wise substitution. Furthermore, we provided the controllability analysis in section 4.4. We define a hyperparameter $\phi$ to quantify the controllability of the personalized models generated by pFedClub. Also, we add the flexible controllability parameter $\eta$ to decide the generated model size. The results are shown in Figure 3. Compared with pFedHR (shown in blue), the generated models by the pFedClub (shown in red) and pFedClub+(shown in green) are significantly smaller than the ones generated by pFedHR.

>>> W3

Thank you for the reviewer's valuable comments. As we mentioned in the limitations section, our experimental results are currently limited to image classification tasks. However, given the generalized design and intuitive nature of our approach, we believe it can be adapted to other tasks, provided that the models can be assembled and reassembled. We are committed to exploring these possibilities in future research directions.

>>> W1

Thank you for pointing this out. We follow existing work [1] to maintain the natural order of the blocks, as we aim for the generated candidate models to be similar to handcrafted network structures. Here, the natural order index is defined as the position of each block. For example, CNN1 listed in Appendix Section C consists of four layers: Conv1, Conv2, FC1, and FC2, with corresponding block order indexes of 1, 2, 3, and 4. In the generated model, we do not expect the FC1 layer to appear before the two convolutional layers, Conv1 and Conv2. Therefore, we constrain the order of blocks when generating candidate models.

We also have an example in Figure 2. In step 2: Order-constrained Block Search, if the first selection for the target network (in red) is the second block (in green), the next selected block for block 2 of the target network must have an index larger than 2. In this example, we select two “3” blocks as candidates for the second block replacement. Similarly, the third block replacement will use indexes “4” and “5”. We will add these details in the final version of our paper.

[1] Yang et al. Deep model reassembly. NeurIPS, 2022.

>>> W2

Thanks for the reviewer’s suggestion. CKA represents the centered kernel alignment, which is a method generally used to calculate the similarity of the neural network representations [2]. Specifically, given two kernels of feature representations K and L, we first calculate the Hilbert-Schmidt Independence Criterion (HSIC) by HSIC(K,L) = $1/(n-1)^2$ tr(KHLH), where n is the dimension of the representation of the features, and H is the centering matrix. CKA(K,L) = HSIC(K,L)/$\sqrt{(HSIC(K,K)HSIC(L,L))}$. Due to the page limit, we did not add those specific details in the original paper. We will add them in the final version.

As we stated in Sec. 3.2, all the operations including CKA occur on the server. The clients do not need to transmit x and output to the server. Instead, clients only transmit their model parameters to the server, following the conventional federated learning approach. Upon receiving these parameters, the server uses public data as input to the initial block of models, with the output of each block serving as input for the subsequent block. We have detailed the experimental settings with public data, demonstrating that our method performs effectively with both labeled and unlabeled public data, outperforming other baselines.

[2] Kornblith et al. "Similarity of neural network representations revisited." ICML, 2019.

>>> W3

Thanks for the reviewer’s valuable question. To ensure the generalization of our approach, the dimensions of the two features do not have to be the same, and CKA is able to handle this situation based on what we have addressed in W2 [2].

>>> W4

Thanks for the reviewer’s suggestion. The anchor block selection process considers the groups G. For model $w_m^t$, assuming its the first block $B_{m,1}^{t}$ belongs to the group $G_k^t$, we randomly select one block from $G_{k}^{t}$ as the substitution. Any block from the group $G_k^t$ has the equal probability $1/G_k$ to be selected (Line 169), where $G_k$ is the number of the blocks belonging to $G_k^t$. With such a strategy, compared with the strategy using the two blocks with the largest similarity score, it increases the diversity of the generalized candidates. At the same time, this random strategy is effective, generalized to all models, and simple to implement. As the reviewer suggested, we will add more descriptions of the anchor block selection process in the final version.

We do appreciate the reviewer’s suggestion. We introduced the proposed algorithm CMSR in the appendix A. As the reviewer suggested, we will further enhance the readability and reproducibility in the final version.

>>> Q1

The clustering processing is based on the functionality in Line 139 with no direct relation with the number of clients based on our design. With an increasing cluster number in some range, it indicates that we will get a more specific approach to differentiate the functions of each block, which may have some boost on the performance, as shown in Table 5.

We totally agree with the reviewer about the exploration of the activated client number. We have conducted the experiments and shown the results in Table 4. In Table 4, we maintain the same value of K = 4 as the main experiment. Then we study a different total number of clients and different active ratios under the IID and non-IID settings. Furthermore, we provided the experiment observations and respective model scalability analysis in Appendix F. Combining the results in Table 4 and Table 5 and its analysis in Appendix G and F, it demonstrates the scalability and robustness of our proposed method.

>>> Q2

Thanks for the valuable question. In our limitation part, we admitted that our existing approach has not been tested on other tasks, such as those with the transformer-based structures. We sincerely appreciate the constructive suggestions from the reviewer. We will add more discussion in our limitation part and expand our work following this direction in the future.

>>> Q3

We appreciate the reviewer's suggestion. Our experiment setup, detailed in Section 4.1, includes descriptions of the datasets, baselines, client model deployment, and implementation specifics. We have ensured that common parameters are consistently maintained across different baselines while retaining the unique parameters of each baseline at their default values. As suggested by the reviewer, we will include a more comprehensive set of implementation details in the final version of our paper.

We are sincerely grateful for the reviewer's constructive questions and suggestions. We hope our responses have adequately addressed your concerns. Thank you once again for your valuable feedback, which has significantly contributed to enhancing the quality of our paper.

>>> W1

Thank you for your comments. We would like to emphasize that our research question addresses a significantly challenging problem, where each client maintains a unique model. Aggregating these heterogeneous models on the server is particularly difficult without the use of public data. Additionally, we adhere to established work that serves as baselines in our experiments, which also incorporate public data during model training. While we acknowledge that differences in data distributions between public and private data can affect performance, our model design mitigates this issue, similar to the approach taken by pFedHR.

Following the reviewer’s suggestion, we conducted an additional experiment where clients train models using CIFAR-10, and the server utilizes a public dataset from ImageNet. All other hyperparameters were kept consistent with those listed in Table 1 for Model Zoo 1 under the non-IID setting. The results of this experiment are as follows:

We observe that using ImageNet as the public dataset does decrease performance compared to using CIFAR-10 directly. However, the performance drop is limited, and it still outperforms the best baseline, pFedHR (Labeled: 69.88 and Unlabeled: 68.54), as shown in Table 1.

>>> W2

Thank you for your feedback. In our current work, we focused on reassembling convolutional neural networks in our experiments, as they consist of distinguishable layers. However, our model design is flexible and can also be applied to transformer-based models, as demonstrated in [1], which segments transformer models into functional blocks and reassembles them into candidates. We did not test the transformer or more advanced models mentioned by Reviewer 5tH7 due to the following reasons: This work focuses on controllable personalized model generation, specifically designed for resource-constrained clients. Training transformer-based models from scratch typically requires a large amount of training data, which may be impractical for clients using mobile devices due to computational costs. Therefore, we did not include experiments with transformers in this study. We can include additional results involving transformer models in the final version of our paper if required.

[1] Yang, Xingyi, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. "Deep model reassembly." Advances in neural information processing systems 35 (2022): 25739-25753.