DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices

We sincerely thank you for your constructive and helpful comments. Below we address your concerns in order.

Response to Weakness 1:

Clarification on System Heterogeneity: In our paper, we emphasize that low-capability clients "fail to complete" local training rather than being "unable to perform" local training. Following the conventional setting [1,2,3], low-capability clients (i.e., stragglers) possess the basic capability to train models. However, they fail to complete training within the maximum allowed time due to limited resources, such as battery status or CPU power. MFP addresses this issue by reducing resource consumption, thereby enabling the stragglers to participate in FL.

Clarification on Domain Shifts Example: Images collected by the cameras are not exclusively "multi-view" data, particularly when factors such as lighting conditions or weather vary which are classified as domain shifts. Additionally, images with different view angles only (i.e., "multi-view") are also multi-domain data. For instance, previous work [4] has utilized rotated images to create multi-domain datasets.

Application Scenarios and Illustration: Thank you for your insightful comments. Due to the length limitations, please refer to Responses to Weakness 3 & Question 2 from Reviewer Ct3S for the details.

[1] Lim, Wei Yang Bryan, et al. Federated learning in mobile edge networks: A comprehensive survey. IEEE Commun. Surv. Tutorials 2020.

[2] Nguyen, Dinh C., et al. Federated learning for Internet of Things: A comprehensive survey. IEEE Commun. Surv. Tutorials 2021.

[3] M. Gerla, et al. Internet of Vehicles: From intelligent grid to autonomous cars and vehicular clouds, WF-IoT 2014.

[4] Nguyen, A. Tuan, et al. FedSR: A simple and effective domain generalization method for federated learning. NIPS 2022.

Response to Weakness 2 & Question 1 & Question 2:

Innovative and Practical Differences of MFP: As detailed in lines 143-160, Algorithm 1, and Fig. 1, a dynamic model fusion mechanism is proposed within MFP. This mechanism is distinct from regular FL. Inspired by transfer learning, DapperFL incorporates the global model into the fine-tuned model to assimilate cross-domain knowledge (formally represented in Eq. 1). Furthermore, a dynamic adjustment mechanism controls the quality of transferred knowledge (as described in Eq. 2), preventing overfitting to the local domain and improving the model's generality. Our experiments in the presence of domain shifts show that DapperFL with MFP improves accuracy by up to 7.45% (on the Digits benchmark) compared to SOTA frameworks that support system heterogeneity.

Clarification on Additional Computations and Global Model Deployment: It is important to note that MFP does NOT entail additional computations. MFP requires clients to fine-tune the model during the initial epoch, consuming resources comparable to regular FL. In subsequent epochs, the pruned model undergoes updates, significantly reducing resource consumption (up to 80%). Moreover, fine-tuning the global model for at least one epoch is essential in popular FL methods to capture data features, and this is not unique to ours.

Clarification on Pruning Strategy: We would like to highlight that, the pruning strategy used in DapperFL is structural pruning (channel pruning), rather than unstructural pruning. Channel pruning is hardware-friendly and can be easily implemented with popular ML libraries such as PyTorch, making it suitable for edge devices. We provide our code in the Abstract, which details the implementation of our approach. We want to emphasize that "unstructured pruning" is not presented in our paper. Nevertheless, we acknowledge the importance of clarity and will include a description of the adopted structural pruning.

Response to Weakness 3:

We appreciate your constructive suggestion. Specifically, the global model transitions to $\mathcal{W}^t$, the fine-tuned model transitions to $\boldsymbol{w}^{t,i}\_{ft}$, and the fused model transitions to $\boldsymbol{w}\_{fs}^{t,i}$.

Response to Weakness 4:

Thank you for highlighting these studies that offer valuable insights into heterogeneous FL and model pruning. We will incorporate them into our paper to discuss their relationship to ours. Despite the similarities, our work contributes novel solutions by focusing on domain shifts and the integration of MFP, DAR, and heterogeneous aggregation.

Response to Weakness 5:

Thank you for highlighting these important works. We did not include distillation-related work as they typically require maintaining both a teacher model and a student model. This imposes a significant burden and is impractical in our targeted scenarios. Nonetheless, we acknowledge the relevance of them and will incorporate them to enrich our literature review.

Response to Limitation 1:

Hyper-Parameter Selection: Please refer to Responses to Weakness 1 & Questions 1 from Reviewer 1ipw for the details.

Solution for System Heterogeneity: Please refer to Response to Weakness 2 & Question 1 & Question 2 for the details.

Theoretical Convergence Analysis: It is noteworthy that extensive prior work has established the convergence of model pruning-based FL [5,6,7,8]. Our DapperFL, which also incorporates model pruning, does not violate the convergence of FL. The experimental results in our paper further support the convergence of DapperFL.

[5] Zhou, Hanhan, et al. On the convergence of heterogeneous federated learning with arbitrary adaptive online model pruning. arXiv 2022.

[6] Li, A., et al. Hermes: an efficient federated learning framework for heterogeneous mobile clients. MobiCom 2021.

[7] Zhida Jiang, et al. Fedmp: Federated learning through adaptive model pruning in heterogeneous edge computing. ICDE 2022.

[8] Jiang, Yuang, et al. Model pruning enables efficient federated learning on edge devices. TNNLS 2022.

We sincerely thank you for your constructive and helpful comments. Below we address your concerns in order.

Response to Weakness 1:

We would like to emphasize that, our proposed framework is NOT a combination of existing approaches. While our work builds on established concepts, it introduces novel components and methods that address significant gaps in the current literature, particularly concerning domain shifts in resource-constrained edge environments. We outline the substantial differences and innovative contributions of our DapperFL framework from existing works as follows:

1. Transfer Learning-Inspired Model Fusion Pruning. Existing heterogeneous FL frameworks, such as [1,2,3], focus on addressing data heterogeneity (e.g., variations in sample and label distributions) but do not directly tackle the challenge of domain shifts, which can significantly impact the performance of pruned models. Our work introduces Model Fusion Pruning (MFP), which not only accounts for data heterogeneity but also addresses domain shifts. MFP dynamically combines local domain information with cross-domain information to prune local models effectively, enhancing their performance under domain shift conditions.

2. Lightweight Solutions for Domain Shifts. Current methods often require extensive data collection transmission to handle domain shifts, which is impractical in resource-constrained environments. Our DapperFL framework, however, offers a novel resource-efficient approach to handling domain shifts through the MFP and Domain Adaptive Regularization (DAR) modules. The MFP module prunes local models leveraging local and cross-domain information. The DAR module introduces a regularization term that encourages the generation of domain-invariant representations at the local level, eliminating the need for transmitting domain-invariant representations, which is commonly required in existing studies.

3. Specific Model Aggregation in Cross-Domain Environments. Existing works, including [3], lack mechanisms to ensure that domain-specific features are retained in heterogeneous environments. In contrast, we propose a novel model recovery mechanism during heterogeneous model aggregation. This mechanism ensures the preservation of specific domain knowledge while transferring global knowledge, as detailed in Eq.6 of our original paper. This unique operation reinstates the pruned model's architecture and incorporates global knowledge, differentiating our approach from existing methods.

[1] Jiang, Z., et al. Computation and communication efficient federated learning with adaptive model pruning. TMC, 2023.

[2] Li, A., et al. Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking. SenSys 2021.

[3] Li, A., et al. Hermes: an efficient federated learning framework for heterogeneous mobile clients. MobiCom 2021.

Response to Weakness 2 & Question 1:

We follow a conventional setting commonly used in representation learning research to segment the model. Specifically, all layers except the final linear layer act as the encoder, while the last linear layer of the model acts as the predictor. We acknowledge the importance of clarity and will include a detailed description of this segmentation process.

Response to Weakness 3 & Question 2:

Consider a mobile voice assistant like Apple Siri, which requires an efficient FL framework to collect and analyze user voice data [4]. Users may interact with Siri using different accents even when speaking the same language. This variation leads to domain shift problems, as the system must adapt to different speech patterns. Additionally, the diversity in mobile phone capabilities introduces system heterogeneity, as devices with varying processing powers may affect the performance of local computations. Another pertinent example is the IoV network, where smart vehicles collect, compute and communicate data for tasks like navigation and traffic management [3,5]. The data collected, such as images, can vary significantly in quality due to factors like motion blur, leading to multi-domain datasets. On the other hand, the varying computing power and network conditions of vehicles create system heterogeneity challenges [5]. Beyond these scenarios, our framework is relevant to various scenarios such as smart homes and smart health systems [1,2].

We have added Fig. 1 to the rebuttal PDF to visually demonstrate the impact of these issues on FL performance. Specifically:

1. System Heterogeneity: Device 1, with stringent resource constraints, fails to complete its model update within the maximum allowed time. As a result, the central server excludes it from the FL process and misses out on the data features captured by Device 1.
2. Domain Shifts: Devices 2 and 3 collect data from different domains, leading to diverse local models. With non-IID data, the global model aggregation may become biased, particularly if Device 3 contributes more data, thus potentially limiting the benefit for Device 2.

[1] Lim, Wei Yang Bryan, et al. Federated learning in mobile edge networks: A comprehensive survey. IEEE Commun. Surv. Tutorials 2020.

[2] Nguyen, Dinh C., et al. Federated learning for Internet of Things: A comprehensive survey. IEEE Commun. Surv. Tutorials 2021.

[3] M. Gerla, et al. Internet of Vehicles: From intelligent grid to autonomous cars and vehicular clouds, WF-IoT 2014.

[4] Paulik, Matthias, et al. Federated evaluation and tuning for on-device personalization: System design & applications. arXiv 2021.

[5] D. Ye, et al. Federated learning in vehicular edge computing: A selective model aggregation approach, IEEE Access 2020.

Response to Limitation 1:

Thank you for your insightful comments. Due to the rebuttal's length limitations, please refer to our Responses to Weakness 1 & Questions 1 from Reviewer 1ipw for detailed information.

We sincerely thank you for your constructive and helpful comments. Below we address your concerns in order.

Response to Weakness 1:

We utilize one epoch of local training to determine the pruned models for the following reasons:

1. Additional local epochs do not significantly enhance the model's performance, which justifies the use of a single epoch for efficiency. As noted in [1], experiments have demonstrated that extending local training beyond one epoch yields results comparable to those achieved with just one epoch.

2. In previous domain generalization-related FL research, such as [1], one epoch is also employed to collect local domain information. This method has proven adequate for capturing essential features and domain characteristics.

3. Pioneering research in model design and neural architecture search, such as in [2], has demonstrated that a few epochs are sufficient to obtain a coarse estimate of a sub-model. This approach is effective in quickly assessing model configurations without extensive computation.

[1] Zhang, Jianqing, et al. Eliminating domain bias for federated learning in representation space. NIPS 2024.

[2] Tan, Mingxing, et al. Mnasnet: Platform-aware neural architecture search for mobile. CVPR 2019.

Response to Weakness 2:

We have provided the default values for three framework-specific hyper-parameters in FedProx and MOON, as well as all four hyper-parameters in our DapperFL framework. Additionally, we have included more default values for the hyper-parameters in the comparison frameworks in the rebuttal PDF file, which will be incorporated into our final version. Specifically, the default values are $\alpha^{L2R}=0.01$ in FedSR, $\alpha^{CMI}=0.001$ in FedSR, and $\tau=0.02$ in FPL. These values are consistent with those used in the original implementations of these frameworks, ensuring accuracy and reliability in our comparisons.

Response to Weakness 3:

Thank you for your meticulous observation. To avoid confusion and ensure clarity, we will change the notation for client-level subscripts in the experiments section to $l$. We hope this adjustment will improve the readability and precision of our paper.

Response to Weakness 4:

We acknowledge that further elaboration on the limitations of this work could provide the reader with a fuller understanding. As an example, the limitations of hyper-parameter selection will be described below:

“Despite DapperFL shows promise in addressing system heterogeneity and domain shifts, it also introduces four hyper-parameters: $\alpha_0$, $\alpha_{min}$, $\epsilon$, and $\gamma$. These hyper-parameters influence the domain generalization performance of the global model. One limitation is the manual tuning required for these hyper-parameters, which may be time-consuming and require domain-specific expertise. As such, a potential future direction is the development of an automatic selection mechanism for these hyper-parameters. This enhancement would not only improve the flexibility of DapperFL but also make it more accessible to users without extensive knowledge in hyper-parameter tuning, thereby broadening the applicability and ease of deployment of our framework.”

Response to Question 1:

We opted for the $\ell_1$ norm instead of the $\ell_2$ norm for the following reasons: The $\ell_1$ norm requires fewer computational resources, making it more suitable for our framework, which targets resource-constrained edge devices. Additionally, as shown by pioneering work [3], there is no significant difference in the effectiveness of using $\ell_1$ norm versus $\ell_2$ norm for calculating mask matrices in the context of model pruning. This finding supports our choice, as the $\ell_1$ norm offers a more resource-efficient alternative without compromising the quality of the pruning process.

[3] Li, Hao, et al. Pruning Filters for Efficient ConvNets. ICLR 2022.

Response to Question 2:

In our proposed framework, the heterogeneous models are transmitted to the server along with the mask matrices. These mask matrices indicate the architecture of the heterogeneous models. Specifically, the mask matrices encode the structure of each model by highlighting which parts of the model are retained and which are pruned. This information allows the central server to understand and manage the varying architectures of the heterogeneous models during aggregation.

We sincerely thank you for your constructive and helpful comments. Below we address your concerns in order.

Response to Weakness 1 & Question 1:

Thanks for your suggestion to provide default values or an automatic selection mechanism for hyper-parameters. We acknowledge that hyper-parameter selection is indeed crucial, especially for practitioners looking to effectively apply the DapperFL framework.

In DapperFL, $\alpha \in [\alpha_{min}, \alpha_0]$ control the balance between local and global information, which can impact how well the model generalizes across different domains. The sensitivity factor $\epsilon$ determines the rate at which $\alpha$ decreases from $\alpha_0$ to $\alpha_{min}$, while $\gamma$ influences the regularization strength. In our paper, we have conducted ablation experiments (in Section 4.3) to identify suitable hyper-parameter settings. The experimental results suggest that:

1. Increasing $\alpha_0$ or decreasing $\alpha_{min}$ generally enhances the incorporation of global knowledge, improving generalization. In our experiments, setting $\alpha_0$ to 0.9 and $\alpha_{min}$ to 0.1 resulted in good model accuracy.
2. As $\epsilon$ increases, model accuracy suffers from a decrease due to insufficient guidance from the global knowledge except for $\epsilon$ is less than 0.2.
3. $\gamma$ helps regularize the model and prevent overfitting. In our experiments, setting it to an optimal value of 0.01 can balance model personalization on the local domain and generalization across multiple domains.

Furthermore, recognizing the abnormal relationship between model accuracy and hyper-parameter $\epsilon$ as it is less than 0.2, we implement Bayesian search as an automatic selection mechanism to find a better $\epsilon$. Due to limited rebuttal time and hardware constraints, we have tried our best to include as many single runs as possible to search for a better $\epsilon$ for DapperFL. Specifically, we run DapperFL on the Office Caltech benchmark 40 times, adopting a distinct $\epsilon$ of less than 0.2 each time. The values are selected using the Bayesian search algorithm. The results are presented in Fig. 2 of the rebuttal PDF file. As illustrated, the Bayesian search-based automatic selection mechanism indicates that model accuracy is likely to reach a higher level when $\epsilon$ approaches 0.2, aligning with our original paper's default setting of $\epsilon=0.2$.

Response to Weakness 2 & Question 2:

The relationship between model accuracy and model footprint is complex and not necessarily linear, as evidenced by previous studies [1, 2]. In our work, we employ the MFP module within DapperFL, which addresses the over-fitting issues that can arise from the over-parameterization of neural networks. This module not only reduces the model footprint but also enhances the model's generalization capabilities, leading to more robust performance across diverse domains. Additionally, the DAR module is incorporated to further improve model accuracy, particularly in scenarios involving distributed clients with domain shifts.

[1] Z. Liu, et al. Learning efficient convolutional networks through network slimming, ICCV 2017.

[2] S. Shen, et al. Efficient deep structure learning for resource-limited IoT devices, GLOBECOM 2020.

Response to Weakness 3:

One primary objective of DapperFL is to optimize a robust global model with strong domain generalization capabilities across distributed local data. In the test or production phase, this optimized global model can be deployed on new devices, ensuring good performance across different environments. Additionally, newly introduced devices can also participate in the DapperFL framework. They can contribute to the continual improvement of the global model by optimizing a local model based on their own local data, which is then aggregated to update the global model. This approach not only enhances the global model's adaptability to new data distributions but also leverages the unique data available on new devices to improve overall model robustness.

Response to Weakness 4:

Thank you for highlighting these relevant studies. These studies offer valuable insights into heterogeneous FL and model pruning on devices, aligning well with the themes of our research. We will incorporate these references into our paper to discuss their relationship to our study, highlighting similarities, differences, and how our work extends or diverges from these existing approaches.

Response to Question 3:

In our work, “local knowledge” refers to the feature extraction capabilities of the local model, which are derived from the specific data available in its local domain. This knowledge encapsulates the nuances and characteristics of the data that the local model has been trained on. On the other hand, “global knowledge” represents the aggregated feature extraction capabilities of the global model, which are informed by data across all participating domains. The global model synthesizes diverse knowledge from different local models to provide a more generalized understanding that is applicable across multiple domains. By combining these two forms of knowledge, we aim to leverage both the specialized insights of local models and the generalized capabilities of the global model, thereby enhancing the performance and adaptability of DapperFL.