A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learning

We thank Reviewer PKSf for the very positive assessment.The acknowledgement of the paper’s clear presentation, meaningful focus, comprehensive theory, and strong empirical gains is greatly appreciated.

The remaining points are minor clarifications, all addressed below without affecting our core contributions and claims.

1. Clarification on High-Heterogeneity Regime ($\lambda > \lambda^*$)

Our analysis in Section 4.2 focuses on combined concept + covariate shift under a low-heterogeneity regime. As shown in Theorem 4.5, LP-FT provably outperforms FT when $\lambda \le \lambda^{*}$. Bounding heterogeneity is standard in FL theory; when client distributions become completely disjoint, the global objective decomposes into a sum of unrelated losses, so a single model provides no benefit over training one model per client [1, 2]. For completeness, we empirically sweep a wide range of $\lambda$ in Appendix F to give a better understanding of our Theorem 4.5.

[1] Li, Tian, et al. "Federated optimization in heterogeneous networks." Proceedings of Machine learning and systems 2 (2020): 429-450. [2] Karimireddy, Sai Praneeth, et al. "Scaffold: Stochastic controlled averaging for federated learning." International conference on machine learning. PMLR, 2020.

2. Response to Weaknesses W1–W3

W1: Paper Organization. Our Section 1 Introduction already lists the three core contributions (methodological, empirical, theoretical) and each main Section 3, 4, 5 begins with a short roadmap to reinforce that structure. We will add brief sign-posting statements to further improve readability.

W2: Choice of Baselines. Process-integrated PFL frameworks modify aggregation, add coordination, or embed new objectives—precisely the variables we hold constant. Including them in the main table would mix improvements from modified global training with those from post-hoc personalization, obscuring the effect we aim to isolate. Thus, Table 2 reports only post-hoc fine-tuning variants, all applied to the same FedAvg global model (stated in Section 3.2 Line 151 to 157). This setup eliminates confounding factors from global training, enabling a clean and fair assessment of post-hoc methods.

W3: Fairness Metrics. Although fairness is orthogonal to our core goal of balancing local and global accuracy, Table 2 already reports client-wise standard deviation and worst-case accuracy, metrics that directly quantify performance disparities across clients.

We appreciate Reviewer FGp1’s positive remarks on our method’s simplicity and the breadth of our empirical study, highlighting that our method is “easy to understand and follow” and that we provide a “comprehensive empirical sweep” across seven datasets and 50 corruption types.

The remaining points mainly request additional implementation and experimental clarifications (e.g., two-stage rationale, aggregation variants, backbone details, modality scope). We provide those clarifications and brief analyses in the responses below.

1. Response to Baseline Selection and Motivation of LP-FT

Scope & Positioning. Our objective is not to design another process-integrated PFL algorithm, but to answer a research question: once a standard global model is trained, how best to personalize it post-hoc to balance local and global performance? To keep this question focused, we lock the global-training phase (FedAvg) and study only methods that act after training, with no extra communication or server changes (see Figure 1 Caption Line 3).

Baseline Choice. Process-integrated PFL frameworks modify aggregation, add coordination, or embed new objectives—precisely the variables we hold constant. Including them in the main table would mix improvements from modified global training with those from post-hoc personalization, obscuring the effect we aim to isolate. Thus, Table 2 reports only post-hoc fine-tuning variants, all applied to the same FedAvg global model (stated in Section 3.2 Line 151 to 157). This setup eliminates confounding factors from global training, enabling a clean and fair assessment of post-hoc methods.

Why adopt a two-stage LP-FT scheme?

• Head-only tuning preserves shared features but provides limited personalization, so local accuracy remains modest.
• Full tuning from the outset personalizes strongly yet perturbs the backbone, leading to over-fitting and poorer global accuracy.

The sequential LP-FT procedure reconciles these extremes:

• Linear probing (LP)  — update only the classifier head while the backbone is frozen, quickly aligning predictions to client labels without disturbing global representations.
• Full fine-tuning (FT)  — with the head stabilised, softly adapt the backbone; the anchored representation lets local accuracy rise without sacrificing global performance.

This behaviour appears empirically in Figure 2 and is formally supported by Theorems 4.4 & 4.5. Hence, the two-stage design is crucial for balancing personalization and generalization in heterogeneous FL.

2. Response to Narrow Modality Our paper focuses on image classification tasks, following the precedent set by the original LP-FT paper setup to allow direct analysis of centralized versus federated settings. However, the LP-FT framework itself is not restricted to image data.

To address this concern, we tested LP-FT on the Shakespeare text dataset (117 clients). Results below show that LP-FT consistently outperforms baselines on this language task as well, indicating that the approach generalizes to heterogeneous, non-vision modalities:

We will include these results and implementation details in the revised manuscript.

3. Response to Experimental Setup & Pre-trained Model

We would like to clarify that all of our specifics are detailed in Appendix C.3 and Table 4 due to the space limit in the main paper.

• Architectures

  • ResNet-18 — Digit5, CIFAR
  • ResNet-50 — DomainNet, CheXpert, CelebA
  • ViT — DomainNet (transformer experiment)

• Pre-training
  Each model is first trained with FedAvg on the all clients’ data, yielding the global model.

• Client partitioning
  Dataset-specific splits are described explicitly in the appendix.

The term “pre-trained model” refers to this global model obtained after the GFL stage (FedAvg in our experiments). It serves as the starting point for all local fine-tuning. Our theoretical results are agnostic to the specific aggregation algorithm used to obtain the global model.

4. Response to LP-FT Not Always Being the Best

As our results suggest, no method simultaneously achieves optimal local, global, and worst-case accuracy across all heterogeneity levels. In relatively simple tasks, more aggressive adaptation methods such as Proximal FT or Sparse FT may only achieve marginally higher local accuracy than LP-FT. However, the trade-off changes once task difficulty increases (e.g., for CIFAR-10-C and CIFAR-100-C), As shown in Table 2, LP-FT preserves global and worst-case accuracy far better than other baselines, while still delivering strong local gains. This behavior is consistent with our theory: Theorem 4.5 guarantees LP-FT’s global advantage under bounded heterogeneity without claiming it must top local accuracy on the simplest cases.

Thank you to the authors for the detailed rebuttal. I appreciate the clarifications provided, and I acknowledge that some of my concerns (such as points 2 and 3) have been addressed. However, I share Reviewer 9Ytb’s perspective that the approach of fine-tuning both the backbone and the classifier is relatively classic approach, particularly for image tasks using CNN-based architectures in FL settings. As such, I tend to maintain my original score.

We thank Reviewer 9Ytb for highlighting several positives. We are glad the reviewer recognizes that the paper “systematically introduces the LP-FT strategy … aiming to balance generalization and personalization in FL,” that the approach is “lightweight and applicable to various types of data heterogeneity,” and that the accompanying theoretical analysis is “rigorous.”

The perceived limitations stem from a misunderstanding of our setting and positioning. Our study targets the post-hoc PFT stage—orthogonal to process-integrated PFL—and this scope is made explicit in Fig. 1 and the Section 1 Paragraph 2. Hence SOTA PFL algorithms are not primary baselines; nevertheless, we included representative PFL + PFT results in the Appendix to show LP-FT’s plug-and-play advantage. Regarding novelty, our contribution extends far beyond re-using LP-FT: we provide new FL-specific theory (multi-client ground truths, combined concept–covariate shifts) and the first large-scale PFT benchmark across diverse distribution shifts. These aspects, already highlighted in the paper, demonstrate substantive originality beyond the core algorithm.

1. Response to Positioning and Novelty

Scope & Positioning. Our objective is not to design another process-integrated PFL algorithm, but to answer a research question: once a standard global model is trained, how best to personalize it post-hoc to balance local and global performance? To keep this question focused, we lock the global-training phase (FedAvg) and study only methods that act after training, with no extra communication or server changes (see Figure 1 Caption Line 3).

Baseline Choice. Process-integrated PFL frameworks modify aggregation, add coordination, or embed new objectives—precisely the variables we hold constant. Including them in the main table would mix improvements from modified global training with those from post-hoc personalization, obscuring the effect we aim to isolate. Thus, Table 2 reports only post-hoc fine-tuning variants, all applied to the same FedAvg global model (stated in Section 3.2 Line 151 to 157). This setup eliminates confounding factors from global training, enabling a clean and fair assessment of post-hoc methods.

Novelty: Moving Beyond a Simple Adaptation of LP-FT

• Methodology. Contrary to the perception that we simply transplant LP-FT into a federated setting, our work presents a purposeful shift: we apply LP-FT exclusively during the final local training phase in FL (Section 1 Paragraph 2, Contribution 1). Unlike most personalized federated learning (PFL) methods that modify the entire training pipeline, our plug-and-play approach focuses on the critical—but underexplored—last stage of local fine-tuning (Table 1, Figure 1). This simple yet effective design on FL is a core strength, enabling broad deployability and improvements over existing techniques without altering global training.
• Theory. While centralized LP-FT assumes a single ground-truth function, our theoretical framework captures multiple client-specific ground truths (Section 1 Contribution 3). Specifically, we define data-generating functions with unique linear heads for each client and allow both concept shifts and covariate shifts—a stark contrast to centralized assumptions. These FL-specific assumptions (Assumption 4.1) are crucial for analyzing feature distortion under heterogeneous conditions. Furthermore, our gradient-descent-based comparison of FT vs. LP-FT in FL reveals why LP-FT preserves global features more effectively, offering new insights into the interplay of local updates and shared representations.
• Empirical Contributions. We extensively benchmark across seven datasets and multiple distribution shifts (Section 1 Contribution 2). Our findings expose pronounced overfitting issues in standard PFT methods (Figure 2), present in-depth ablations (Figure 3, Table 2) and connect our theory on heterogeneity (Table 6 in Appendix) and feature distortion to real-world FL performance. These experiments go beyond the scope of centralized LP-FT by exploring more nuanced data shifts, validating LP-FT’s robustness and shedding light on why it outperforms existing personalization strategies.

Taken together, these points demonstrate that our method is not a mere adaptation of LP-FT, but a tailored solution for FL personalization that marries simplicity, strong theory, and comprehensive empirical validation.a

2. Response to formal equations for procedural steps

We already outline the experimental setup and steps in Sec. 3.2 (lines 151–157), Fig. 1, and Appendix C, but will add the concise formalism below for adding more details:

Model. Each client has a feature extractor $\theta_i^{f}$ and a linear head $\theta_i^{\ell}$ with local loss $\mathcal{L}_i\bigl(\theta_i^{\ell},\theta_i^{f}\bigr)$.

LP stage: update only the head $\theta_i^{\ell} \leftarrow \theta_i^{\ell} - \eta\,\nabla_{\theta_i^{\ell}}\mathcal{L}_i$ (freeze $\theta_i^{f}$).

FT stage: update both blocks $(\theta_i^{\ell},\theta_i^{f}) \leftarrow (\theta_i^{\ell},\theta_i^{f}) - \eta\,\nabla\mathcal{L}_i$.

All clients start from the same trained FedAvg global model initialization; no further communication is required.

3. Clarification on High-Heterogeneity Regime ($\lambda > \lambda^*$)

Our analysis in Section 4.2 focuses on combined concept + covariate shift under a low-heterogeneity regime. As shown in Theorem 4.5, LP-FT provably outperforms FT when $\lambda \le \lambda^{*}$. Bounding heterogeneity is standard in FL theory; when client distributions become completely disjoint, the global objective decomposes into a sum of unrelated losses, so a single model provides no benefit over training one model per client [1, 2]. For completeness, we empirically sweep a wide range of $\lambda$ in Appendix F to give a better understanding of our Theorem 4.5.

References

• [1] Li, Tian, et al. "Federated optimization in heterogeneous networks." Proceedings of Machine learning and systems 2 (2020): 429-450.
• [2] Karimireddy, Sai Praneeth, et al. "Scaffold: Stochastic controlled averaging for federated learning." International conference on machine learning. PMLR, 2020.

We thank Reviewer 97oc for their careful reading and positive assessment. We are pleased the reviewer describes our empirical study as “comprehensive and methodologically sound” (seven datasets × five baselines × five metrics), calls federated feature distortion a “valuable conceptual contribution,” and notes the “important formal grounding” of our non-trivial theoretical extension of LP-FT to Federated Learning.

The main outstanding issue is that our limitations section does not yet mention potential calibration degradation after LP-FT; we will add this discussion and cite relevant papers and standard mitigation techniques in the revision. Thank you for bringing this to our attention.

1. Response to Calibration

Calibration, though important, falls outside our accuracy-centric personalization–generalization scope. We will add it to Limitations, cite the LP-FT calibration literature. Our MSE-regression theory analysis is norm-independent; extending it to probabilistic outputs and calibration metrics is left for future work.

Metric Scope & Focus. Thank you for pointing out the need to discuss calibration; we will add an explicit note in the Limitations section and cite recent LP-FT calibration work [1, 2]. Our study focuses on a different axis—balancing personalization and generalization in post-hoc FL. Accordingly, we follow prior LP-FT evaluations and report accuracy-based metrics to enable direct comparison between centralized and federated settings. A full calibration investigation in FL would require different metrics, extra experiments, and additional space, potentially shifting the paper’s focus. Nonetheless, we acknowledge calibration’s practical importance and will (i) flag it clearly in the Limitations, (ii) cite the few existing FL-calibration papers, and (iii) note that LP-FT’s two-stage structure permits application of standard post-hoc calibration methods (e.g., temperature scaling).

Theoretical Framework. While calibration is relevant empirically, our theoretical framework does not model it directly. This is because it is based on a regression setting, where the output $y \in \mathbb{R}$ and the model is trained using the MSE loss. As such, the theoretical formulation does not involve probabilistic outputs, confidence scores, or calibration metrics such as expected calibration error (ECE). In contrast, in classification settings—particularly when cross-entropy loss is used—temperature scaling can be applied as a post-hoc calibration method to mitigate the effects of overconfident predictions, which are often linked to large classifier norms [1]. However, our analysis does not rely on the norm of the classifier, and our generalization results are independent of the norm of the linear head.

We acknowledge that the analysis is based on a simplified linear model, which serves to abstract and isolate the key effect of federated feature distortion under personalization (as formalized in Theorems 4.4 and 4.5). While this abstraction is analytically useful, it is not designed to capture the full complexity of real-world FL systems, particularly those related to predictive uncertainty or calibration.

References

• [1] Tomihari, Akiyoshi, and Issei Sato. "Understanding linear probing then fine-tuning language models from ntk perspective." Advances in Neural Information Processing Systems 37 (2024): 139786-139822.<br>
• [2] Mai, Zheda, et al. "Fine-tuning is fine, if calibrated." Advances in Neural Information Processing Systems 37 (2024): 136084-136119.

2. Response to Positioning and Baseline Scope & Positioning. Our objective is not to design another process-integrated PFL algorithm, but to answer a research question: once a standard global model is trained, how best to personalize it post-hoc to balance local and global performance? To keep this question focused, we lock the global-training phase (FedAvg) and study only methods that act after training, with no extra communication or server changes (see Figure 1 Caption Line 3).

Baseline Choice. Process-integrated PFL frameworks modify aggregation, add coordination, or embed new objectives—precisely the variables we hold constant. Including them in the main table would mix improvements from modified global training with those from post-hoc personalization, obscuring the effect we aim to isolate. Thus, Table 2 reports only post-hoc fine-tuning variants, all applied to the same FedAvg global model (stated in Section 3.2 Line 151 to 157). This setup eliminates confounding factors from global training, enabling a clean and fair assessment of post-hoc methods.

3. Response to LP-FT vs. LoRA LoRA is parameter-efficient, but it still injects trainable adapters into the backbone, inevitably perturbing the shared representations—similar in effect (though lower rank) to full fine-tuning. LP-FT, in contrast, freezes the backbone during its linear-probing phase, safeguarding global features before any further adaptation. As shown in Appendix D.1 (Table 5), this design yields higher local and global accuracy on transformer backbones than federated LoRA. In short: LoRA saves parameters yet still distorts features, whereas LP-FT preserves global knowledge and achieves a superior personalization-generalization balance.

4. Clarifying “Federated Feature Distortion (FFD)”

Federated Feature Distortion (FFD) is a new, FL-specific concept we introduce to capture how client-side fine-tuning shifts the model’s learned features. Specifically, FFD measures the average $\ell_2$ distance between representations from the global model and the locally fine-tuned models (see Section 3.5).

How FFD relates to existing concepts:

• Classifier Bias: Prior FL work focuses on misaligned classification heads due to non-IID data. In contrast, FFD measures drift in the feature extractor itself and complements these head-focused analyses.
• Catastrophic Forgetting: Forgetting usually refers to loss of global knowledge across rounds or tasks. FFD, instead, captures one-step representation drift caused by local post-hoc fine-tuning.

Why FFD is new and important: FFD is, to our knowledge, the first formal measure of feature drift in post-hoc FL personalization. We define it explicitly, quantify it across clients, and provide theoretical analysis in Theorems 4.4 and 4.5.

We will update the paper to include a brief comparison to classifier bias and forgetting, and explicitly highlight FFD’s novelty and scope.

Thanks for the rebuttal. I will keep my initial score.