(For clarity in our response, we refer to the reviewer's points using the following convention: W.x (Weaknesses), Q.x (Questions), and L.x (Limitations).)

We sincerely thank the reviewer for their valuable and constructive feedback. Your comments have helped us to significantly improve the empirical rigor and ethical considerations of our paper. We have grouped our responses to address your main points regarding empirical validation and societal impact.

(W.1, W.2, Q.1, Q.2) On Empirical Rigor: Result Stability and Client-Side Costs

A key theme of your feedback (W.1, W.2, Q.1, Q.2) was the need for more rigorous quantification of our results' stability and the client-side overhead. We agree this is crucial and have conducted detailed new analyses to address these points.

1. Statistical Stability of Results (W.1, Q.1)

To demonstrate the stability of our findings, we have re-run all experiments three times with different random seeds and calculated the standard deviations for all metrics. The results confirm that our method is stable and the performance gains are consistent.

Due to space constraints, we present a representative table of standard deviations below and will include the complete set of tables in a new Appendix. The key finding is that the standard deviations are consistently small across all tasks and methods, confirming the reliability of our claims. Our method, LEFF, shows stability comparable to or better than the baselines.

2. Client-Side Overhead Analysis (W.2, Q.2)

We have performed a detailed analysis of client-side trainable parameters and peak GPU memory usage across various models. The results highlight LEFF's unique and superior efficiency profile.

Our analysis reveals two key insights:

• Lowest GPU Memory Pressure: LEFF drastically reduces peak GPU memory—for instance, achieving a 79% reduction on GPT2-Large compared to FedAvg and 66% compared to FedLoRA. This is possible because LEFF's architecture selectively loads only the necessary layers into GPU memory, whereas other PEFT methods must load the entire frozen model, resulting in much higher memory footprints.
• Optimal Performance-Efficiency Trade-off: LEFF deliberately retains more trainable parameters than methods like FedLoRA. This maintains greater model expressiveness, leading to superior task performance. Thus, LEFF strikes an optimal balance: it achieves the lowest memory cost for deployment on constrained devices while preserving enough capacity for high-accuracy fine-tuning.

These quantitative results confirm LEFF's superior design for practical, resource-constrained FL. We will integrate these findings into a new appendix.

(W.3) On Societal Impacts

We thank you for raising the important ethical considerations of environmental cost and dual-use risks (W.3). We will add a dedicated section to discuss these.

• Environmental Cost: We acknowledge the server-side computation in LEFF is a necessary trade-off. However, by enabling efficient fine-tuning on numerous distributed, low-power edge devices, our approach reduces the barrier to entry and can lessen the overall system's reliance on large, consistently energy-intensive data centers for training.
• Dual-Use Risks: We recognize this concern. A key strength of LEFF's federated architecture is that the central server acts as a natural control point. This allows for the implementation of safeguards like client vetting, anomaly detection, and monitoring model updates to mitigate misuse—a governance feature absent in purely decentralized or local fine-tuning scenarios.

We hope these detailed analyses and clarifications fully address your concerns. We are confident that incorporating these additions will significantly strengthen the paper.

(For clarity in our response, we refer to the reviewer's points using the following convention: W.x (Weaknesses), Q.x (Questions), and L.x (Limitations).)

We thank the reviewer for their insightful questions, which have helped us to clarify the efficiency, scalability, and theoretical underpinnings of our work. We have grouped our responses thematically.

(W.2) On Client-Side Efficiency: New Quantitative Results

You rightly pointed out the need for quantitative client-side efficiency metrics to substantiate our claims (W.2). To address this, we have conducted new experiments measuring peak GPU memory and the number of trainable parameters. The results below demonstrate LEFF's significant advantages.

This new analysis reveals two key insights:

1. Lowest Memory Footprint: LEFF drastically reduces peak GPU memory—for instance, achieving a 79% reduction on GPT2-Large compared to FedAvg and 66% compared to FedLoRA. This is possible because LEFF's architecture selectively loads only the necessary layers into GPU memory. In contrast, other PEFT methods must load the entire frozen model, resulting in much higher memory footprints.
2. Optimal Efficiency-Performance Trade-off: While achieving the lowest memory cost, LEFF deliberately retains more trainable parameters than methods like FedLoRA. This maintains greater model expressiveness, leading to superior task performance. LEFF thus strikes an optimal balance, delivering the minimal resource cost for client deployment while preserving enough capacity for high-accuracy fine-tuning.

We will add this detailed analysis to the appendix of our paper.

(W.1, Q.2) On Server-Side Cost and Scalability

We appreciate your important questions regarding server-side computational costs and scalability (W.1, Q.2). Our framework is designed to be highly efficient and scalable.

1. Low Per-Round Distillation Cost (for W.1)

The server-side knowledge distillation is computationally inexpensive, as its total workload is minimized by three key factors:

• Amortized Cost via Caching: The server does not perform a separate distillation for every client. Instead, if multiple clients request the compression of the exact same set of layers, the server performs this task only once. The result is then cached and distributed to all relevant clients, drastically reducing the total number of required distillation processes per round.
• Incremental Fine-tuning (Warm Start): The process is not training from scratch. The distilled "student" model from round t-1 serves as a warm start for round t. Since the global "teacher" model evolves gradually, the student only needs a brief fine-tuning to catch up.
• Small Target Model: By definition, the distillation operates on a "student" model that is significantly smaller than the full model (controlled by r), making each individual training process inherently fast.

2. High Scalability with Increasing Clients (for Q.2)

The server's computational overhead scales efficiently for large-scale deployments, as neither of its main tasks grows linearly with the number of clients:

• Layer Importance Calculation: This is performed only once per round, requiring a single forward and backward pass on the global model. Its cost is therefore constant regardless of the number of clients (e.g., 100 or 10,000).
• Distillation Workload: As explained above, the caching mechanism decouples the distillation workload from the client count. The total cost is tied to the number of unique compression requests, which is significantly smaller than the total number of clients in a large-scale setting. This ensures the distillation overhead scales sub-linearly, making the entire framework highly scalable.

(W.3, Q.1, Q.3) On the Proxy Dataset and Theoretical Guarantees

Your questions about the proxy dataset (W.3, Q.3) and the interpretation of our theory (Q.1) are spot-on. These two aspects are deeply connected.

1. Proxy Dataset Robustness: To empirically address your concern, we ran a new ablation study on the E2E NLG task, showing that LEFF is highly robust to the choice of proxy data.

1. Robustness to Data Distribution: Our method is robust to the choice of proxy data because we distill functional representations, not task-specific knowledge. As described in Section 3.3, our objective is to match the intermediate hidden states and attention matrices. This process preserves general representational capabilities and is far less sensitive to the specific domain of the input data.
2. Practical Availability: Since our method only requires unlabeled text, readily available public corpora (e.g., C4, Wikipedia) are sufficient. In scenarios where external data is prohibited, the server can use the global model to generate a synthetic proxy dataset, making the framework entirely self-contained — a promising direction we are exploring.
3. New Ablation Study: We empirically demonstrate LEFF’s robustness to the proxy dataset via a new ablation study on the E2E NLG task. As shown in the table, performance remains remarkably stable across diverse corpora, from the in-domain WebNLG to general-purpose WikiText-103 and OpenWebText. While the in-domain data yields a slight, expected performance gain, the minimal fluctuation across all metrics confirms that LEFF is not reliant on a perfectly-matched proxy corpus. This high degree of robustness validates its practical applicability. We will include this study in the appendix.

2. Theoretical Interpretation: Your interpretation of our theory (Q.1) is entirely correct. Our convergence bound (Thm. 1) explicitly shows that the performance floor depends on the compression quality, captured by the approximation error term $\bar{\Delta}^2$. This dependency is not a weakness but the core, deliberate trade-off our work models and manages.

Our contribution lies in making this trade-off explicit and demonstrating empirically (e.g., Fig. 6) that we can manage it to achieve a state-of-the-art balance between efficiency and performance. Our main results (Table 2) confirm that despite this inherent dependency, LEFF's approach is highly effective, outperforming other PEFT methods in challenging FL settings.

We hope these detailed quantitative results and clarifications have fully addressed your concerns. Thank you again for helping us improve the paper.

(For clarity in our response, we refer to the reviewer's points using the following convention: W.x (Weaknesses), Q.x (Questions), and L.x (Limitations).)

We sincerely thank the reviewer for their thorough and constructive feedback. Your suggestions have been instrumental in helping us strengthen our paper's evaluation, clarify its core contributions, and improve its structure. We have grouped our responses to address the main themes of your review.

(W.2, W.3, W.6) On SOTA Models, Baselines, and Literature Review

Your feedback regarding the need for more recent baselines, SOTA models, and a broader literature review (W.2, W.3, W.6) is well-taken. We agree that this is essential for contextualizing our work.

To address this directly, we conducted new experiments on Llama-3.1-8B with the MMLU benchmark, including FLoRA (NeurIPS 2024) [1] and FlexLoRA (NeurIPS 2024) [2] as representative SOTA baselines. The results are below:

The results show LEFF outperforms these recent NeurIPS 2024 baselines by 1.8-2.3 points while reducing client peak memory by over 36%. This demonstrates that LEFF's principles are not only scalable but highly effective for modern, billion-parameter models.

Furthermore, we have studied the suggested papers on federated full-parameter tuning ([3], [4], [5]) and will add a detailed discussion to our Related Work section. We will clarify LEFF's novelty by contrasting its approach:

• Unlike ZOO-based methods [3] that approximate gradients, LEFF utilizes standard first-order optimization (backpropagation). This approach is significantly more computationally efficient on the client side, avoiding the high costs of multiple forward passes per step and typically leading to faster convergence.
• In contrast to update-compression methods [4] which compute a full gradient update before compressing it, LEFF restricts the update scope a priori to a subset of layers. This design directly reduces the client's peak memory and computational load during training by confining backpropagation, rather than only saving communication bandwidth post-computation.
• While cyclical methods [5] update model blocks in a sequential turn-based manner, LEFF employs a standard synchronous and parallel training protocol common in FL. Moreover, our dynamic, importance-based layer sampling enables more adaptive training by focusing resources on the most impactful parameters each round, instead of following a static, pre-determined cycle.

[1] FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations, NeurIPS 2024.

[2] Federated fine-tuning of large language models under heterogeneous language tasks and client resources, NeurIPS 2024.

[3] Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes, ICML 2024.

[4] Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models, arXiv 2409.

[5] Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descent, arXiv 2406.

(W.4) Clarifying LEFF's Advantage Over Conventional PEFT

We appreciate your insightful question (W.4) on why LEFF, as a form of PEFT, overcomes the limitations of other PEFT methods under non-IID conditions. The key distinction lies in the nature and capacity of the parameters being tuned.

1. Conventional PEFT (e.g., LoRA) freezes the original model and injects a small number of new, low-rank auxiliary parameters. This fundamentally constrains model adaptation to a low-dimensional space, which is insufficient to capture the diverse data distributions across clients in non-IID settings [1].
2. LEFF, in contrast, tunes the original, full-rank parameters of entire layers. While each client trains a subset of layers, these updates have the full expressive power of full-parameter tuning for those layers. This high-capacity adaptation is crucial for fitting the complex, client-specific data. From the global model's perspective, all parameters are gradually fine-tuned over the course of training. This principle is supported by recent findings in centralized tuning (e.g., LISA [2]), where layer-wise methods have also demonstrated superior performance over LoRA-style tuning.

The following table for DeBERTaV3-Large highlights this difference. LEFF tunes significantly more parameters than other PEFT methods, granting it greater expressive power, yet it achieves the lowest peak memory by only loading the necessary layers into GPU memory.

This design explains why LEFF's performance under high data heterogeneity is substantially better than conventional PEFT methods, as shown empirically in our Table 1 and Figure 5. We will add this clarification to the introduction.

[1] SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models, NeurIPS 2023.

[2] LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning, NeurIPS 2024.

(W.5, W.7) Empirical Validation: Proxy Dataset and Convergence

We have conducted new analyses to address your concerns about the proxy dataset (W.5) and convergence curves (W.7).

Proxy Dataset Robustness: To empirically validate our method's robustness, we conducted a new ablation study on the E2E NLG task by varying the proxy dataset.

1. Robustness to Data Distribution: Our method is robust to the choice of proxy data because we distill functional representations, not task-specific knowledge. As described in Section 3.3, our objective is to match the intermediate hidden states and attention matrices. This process preserves general representational capabilities and is far less sensitive to the specific domain of the input data.
2. Practical Availability: Since our method only requires unlabeled text, readily available public corpora (e.g., C4, Wikipedia) are sufficient. In scenarios where external data is prohibited, the server can use the global model to generate a synthetic proxy dataset, making the framework entirely self-contained — a promising direction we are exploring.
3. New Ablation Study: We empirically demonstrate LEFF’s robustness to the proxy dataset via a new ablation study on the E2E NLG task. As shown in the table, performance remains remarkably stable across diverse corpora, from the in-domain WebNLG to general-purpose WikiText-103 and OpenWebText. While the in-domain data yields a slight, expected performance gain, the minimal fluctuation across all metrics confirms that LEFF is not reliant on a perfectly-matched proxy corpus. This high degree of robustness validates its practical applicability. We will include this study in the appendix.

Convergence Analysis: We agree that showing convergence is vital. Due to format restrictions, we present the validation loss data below (under high heterogeneity, $\alpha=0.05$) and will include the full plot in the final paper.

This data empirically validates our theory (Thm. 1), showing that LEFF converges stably and efficiently, achieving a final loss much closer to full fine-tuning than other PEFT methods.

(W.1, L.1) Paper Structure and Presentation

Finally, we appreciate your suggestions for improving the paper's presentation (W.1, L.1).

• We agree with your assessment of Figure 1. We will remove it and reallocate the space to our new experimental results, which provide stronger empirical evidence.
• Following your advice, we will create a dedicated "Section 6: Limitations and Future Work" before the conclusion to improve clarity and transparency.

We are grateful for your thorough review. We believe these extensive new experiments, clarifications, and structural changes directly address all your concerns and significantly strengthen the paper.

(For clarity in our response, we refer to the reviewer's points using the following convention: W.x (Weaknesses), Q.x (Questions), and L.x (Limitations).)

We sincerely thank the reviewer for their insightful and constructive feedback, which has helped us to significantly strengthen our paper. We have grouped our responses thematically to address the core concerns raised about SOTA model applicability, approximation error, proxy data dependency, and computational cost.

(W.3, Q.2, Q.3, L.1) Performance on SOTA Models (Llama-3.1-8B) & Benchmarks (MMLU)

A central theme of your feedback (concerns W.3, Q.2, Q.3, L.1) was the need to evaluate LEFF on more recent models and challenging benchmarks to assess its competitiveness. We agree this is crucial and have conducted new experiments during the rebuttal period to address this directly.

Our initial experiments on GPT-2/DeBERTaV3 were designed to rigorously validate LEFF’s core contributions to efficiency and robustness in controlled, heterogeneous FL settings, where tasks like GLUE allow for clear analysis of factors like label skew.

To prove LEFF's effectiveness on modern architectures, we have now evaluated it on Llama-3.1-8B using the MMLU benchmark. We included FLoRA (NeurIPS 2024) [1] and FlexLoRA (NeurIPS 2024) [2] as highly relevant SOTA baselines. The results are summarized below:

As shown, LEFF not only applies to SOTA models but excels, outperforming recent NeurIPS 2024 methods FLoRA and FlexLoRA by 1.8-2.3 points on MMLU while simultaneously reducing client-side peak memory by over 36%. These new results confirm that LEFF is highly competitive and its benefits are even more pronounced on larger models where client-side resources are the primary bottleneck. We will add these results to the appendix.

[1] FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations, NeurIPS 2024.

[2] Federated fine-tuning of large language models under heterogeneous language tasks and client resources, NeurIPS 2024.

(W.1) On Approximation Error and Model Accuracy

Regarding your concern (W.1) that layer compression introduces approximation errors that cap accuracy, we agree that managing this error is critical. LEFF is designed from the ground up to actively model and minimize this trade-off.

1. System-Wide Robustness: The "high-compression scenario" only applies to the most resource-constrained clients. As detailed in Sec. 3.1, LEFF is heterogeneous-aware. Higher-capacity clients perform updates with less (or no) compression, submitting high-fidelity gradients. During server-side aggregation, these high-quality updates compensate for the higher-error gradients from weaker clients, preventing the global model's accuracy from being capped by the worst-case devices. Our strong results in Table 2, where LEFF nearly matches full fine-tuning (FedAvg), empirically validate that this error is well-controlled.
2. Principled Error Minimization: LEFF is not a random compression scheme. The server tailors a model for each client by using importance-based sampling (Sec. 3.2) to select critical layers for full-parameter updates, while using knowledge distillation (Sec. 3.3) to create compact, function-preserving representations of the rest. This principled approach ensures that clients perform local training with high-fidelity gradients, directly minimizing the approximation error $\bar{\Delta}^2$ from our convergence analysis (Thm. 1).

(W.2) On the Role and Robustness of the Proxy Dataset

We thank the reviewer for raising the practical point (W.2) about the dependency on a proxy dataset. We have designed LEFF to be robust to this.

1. Robustness to Data Distribution: Our method is robust to the choice of proxy data because we distill functional representations, not task-specific knowledge. As described in Section 3.3, our objective is to match the intermediate hidden states and attention matrices. This process preserves general representational capabilities and is far less sensitive to the specific domain of the input data.
2. Practical Availability: Since our method only requires unlabeled text, readily available public corpora (e.g., C4, Wikipedia) are sufficient. In scenarios where external data is prohibited, the server can use the global model to generate a synthetic proxy dataset, making the framework entirely self-contained — a promising direction we are exploring.
3. New Ablation Study: We empirically demonstrate LEFF’s robustness to the proxy dataset via a new ablation study on the E2E NLG task. As shown in the table, performance remains remarkably stable across diverse corpora, from the in-domain WebNLG to general-purpose WikiText-103 and OpenWebText. While the in-domain data yields a slight, expected performance gain, the minimal fluctuation across all metrics confirms that LEFF is not reliant on a perfectly-matched proxy corpus. This high degree of robustness validates its practical applicability. We will include this study in the appendix.

(Q.1) Computational Cost of Layer Importance

In response to your question (Q.1), the computational cost of calculating layer importance is minimal and highly scalable.

The calculation uses a first-order Taylor approximation, requiring only a single forward and backward pass on the global model. This operation is performed once per round on the central server. Crucially, its cost is independent of the number of clients in the federated network. Therefore, the overhead remains constant and negligible as the system scales, posing no practical bottleneck.

Once again, we thank the reviewer for their valuable and detailed feedback. We believe these clarifications and new experiments, which will be integrated into the final version, substantially strengthen our paper's contribution and address all raised concerns.