Clients Collaborate: Flexible Differentially Private Federated Learning with Guaranteed Improvement of Utility-Privacy Trade-off

We sincerely appreciate the reviewer’s positive feedback on our theoretical and empirical contributions. Below are our point-to-point responses to the raised concerns and suggestions.

W1: Lack of Depth in Technical Details: The detailed discussions on parameter selection are somewhat brief in the main text. More technical details and discussions on parameter tuning could further enhance the reproducibility and practicality of the paper.

The reviewer noted brevity in parameter selection discussions. We have detailed hyperparameter search ranges and final configurations in Appendix C.1 (Table 2). For example, on CIFAR-10 with σ_g=1.5, we set λ=10, ϑ=1.07, and I=10. These parameters are selected via grid search and adaptively adjusted based on privacy requirements (e.g., smaller λ for stricter privacy). Please refer to the response to the reviewer RVpv (Experimental Designs/Analyses) for additional experiments.

W2 & Q2: Unclear Descriptions of Concepts: Some concepts or keywords in the paper, such as "semantic collaboration" and "global semantic space smoothing," could be described more clearly and in greater detail. More precise definitions and explanations would help readers better understand these concepts.

• Global Semantic Space Smoothing: Refers to the low-rank representation of client parameters in the spectral space after tensor decomposition, where high-frequency components are truncated to preserve semantic correlations

• Semantic Collaboration: Achieved by integrating complementary semantic information across clients via low-rank optimization (Section 3.2). Please refer to the responses to Reviewer kpz4 (Q2) for specific examples.

W3: Breadth of Comparative Experiments: Although the paper compares with some existing methods, the scope of comparative experiments could be expanded to include more recent federated learning and differential privacy methods. This would provide a more comprehensive demonstration of FedCEO's advantages.

We appreciate your suggestion and will incorporate additional baseline comparisons in the revised manuscript to provide a more comprehensive validation of our findings.

Q1: Could the authors further explain why FedCEO is referred to as a "flexible" DPFL framework, as indicated in the title?

As illustrated in Figure 2, our FedCEO can flexibly adapt to different privacy settings and noise accumulation during continuous training by setting different initial values and employing an adaptive thresholding rule based on a geometric series. The specific advantages of this "flexibility" are demonstrated in the experimental results in Table 1.

Q3: How exactly is the non-iid data distribution set up in Table 5?

Apologies for the lack of details on the degree of non-iid data. For non-iid data, we used a common method to construct data heterogeneity.

Taking CIFAR-10 as an example:

• First, we sorted all images in the dataset by their labels.

• Then, for each client, we randomly selected 250 consecutive images twice. Since CIFAR-10 has ten categories, each client has at least one category and at most four categories, verifying that our framework works even in highly heterogeneous situations.

We sincerely appreciate the reviewer’s constructive feedback and recognition of our work. Below are our point-by-point responses:

W1 & Comments1, 2

Condensing Background Knowledge: We agree with the suggestion. In the revised manuscript, we will streamline Section 2 (Preliminaries) by moving detailed definitions to the appendix, allowing greater emphasis on the core methodology (e.g., the optimization objective of FedCEO in Section 3 and the motivation for low-rank modeling).

Abstract Experiment Description: Thank you for the correction. We will update the abstract to include results on text datasets (e.g., Sentiment140): "...with experiments on representative image and text datasets…".

W2 & Comments3

We will add a discussion of 'Provable Mutual Benefits from Federated Learning in Privacy-Sensitive Domains' in the Related Work (Section 1.1). This work proposes a theoretical framework for designing personalized privacy-preserving protocols that provably benefit all participants in privacy-sensitive domains. Compared to our approach, it does not explicitly provide quantitative utility-privacy bounds but focuses on existence proofs. Both works involve inter-client collaboration, representing two complementary approaches: protocol design controlling client-side noise versus post-processing algorithms correcting inter-client semantics.

Q1

"Semantic complementarity" refers to the fact that local DP noise disrupts semantic information differently across clients. In DPFL, due to the randomness of noise introduced by each client, specific semantic information might be disrupted in some clients while remaining relatively intact in others, leading to semantic complementarity among different clients. By integrating the commonalities of parameters via tensor low-rank optimization, the server recovers global semantic smoothness. For example, in Figure 1 (CIFAR-10), the semantic space of the 10th class is corrupted (red row), but FedCEO leverages intact semantic features from other clients (e.g., client 7, 9) to restore smoothness (blue row), improving classification accuracy.

Q2

Consider a DPFL framework with only two clients performing animal image classification. This collaboration mechanism enables semantic complementarity between the two clients' models, thereby improving the global model's performance in DPFL.

For one client, the noise might severely corrupt parameters responsible for recognizing facial features, while for the other client, the noise might primarily distort parameters for limb recognition. However, by performing a Fourier transform, each slice will contain information from all clients. Subsequently, applying truncated SVD to all slices can be viewed as projecting the perturbed parameters back into a smooth semantic space, leveraging knowledge from all clients.

Finally, after performing the inverse Fourier transform, each slice (representing a client's parameters) will have adaptively incorporated information from other clients while retaining some of its original knowledge.

Q3

T-tSVD is chosen for its efficiency in dynamic thresholding via Fourier-domain matrix factorization (Table 4). Future work may explore other methods (e.g.,deep tensor learning), provided they align with privacy constraints. Existing attempts can be seen in our response to Reviewer RVpv (W3 & Other Comments/Suggestions).

Q4

Equation (2) is not a denoising process, as we do not utilize any prior information about the DP noise in our modeling. Essentially, it is a controlled fusion operation, and Proposition 3.2 demonstrates that FedAvg is in fact a special case of our FedCEO method—yet FedAvg is not considered to possess denoising capabilities. Furthermore, both Theorem 4.5 and the experiments in Section 5.3 confirm that the server-side operations in Algorithm 2 do not compromise the privacy guarantees of the DPFL framework.

We sincerely appreciate your constructive feedback and the opportunity to improve our paper. Below are our point-by-point responses to your comments:

W1

Figure 4 in the paper visualizes the robustness of our method to different noise levels on CIFAR-10. Compared to other methods, our FedCEO (orange curve) exhibits less utility degradation as noise (privacy) intensity increases, demonstrating stronger utility-privacy trade-off and robustness to privacy noise (as reflected by the minimal slope of the orange curve, second only to non-private FedAvg (blue curve)). In the revised version, we plan to include visualizations on additional datasets to further strengthen this conclusion.

W2

In addition to EMNIST and CIFAR-10 used in the main text, we have also evaluated FedCEO on text data (Sent140) with LSTM (see Appendix C2.3 Table 6), where our method consistently demonstrates superior performance.

W3 & Other Comments/Suggestions

FedCEO is a novel federated learning parameter update framework that is architecture-agnostic (operating on parameter tensors) and compatible with deep architectures (e.g., AlexNet, see Table 6). Its key advantage lies in efficiency—it avoids the high computational costs of deep personalized methods like PPSGD and CENTAUR (see efficiency experiments in Appendix C2.1 Table 4).

Furthermore, we have integrated the deep tensor method CoSTCo [1] (modified for low-rank approximation task) into our framework. Experimental results show that our approach (via T-tSVD) outperforms CoSTCo on CIFAR-10, likely because CoSTCo is more suited for sparse tensors and less compatible with noisy parameter tensors. Future work will explore deep tensor methods better aligned with FedCEO.

[1] CoSTCo: A Neural Tensor Completion Model for Sparse Tensors (KDD 2019).

Experimental Designs/Analyses

The initialization coefficient λ and geometric ratio ϑ are designed to adaptively adjust the truncation threshold $\frac{1}{2\lambda} \vartheta^{\frac{t}{I}}$ based on noise levels (Sec. 3.2). For stronger privacy guarantees (larger σg), smaller λ enhances smoothness (see Table 2), while ϑ > 1 accounts for accumulating DP noise during training. Detailed parameter selection guidelines are provided in Appendix C.1.

Beyond the robustness analysis of parameter I in Table 3, we have added the following analysis for ϑ:

It can be observed that when ϑ > 1, the model performance shows significant improvement compared to ϑ = 1, while demonstrating strong robustness to this parameter when ϑ > 1. We will also include a visual analysis of mixed λ-ϑ robustness in the revised manuscript.

Relation to Broader Literature

FedCEO introduces a client-collaborative tensor low-rank optimization framework for global model updates in DPFL, explicitly leveraging inter-client semantic complementarity in spectral space. Unlike prior spectral methods (e.g., CENTAUR [Shen et al., 2023] and [Jain et al., 2021]) that apply singular value decomposition (SVD) independently to client matrices, FedCEO stacks client parameters into a higher-order tensor and performs truncated tensor-SVD (T-tSVD). This enables adaptive truncation of high-frequency components across clients while preserving low-rank structures, thereby improving global model utility through inter-client information fusion.

Thank you for your valuable feedback and for taking the time to provide thorough reviews! Below are our point-by-point responses to your comments:

Claims And Evidence (Concern 1)

We appreciate the reviewer's suggestion.

• The product form was chosen because it directly reflects the joint tightness of the utility-privacy boundary. This form highlights the intrinsic trade-off: improving one objective often requires relaxing the other.
• Secondly, we maintain the same joint objective formulation ($\epsilon_u \cdot \epsilon_p$) as the prior SOTA method CENTAUR [1], enabling direct theoretical bound comparisons in our paper. Moreover, as shown in Theorems 4.3, 4.5, and Corollary 4.6, the product form eliminates the influence of the stochastic factor K.
• Compared to weighted averaging [2] or Pareto optimization [3], the product form is more concise and interpretable. We will add relevant discussions and citations in the revised version.
  • Product vs. Weighted Average: Weighted averaging (e.g., $\alpha \epsilon_u + (1-\alpha)\epsilon_{p}$) requires manual weight selection ($\alpha$), which may introduce subjectivity. The product form avoids weights and directly captures the coupling between objectives, making it more suitable for theoretical analysis. Additionally, weighted averaging is more susceptible to scale differences.
  • Product vs. Pareto Optimization: Pareto requires simultaneous optimization of both objectives, but in practice, privacy and utility are often conflicting. The product form allows controlled compromise on one objective to achieve significant gains in the other, better aligning with practical needs.

[1] "Share Your Representation Only: Guaranteed Improvement of the Privacy-Utility Tradeoff in Federated Learning" (ICLR2023)

[2] "No free lunch theorem for security and utility in federated learning" (TIST2022)

[3] "Optimizing privacy, utility, and efficiency in a constrained multi-objective federated learning framework" (TIST2024)

Claims And Evidence (Concern 2)

For all compared methods in experiments, we adopted the same DP mechanism (user-level DP) and unified implementation (based on Opacus). Thus, their privacy guarantees are identical (i.e., same $\epsilon_{p}$). In the revision, we will supplement the quantitative relationship between $\sigma_g$ and $\epsilon_{p}$ (e.g., adding an $\epsilon_{p}$ column in Table 1) based on Theorem 4.5, demonstrating our method's superior utility under identical $(\epsilon_{p}, \delta)$.

Claims And Evidence (Concern 3)

Lemma 4.4 provides the privacy guarantee for the baseline algorithm UDP-FedAvg, while Theorem 4.5 proves that FedCEO's low-rank optimization (a deterministic operation without noise-dependent priors) preserves privacy (post-processing immunity). Thus, their bounds align. We will explicitly emphasize this in the revision.

Experimental Designs Or Analyses

Thank you for the suggestion. We will add standard deviations in the revision (e.g., FedCEO 78.05%±0.2 vs. CENTAUR 77.26%±0.3 for EMNIST at $\sigma_g=1.0$), confirming statistically significant improvements.

Others

We will correct the usage of return in Algorithm 2 (e.g., replacing it with yield or annotations) and move key assumptions to the main text in future versions.