Rising from Ashes: Generalized Federated Learning via Dynamic Parameter Reset

AQ1 regarding the related work：

Thank you for your suggestion. We will provide a more detailed introduction to the related work in the revised version.

AQ2:

Thank you for your suggestion. Adding comparisons regarding extremely heterogeneous data will further highlight the advantages of our method. The following table results show that our method still achieved the best outcome compared to the baseline.

Answer to Q3 (Analysis of Computational Overhead):

Sorry for the confusion. We conduct a quantitative analysis of the FLOPs (Floating-Point Operations) for each round of the Reset operation.

Notations

• $M$ denotes the total number of convolutional layer parameters;
• $K$ denotes the number of activated clients, $|S_l|$ denotes the total number of convolutional layers;
• $\theta \in (0,1)$ indicates the perturbation ratio, where $\lfloor \theta n_i \rfloor$ kernels are reset per layer (with $n_i$ kernels in layer $l_i$)
• For each layer $l_i$, $\mu_i = \mathbb{E}[W_{l_i}]$ and $\sigma_i^2 = \text{Var}(W_{l_i})$ are computed from the current layer parameters;
• The reset kernels are sampled from $\Delta_{l_i} \sim \mathcal{N}(\mu_i, \sigma_i^2)$.

FLOPs estimation:

• Computing mean $\mu_i$ requires ≈2 FLOPs per parameter (sum and divide).
• Calculating variance $\sigma_i^2$ requires ≈4 FLOPs per parameter (subtractions, squarings, sum, and divide).
• Sampling from $\mathcal{N}(\mu_i, \sigma_i^2)$ requires ≈10 FLOPs per parameter (including random number generation and transformation, e.g., via Box-Muller).
• Minor operations like random kernel selection (e.g., generating masks or indices) are $O(|S_l| \times \max(n_i))$ and negligible (<1% of total overhead).

Computation overhead of FedPhoneix

In each round, FedPhoenix requires $2M$ FLOPs to compute the mean and $4M$ FLOPs for sampling in the cloud server. For the generation of each local model, FedPhoenix reset $\theta M$ parameters, and each parameter requires 10 FLOPs. Therefore, FedPhoenix requires $K \times 10 \theta M$ FLOPs for local model generation. In addition, the model aggregation requires $KM$ FLOPs. Overall, in each round, FedPhoenix requires $(6 + (10 \theta+1)K)M$ FLOPs. Note that the computation overhead of the reset operation is similar to that of the aggregation operation, and all the reset operations are performed in the cloud server. Due to the powerful computing resources of cloud servers, the additional computation overhead of the reset operation is tolerable.

Additionally, we analyze the computational overhead of the baselines. For FedProx, assume that $P$ is the number of local update steps per client; its additional computation overhead is $3 P \times K \times M$ FLOPs. For FedMut, the additional computation overhead is $M(1 + 2K)$ FLOPs. For ClusteredSampling, its additional computation overhead is $3 K N M$ FLOPs, where $N$ is the total number of clients. Therefore, the additional computation overhead of FedPhoneix is similar to the baselines.

AQ4：

Thank you for your suggestion. We will adjust the page layout in the revised version.

I just carefully reviewed your responses to all the reviewers’ comments. My concerns have been largely addressed, and it seems the other reviewers’ issues have also been resolved. Please make sure the final version fills the full 9 pages. I will raise my score to 5.

Answer to Q1

Sorry for the confusion. Since the reasoning of deep layers in neural networks is based on the output from shallow layers, we prioritize reducing the reset rate of shallow layers to ensure training stability. From the perspective of feature levels, as established by seminal works such as Zeiler & Fergus (2014, "Visualizing and Understanding Convolutional Networks"), shallow layers learn general-purpose, low-level features (e.g., edges, textures) that stabilize quickly, and high-level features are extracted from these low-level features. Therefore, we first reduce the reset rate of shallow layers to stabilize the learned low-level features and facilitate the learning of high-level features based on stable low-level features.

Answer to Q2

Thanks for your insightful suggestion.

Our primary motivation for resetting the entire kernel is to treat it as a single feature extraction unit. By completely re-initializing it, we aim to force the model to explore entirely new feature patterns, thereby enhancing generalization and feature diversity, which is the core goal of FedPhoenix.

We conducted new ablation studies where we selectively reset only a portion of the parameters within a kernel based on their magnitude (top 1/3 and top 2/3). We conducted these experiments by keeping the kernel reset ratio ($\theta$) constant and by increasing it to maintain a constant total number of reset parameters. The experimental results are as follows:

The results show that compared with the new plan, the original plan performs better. This is mainly because partial resets disrupt the learned structure of a kernel, potentially creating gradient conflicts during training. The holistic reset of the entire unit appears to be more effective for our purpose.

Answer to Q3

Sorry for the confusion. From a geometric perspective, despite limited participation of each client, learning non-generalized features will still cause the model to be stuck in a local optimum, i.e., a sharp area. Please note that due to the similarities in the characteristics of different data, in this area, the local optimum of different clients will be close but different. As the training progresses, the optimization directions of different clients will conflict, making it difficult to reach a consensus and thus getting stuck in the local optimum. Parameter reset can enable the model to directly jump out of the local optimal area and find a new solution. Please note that the generalization area is relatively flat. Resetting part of the model's parameters is not enough to cause the aggregated model to jump out of this area.

Answer to Q4

Sorry for the confusion. The optimal perturbation ratio is determined by a delicate trade-off between promoting exploration and preserving the stability of learned representations, and this balance shifts with the degree of data heterogeneity. Increased data heterogeneity will lead to greater differences in local model optimization directions, making it challenging to reach a consensus on the optimal direction during training. In these scenarios, resetting more parameters will seriously increase the time overhead of FL.

Answer to Q5

Sorry for the confusion. We will unify the notations in the final version.

Answer to Q6

Thanks for your suggestion. We conduct a comprehensive ablation study to investigate the impact of various distributions on the performance of our proposed method. We expanded our experiments to include several widely recognized and statistically diverse distributions:

• Xavier/Glorot (Normal & Uniform): To validate the effectiveness of our reset mechanism with another mainstream initialization standard.
• Truncated Normal: To evaluate a more conservative reset strategy that enhances stability by limiting extreme sample values.
• Orthogonal: To test whether introducing "structured" randomness, which is known to preserve gradient norms, offers advantages over purely random sampling.
• Laplace: To explore the effect of a distribution with different statistical properties (more peaked and promoting sparsity).

For a fair comparison, all these distributions were implemented using the "adaptive" strategy, i.e., they were scaled using the mean and variance of the current layer's weights, consistent with our proposed Adp_Normal method. The experiments were conducted on the CIFAR-10 dataset and the ResNet-18 model.

Based on these results, we can find that the core strength of our method lies in the adaptive reset strategy itself, rather than the specific choice of distribution. The consistently high performance across diverse distributions (Normal, Uniform, Orthogonal, etc.) demonstrates the mechanism's robustness and low sensitivity. While the Orthogonal distribution shows a slight advantage in some cases, our proposed Adp_Normal is highly competitive across all scenarios. This confirms that adapting the perturbation to the layer's current statistics is the key factor for success, making our approach broadly effective.

Answer to Q7

Thanks for your suggestion. We conducted a new experiment on the Office-31 dataset, a standard benchmark for cross-domain scenarios, where the feature space of each client is quite different. We partitioned the dataset by its three distinct domains (Amazon, DSLR, Webcam), assigning each domain to a separate client. The experimental results are as follows:

As shown in the table, our method achieves a significant performance gain over the baselines. This demonstrates that our approach not only performs well but also excels in this challenging, feature-skewed environment. By mitigating client drift and encouraging the exploration of a shared feature space, our method facilitates more effective collaboration, leading to a more robust and generalized global model.

AQ1:Analysis of Server-Side Computational Overhead of the Parameter Reset Mechanism

Sorry for the confusion. We conducted a quantitative analysis of the FLOPs (Floating-Point Operations) for each round of the Reset operation.

Notations

• $M$ denotes the total number of convolutional layer parameters;
• $K$ denotes the number of activated clients, $|S_l|$ denotes the total number of convolutional layers;
• $\theta \in (0,1)$ indicates the perturbation ratio, where $\lfloor \theta n_i \rfloor$ kernels are reset per layer (with $n_i$ kernels in layer $l_i$)
• For each layer $l_i$, $\mu_i = \mathbb{E}[W_{l_i}]$ and $\sigma_i^2 = \text{Var}(W_{l_i})$ are computed from the current layer parameters;
• Reset kernels are sampled from $\Delta_{l_i} \sim \mathcal{N}(\mu_i, \sigma_i^2)$.

FLOPs estimation:

• Computing mean $\mu_i$ requires ≈2 FLOPs per parameter (sum and divide).
• Computing variance $\sigma_i^2$ requires ≈4 FLOPs per parameter (subtractions, squarings, sum, and divide).
• Sampling from $\mathcal{N}(\mu_i, \sigma_i^2)$ requires ≈10 FLOPs per parameter (including random number generation and transformation, e.g., via Box-Muller).
• Minor operations like random kernel selection (e.g., generating masks or indices) are $O(|S_l| \times \max(n_i))$ and negligible (<1% of total overhead).

Computation overhead of FedPhoneix

In each round, FedPhoenix requires $2M$ FLOPs to compute the mean and $4M$ FLOPs for sampling in the cloud server. For the generation of each local model, FedPhoenix reset $\theta M$ parameters, and each parameter requires 10 FLOPs. Therefore, FedPhoenix requires $K \times 10 \theta M$ FLOPs for local model generation. In addition, the model aggregation requires $KM$ FLOPs. Overall, in each round, FedPhoenix requires $(6 + (10 \theta+1)K)M$ FLOPs. Note that the computation overhead of the reset operation is similar to that of the aggregation operation, and all the reset operations are performed in the cloud server. Due to the powerful computing resources of cloud servers, the additional computation overhead of the reset operation is tolerable.

Additionally, we analyze the computational overhead of the baselines. For FedProx, assume that $P$ is the number of local update steps per client; its additional computation overhead is $3P \times K \times M$ FLOPs. For FedMut, the additional computation overhead is $M(1 + 2K)$ FLOPs. For ClusteredSampling, the additional computation overhead is $3 K N M$ $FLOPs, where$N$ is the total number of clients. Therefore, the additional computation overhead of FedPhoneix is similar to the baselines.

AQ: About writing optimization

Thank you for your rich writing experience. We will make adjustments to Figure 3 and Figure 4 in the revised version. And further optimize the presentation of the core concepts (such as Table 1). In the discussion section, we will refine the framework to make writing more coherent and organized.

Answer to Q1/W1 (Transformer)

This paper primarily focuses on computer vision-based tasks. As mentioned in the Limitation section of the paper, since the different structure of Transformer and CNN, the current version of FedPhoenix cannot adapt to Transformer-based models.

However, we sincerely thank the reviewer for this forward-thinking question. While much of the current research in Federated Learning has concentrated on CNNs for vision tasks—largely due to the proliferation of resource-constrained AIoT devices—we strongly agree that extending these privacy-preserving and collaborative learning paradigms to Large Language Models is a highly valuable research direction and a key focus for our future work.

Conceptual Feasibility: Applying FedPhoenix to Mitigate Transformer Drift

The core philosophy of FedPhoenix is to prevent the model from overfitting to local data by periodically resetting a portion of its learned features. These reset units, possessing high plasticity, are then compelled to rediscover more generalizable patterns. Furthermore, the FFNs in Transformers utilize fully-connected layers. As shown in our experimental results in AQ3, when a model has a robust feature extraction layer, resetting the fully-connected structures can also lead to performance improvements.

Rather than a full parameter reset, which could excessively disrupt the highly structured knowledge within a Transformer, we can adapt the "reset" operation into a more refined blending mechanism. For a selected parameter W, the update formula would be:

W_new = (1 - α) * W_old + α * W_sampled

Here, W_sampled is a new value sampled from a normal distribution N(μ, σ) based on layer-wise statistics, and α is a blending factor (analogous to the original parameter ε, but controlling the intensity of the blend rather than the proportion of reset units).

This approach does not cause complete "amnesia" but rather "nudges" the parameters away from a highly specialized state. It helps to loosen the tight coupling between specific attention patterns or word embeddings and the data distribution of a single client (e.g., a "developer" client might associate "Python" with the programming language, whereas a "zoologist" client might associate it with the reptile), encouraging the model to build robust representations that span across client domains.

Operational Guidelines: Target Modules and Reset Granularity:

• Self-Attention Mechanism (W_q, W_k, W_v, W_o matrices):
  • Target: The query, key, value, and output projection matrices are prime candidates, as they directly govern the learned relationships between tokens.
  • Granularity: We would apply the blending operation to entire rows or columns of these matrices. Conceptually, this is equivalent to perturbing the "functional role" of specific feature dimensions within the attention mechanism.
• Feed-Forward Networks (FFN):
  • Target: The two linear transformation layers within each Transformer block.
  • Granularity: Resetting the incoming weights for individual output neurons.

Anticipated Challenges and a Dynamic Attenuation Strategy

• Hyperparameter Sensitivity: The blending factor α and the proportion of parameters to be perturbed are critical hyperparameters that will require careful tuning to balance knowledge retention and exploration.
• The dynamic stabilization strategy would be adapted as follows: apply stronger/more frequent perturbations to the earlier Transformer blocks during the initial communication rounds. As training progresses, the focus of the perturbation would gradually shift to the later, more abstract blocks. This approach maintains the original design philosophy of first solidifying foundational features while preserving the plasticity of higher-level representations.

We thank the reviewer again for this suggestion and will incorporate this discussion into the "Future Work" section of our paper to broaden its scope.

Answer to Q2/W2 (Analysis of Computational Overhead):

Sorry for the confusion. We conducted a quantitative analysis of the FLOPs (Floating-Point Operations) for each round of the Reset operation.

Notations

• $M$ denotes the total number of convolutional layer parameters;
• $K$ denotes the number of activated clients, $|S_l|$ denotes the total number of convolutional layers;
• $\theta \in (0,1)$ indicates the perturbation ratio, where $\lfloor \theta n_i \rfloor$ kernels are reset per layer (with $n_i$ kernels in layer $l_i$)
• For each layer $l_i$, $\mu_i = \mathbb{E}[W_{l_i}]$ and $\sigma_i^2 = \text{Var}(W_{l_i})$ are computed from the current layer parameters;
• Reset kernels are sampled from $\Delta_{l_i} \sim \mathcal{N}(\mu_i, \sigma_i^2)$.

FLOPs estimation:

• Computing mean $\mu_i$ requires ≈2 FLOPs per parameter (sum and divide).
• Calculating variance $\sigma_i^2$ requires ≈4 FLOPs per parameter (subtractions, squarings, sum, and divide).
• Sampling from $\mathcal{N}(\mu_i, \sigma_i^2)$ requires ≈10 FLOPs per parameter (including random number generation and transformation, e.g., via Box-Muller).
• Minor operations such as random kernel selection (e.g., generating masks or indices) have a time complexity of $O(|S_l| \times \max(n_i))$, which is negligible (<1% of total overhead) in practice.

Computation overhead of FedPhoneix

In each round, FedPhoenix requires $2M$ FLOPs to compute the mean and $4M$ FLOPs for sampling in the cloud server. For the generation of each local model, FedPhoenix reset $\theta M$ parameters, and each parameter requires 10 FLOPs. Therefore, FedPhoenix requires $K \times 10 \theta M$ FLOPs for local model generation. In addition, the model aggregation requires $KM$ FLOPs. Overall, in each round, FedPhoenix requires $(6 + (10 \theta+1)K)M$ FLOPs. Note that the computation overhead of the reset operation is similar to that of the aggregation operation, and all the reset operations are performed in the cloud server. Due to the powerful computing resources of cloud servers, the additional computation overhead of the reset operation is tolerable.

Additionally, we analyze the computational overhead of the baselines. For FedProx, assume that $P$ is the number of local update steps per client; its additional computation overhead is $3 P \times K \times M$ FLOPs. For FedMut, the additional computation overhead is $M(1 + 2K)$ FLOPs. For ClusteredSampling, the additional computation overhead is $3 K N M$ $FLOPs, where$N$ is the total number of clients. Therefore, the additional computation overhead of FedPhoneix is similar to the baselines.

Answer to Q3/W3 (Reset for full-connected layers):

Sorry for the confusion. As mentioned in the motivation of our paper, we aim to adopt parameter reset to guide the model to learn more generalized features. Typically, since CNN-based models employ convolutional layers to construct feature extractors, we prefer to reset the parameters of these convolutional layers. Note that, based on the inefficient feature extractor, it is difficult to improve the model performance by simply resetting the fully connected layer (i.e., the classifier). We conducted experiments on Cifar-10 using the VGG model to evaluate the effectiveness of resetting fully connected (FC) layers. The experimental results are as follows:

We can observe that only resetting FC cannot effectively improve the inference accuracy, but resetting both the convolutional layers and the fully connected layers can achieve the best performance.

Answer to Q4/W4 (Pravicy):

Thanks for your suggestion. Note that the primary objective of FedPhoenix is to enhance the generalization and robustness of the model, which cannot provide the formal (ε, δ)-guarantee in DP. Since FedPhoenix resets part of the parameters, attackers must make more effort to apply model inversion attacks. In addition, since the parameter reset is based solely on the aggregated global model, FedPhoneix supports secure aggregation. Furthermore, enabling us to handle the model reset operation at the client end will result in better protection against malicious server model reverse attacks. This is because each activated client will have a different training starting point. From the analysis of the computational cost of AQ2, it can also be seen that the cost of our Reset operation is relatively small, and each client can fully bear the cost of resetting their model.