GradPS: Resolving Futile Neurons in Parameter Sharing Network for Multi-Agent Reinforcement Learning

Thanks for your valuable comments and suggestions, your comments are very important to us. We will improve our work based on your suggestions. We address your concerns as follows.

Methods:

1. ... visualize the group assignment pattern...

Thank you for your suggestion. It is very valuable, and we will include it in the revised manuscript. The following table shows an example of the grouping of a neuron in the 5m_vs_6m environment. Before 0.5M steps, the neuron is not futile. At 0.6M timesteps, the neuron is futile, and the agents for the neuron were grouped into (1,2,3) and (4,5). The grouping is restored at 1.2M timesteps. At 1.6M timesteps, the agents were grouped into (1,3) and (2,4,5).

2. ...report the training time and memory usage.

Appendix D.6 shows the number of parameters (including factorization networks) used by different methods. We provide more results as follows.

We present the wall-clock training time (in hours) for each algorithm, averaged over 5 random seeds. The computational overhead of GradPS is small. It takes less training time than SNP, SePS and NoPS, most of the time.

The following table depicts the number of parameters for each agent network compared to the full parameter-sharing approach. The additional memory overhead of GradPS is lower than that of SePS and NoPS.

3. ...$\alpha$ sensitivity analysis...

$\alpha$ controls how futile neurons are identified. We have performed a sensitivity analysis for three environments; their results are depicted as follows.

The results show setting $\alpha=0.2$ is a good choice. We agree that such a threshold may be environment-dependent, and we would like to explore it in the future.

Questions:

Method:

...How gradient conflict retained..., introduce significant overhead...

The accumulated gradients are stored in a tensor (T $\times$ agents $\times$ dim), representing the historical gradient values of neurons across each agent over the past T periods. These saved gradients are used to compute neuron efficiency, not for back-propagation. It introduces little computational overhead. Moreover, we have shown in the above parameter table that the memory overhead of GradPS is low.

Experiment:

1. ...higher futile neuron percentages do not seem to correspond to lower performance. ...

We appreciate your insight. MARL performance is influenced by multiple factors, and futile neuron is just one of them. We agree that higher neuron percentages do not necessarily lead to lower performance.

2. ... a cost comparison...

We have already shown the execution time and memory overhead in the above table.

3. ...relationship between the clone ratio and the heterogeneity..

The cloning ratio $K$ is not directly correlated with agent heterogeneity. $K$ is a neuron-level value, it is not an agent-level value. We will explore whether the aggregate $K$ of all the neurons can represent agent heterogeneity.

4. ...$\rho$ does not impact performance, though it seems to tie to the computation overhead...always set a large $\rho$

$\rho$ is the probability of restoring the parameters of a neuron from grouped to shared status. We have conducted experiments that set $\rho$ to a large value (from 0.05 to 0.5), and we find that increased $\rho$ does not impact the computational overhead significantly.

Gradual parameter recovery requires sufficient time to prevent rapid performance degradation; thus, a large $\rho$ may not be the optimal choice, as shown in the following table.

Thanks for viewing our work as interesting. Thanks for your valuable comments regarding the novelty of the futile neuron phenomenon and neuron gradient conflicts in MARL. We will improve our work based on your suggestions. We address your concerns as follows.

Experimental:

1. How do you determine the futile threshold?

We perform a simple parameter sensitivity experiment to determine the futile threshold $\alpha$. The experimental results for SMAC (5m_vs_6m and MMM2) and Predator-Prey are shown as follows.

The sensitivity experiment shows that the performance for $\alpha=0.2$ is good. Thus, we fix the futile threshold of all the experiments as 0.2. It is possible that such a threshold should be flexible for different environments. Moreover, better thresholds, such as deviation from the average futile ratio, could be used. We want to explore these in the future.

2. ...the computational overhead of this method...

In the appendix, the execution time (wall-clock training time) is shown as a ratio over the FuPS approach. The experimental results show that the additional computational overhead brought by GradPS is small; it is only slightly higher than FuPS. The other methods (e.g., SePS, NoPS) require significant computational overhead.

In the following table, we present the wall-clock training time (in hours) for each algorithm, averaged over five random seeds. GradPS achieves lower computational overhead than other parameter-sharing methods (SNP, NoPS).

Weaknesses:

1.Analysis of the futile neuron threshold $\alpha$ is missing.

We have discussed $\alpha$ in the above response.

2. The algorithm lacks sufficient theoretical analysis.

We employ a simple yet effective method to mitigate gradient conflicts while encouraging future research to develop more solutions to gradient conflict issues in MARL. We will perform an in-depth theoretical analysis of GradPS and other parameter-sharing methods.

Suggestions

1. Under what circumstances does gradient conflict usually occur? If agents share a common goal...

Thanks for raising this question; we will discuss it in the discussion section. Gradient conflicts commonly occur in multi-agent systems. Even if agents share a common goal, they may have different partial observations, be located in different positions of the game, and their actions may be different, which may lead to different agent gradients (e.g., divergent actions). For example, when capturing one prey, it is possible that the best action for agent 1 is to move up, while the best action for agent 2 is to move down. The optimal actions for these two agents are different; this will cause gradient conflict, too.

To verify the above analysis, we train two predators to capture one prey in a 5 $\times$ 5 map, and the spawn point of the agents and the prey is fixed. At each time, there is only one prey. Experiments show that the proportion of futile neurons is about 20%, which suggests that there are gradient conflicts, even if the two agents are pursuing the same prey.

2...situations where this approach is not effective...

In a highly stochastic environment, SMACv2, where agent types are randomized each episode, GradPS fails to deliver performance improvements as the gradient conflict patterns vary due to the randomness of agents themselves.

Thanks for your valuable comments, we address your concerns as follows.

Claims:

1. .."neuron-based"..

GradPS is a neuron-based method from the angle of gradient conflict. We agree that pruning-based PS can be viewed as neuron-based too.

2. ..a momentary gradient direction..

We agree that gradients indicate temporary improvements. We will soften the claim regarding the direct connections between gradient and diversity/skill.

3. ..neuron efficiency decreases

In Dec-POMDPs, agents with identical tasks may take different actions due to partial observations, leading to gradient conflicts from divergent optimal actions.

In Figure 3 (Left), these agents share the same goal (removing all enemies) and with the same type. Neuron efficiency decreases with more agents due to higher partial observation diversity. Please refer to our response to Suggestion 1 of Reviewer MPio for experiments regarding agents with identical task.

Methods:

1. ..the functional heterogeneity of agents..

The relationship between gradient conflicts and functional differences is worth exploring. Even agents with an identical task can lead to gradient conflicts due to partial observation issues, as analyzed above. Thus, analyzing the learning efficiency of MARL for gradient conflict is needed.

We do not link gradient conflicts with role (functional) differences. However, according to our experiments (Fig. 10), our approach is able to discover hidden roles.

2. .. more hyperparameters..hinders its practical application.

Our work is practical, requiring (only 4) no more hyperparameters than existing grouping methods without the need of pre-training (unlike SePS/AdaPS). The table below compares hyperparameters across PS/grouping methods.

Theoretical Claims:

...formulas and definitions...

We will fix the notations.

Experimental:

1. ..compare with other grouping methods.. AdaPS, ADMN [1], or MADPS [2]

ADMN and MADPS are not open-source. Reproduction attempts at such a short time would risk introducing unfair biases. GradPS performs better than AdaPS as shown in the following table.

2. ..compared with other restoration methods..

We compare two restoration methods, which use the average value of the group or the value of a random clone to directly replace the neuron weights. Our method is better.

Essential References:

..some diversity-based MARL works..

Thanks, we will discuss all of them in related work.

Weaknesses:

.. computational cost and practicality.. effectiveness in deeper networks.

We have shown in Appendix Table 10 and 11 that the increased overhead is small. It is much smaller than partial PS methods (e.g., SePS and NoPS).

Our work is practical for multi-layer agents. In Appendix D.5.2, for a 4-linear-layer agent, there are more futile neurons in the first linear layer than the other layers. Although there are more layers, such a method (QMIX-4-layer) performs weaker than standard QMIX, due to futile neurons. Applying GradPS in the first linear layer leads to higher performance gain than applying it to the other layers, justifying our first-layer focus.

Moreover, we show that GradPS works in different agent networks (distributional or risk-sensitive) in Appendix D.4. Figure 7.

Suggestions:

..gradient-conflicted neurons and MARL elements..

It is necessary to study the fundamental connection between futile neurons and essential MARL elements following ADMN and MADPS.

Questions:

1. ..gradient be a tensor..the overall knowledge..

Neuron gradient is not a vector (tensor) gradient, as defined in Definition 1. The neuron gradient is the gradient with respect to the output for a neuron, which is just a scalar (positive or negative). A holistic view of all neurons can better represent the whole knowledge.

2. ..the precise meaning..

We will soften the claim regarding the direct relationship between gradient conflicts and policy diversity.

3. ..gradients depend not only on network input/output structure..

Gradient conflicts correlate with diverse agent observations/actions and are influenced by factors such as loss functions and environmental stochasticity.

Claims:

1. ...convergence plots are missing, ...More explicit reproducibility analysis...

All experiments were repeated with 5 different random seeds to ensure reproducibility, with means and variances shown in all experimental result figures of this work.

Convergence of GradPS are evaluated by running GradPS in three environments for 3M instead of 2M steps. The results are depicted as follows. After 2M steps, for MMM2, the performance of GradPS continues to increase; for Predator-Prey and 5m_vs_6m, their performance is stable.

Methods:

1. ...whether baseline methods are implemented optimally...

For fairness, all parameter-sharing (PS) methods (e.g., FuPS and SNP) were implemented in the same repositories (e.g., pymarl). PS implementation for pymarl (located in rnn_agent.py) is the same as pymarl2 and pymarl3. From this point, pymarl2 and pymarl3 are not optimized pymarl. Moreover, we also evaluate distributional and risk-sensitive agents, where are not included in pymarl2&3.

We show in the following table that for pymarl3 QMIX experiments, GradPS outperformed FuPS+ID and FuPS. This shows that our method works in pymarl3.

Experimental:

2 ...the grouping mechanism and sensitivity to hyperparameters...

We have described the grouping method in Section 5.1 and provided more details in Appendix C. We describe the hyperparameters and present more results as follows.

$T$ is the period for identifying futile neurons. An excessively large T may delay futile neuron detection.

$K$ denotes the number of groups. Setting K too large will lead to low sample efficiency.

$\rho$ indicates the recovery probability for a group. In short-term tasks, even $\rho$=0 has a negligible effect on performance.

Sensitivity analysis for $T$, $K$, and $\rho$ is shown in the left, middle, and right of Figure 10 in Appendix D.5.3, respectively.

The $\alpha$ determines futile neuron identification. Our sensitivity analysis (see table) shows $\alpha$=0.2 yields promising results.

Weaknesses:

1. Section 5.2 and 5.3 is hard to read.

We will revise them to increase readability.

Method:

1. ...seems a kind of dormant neuron...

Dormant neurons have low normalized activation scores, while futile neurons suffer from gradient conflicts unrelated to activation. A non-dormant neuron can be a futile neuron.

We have listed the differences as follows.

2. ...the grouping method is not mentioned in the main paper...

We have described the grouping method in Sec 5.1 of the main paper.

Experiment:

2. ...Lack of comparison with naive baselines...

We compared with SOTA partial PS methods: SNP, which prunes a shared network for individual policies, and Kaleidoscope, which learns agent-specific masks. Our method outperforms both, as shown in experiments.

Following the reviewer's suggestion, we also compared with Naive LastPS (sharing all but the last layer), where our approach demonstrates significantly better performance, as it is depicted in following table.

Suggestions:

Put more details in the method section.

We will add more details to improve its readability.

Questions:

2. Is the introduction of "futile" necessary?

In PS networks, gradient conflicts produce "futile neurons" - neurons hindered by conflicting updates. As detailed in our comparison table, these are systematically identifiable through their distinct characteristics.

3. why...the execution time of GradPS increase..., is the same as the fullps baseline...

In GradPS, the number of parameters per agent differs from that in FuPS, as demonstrated in Appendix D.6. GradPS requires additional parameters for cloning neurons and storing gradients. For computational overhead, GradPS requires about 1.1$\times$ the training time of FuPS; it requires significantly less (40-50%) time than SNP and NoPS.