Episodic Control-Based Adversarial Policy Learning in Two-player Competitive Games

Weakness:

8.The conclusion that the agent is able to learn multiple winning strategies and learn adversarial and non-adversarial strategies are all based on a single anecdotal observation. The conclusions would be more convincing if the authors could provide more anecdotal examples

Thanks for pointing out this point. This is a subjective conclusion we made after we watched the videos of the policies in YouShallNotPassHumans. We observed that the actions of our policy in YouShallNotPassHumans combine the lying down on the ground actions (adversarial actions) and the obstacling the victim actions (non-adversarial actions) to fail the victim. Although this kind of policy is only observed in YouShallNotPassHumans, we think it still indicates that our method may have the ability to enable the adversarial agent to learn multiple winning strategies.

We will further clarify this in the paper, for example, we will change

“Fourth, our work demonstrates that by identifying and highlighting the winning states with historical experiences, adversarial agents can achieve higher winning rates against fixed victim agents and possess the capability to integrate multiple winning strategies to defeat the victims.”

in the introduction to

“Fourth, our work demonstrates that by identifying and highlighting the winning states with historical experiences, adversarial agents can achieve higher winning rates against fixed victim agents and may have the potential to possess the capability to integrate multiple winning strategies to defeat the victim, which is demonstrated in one Mujoco game environment in our experiment. “

9.Terms like "non-loss rate" should be explained briefly for better clarity and completeness

“non-loss rate” is just the rate of the games which do not lose. It is equal to the winning rate plus tie rate.

10.In general, I find sections 4.3.1 and 4.3.2 a bit too brief to truly understand what the authors are comparing. For example, in 4.3.1, what is the difference between using the historical scores vs 3-step patterns? Could the authors elaborate more?

Section 4.3.1 shows the ablation study of states and patterns which are introduced in Section 3.2.1. Section 4.3.2 shows the ablation study of historical score and episodic feedback, as episodic feedback serves as an optimization of historical score. In Section 4.3.1, we use the words as “we calculate the episodic feedback with historical scores of both states and 3-step patterns and then revise the rewards with two episodic feedbacks.” So the comparison is based on using the historical score of state and the historical score of 3-step pattern, but not the historical score vs 3-step pattern.

We hope our responses and clarifications have addressed your key questions and concerns. If so, we kindly request that you consider revising your score to better reflect this. If there are any remaining questions, please let us know, and we will do our best to resolve them.

Weakness:

4.In Equation 3, explaining what the "otherwise" scenario entails would make the paper clearer.

The adversarial agent can end the game in three states, win, tie and lose, and there are also three types of the episodic feedback, greater than 0, less than 0 and equal to 0, totally 9 possible scenarios. The otherwise just refers to the rest 7 scenarios except for “the adversarial agent wins and episodic feedback is less than 0” and “the adversarial agent loses and episodic feedback is greater than 0”. Since these conditions are too long to be written into the equation, we use otherwise to replace them.

5.In Section 3.2.2, the authors used a moving average of recent rewards to compute the feedback. What happens the training destabilizes (which is common for on-policy) and the moving average decreases? Would the feedback still be helpful and is the formulation designed to handle such scenarios or is the assumption that the moving average rewards always increases needed?

Since the reward revision in our method is cumulative, the decrease of moving average (in two-player competitive game situations, the amount of losing episodes increases) may not immediately harm the training process. On the other hand, the states in winning episodes from the group with low average may get vastly encouraged compared with those in the winning episodes from the group with high average.

However, we are not sure if the feedback will still be helpful when the training destabilizes. We didn't encounter such examples during our experiments thus we can not give the conclusion.

7.From what I understand, computing and updating the episodic memory every episode increases the computational complexity considerably of this method. Given this increased in complexity, a comparison of the wall-clock running time would be helpful, and would this method work if the memory updates are performed less frequently?

We agree that computing and updating the episodic memory every episode increases the computational complexity and time cost. We have considered this aspect so we use a relatively simple architecture (LSTM+MLP) to build the episodic memory. The experiments also show that such episodic memory is sufficient for Mujuco two-player competitive games. We will also add the time difference between our method and PPO in our experiments in the appendix as following:

YouShallNotPassHumans: 10 hours -> 12 hours

KickandDefned: 22 hours ->24 hours

SumoAnts: 30 hours -> 33 hours

SumoHumans: 24 hours -> 26 hours

Thanks for your invaluable and enlightening review.

Weakness:

1.In the related works section, the authors wrote: "However, the above attacks are argued to be unrealistic since the real-world environment can not be manipulated". While the authors referred to this statement by referencing other works, I believe this statement is slightly misrepresented. In reality, adversarial attacks on the environment via perturbing the observation/action channels are definitely realistic. Furthermore, the proposed approach of two player games is also a form of indirect environmental manipulation, which ultimately affects the observation space. Hence, this statement seems rather contradictory to me given the topic of the paper.

Since RL has been applied in settings like autonomous driving, negotiation and automated trading; in domains such as these, an attacker cannot usually have access to the victim policy’s input. However, adversarial training treats the victim policy as a black box and utilizes natural actions to induce the failure of the victim. Such actions reach the same effect as other adversarial methods (e.g., observation manipulation), but do not need any direct access or manipulation to the victim policy, thus may still work for the real-world situations that the adversary can not access to the victim policy.

2.As mentioned above, the idea using good trajectories from the past to augment the training process to improve efficiency is not new. I would suggest the authors cite more related papers to create a more comprehensive overview. For example, how does the idea proposed in this paper differ from those of hindsight experience replay, prioritized experience replay, etc?

Prioritized experience replay assigns a priority to each transition in the replay buffer, ensuring that more significant or impactful transitions are sampled more frequently. Different from this method, our method first identifies good states from the historical evaluations and then directly modify the environmental rewards given from the environment, improving the training by assigning higher rewards to the good states so that the good states are more likely to be learned and selected.

For Hindsight experience replay, we both modify the rewards but in different ways and for different goals. HER sets an alternative goal for the state transitions of an episode to create a learning signal from failed attempts, while we identify good states from historical evaluations and encourage the training process to learn and select the good states. HER modify the rewards based on whether the alternative goal is reached, for example, it may assign the reward as 0 if the goal is achieved, but we revise the rewards conditionally with the historical evaluations which can evaluate the quality of the states and may contain more information.

As we stated in Section 2.2, we implement our method using episodic control. This kind of method contains an extra episodic memory to save and analyze historical experiences, which aligns with our insight. Different from previous episodic control methods, we focus on the win-lose information from the past episodes and incorporate such information into the rewards from the environment via conditional reward revision. We further implemented the SOTA episodic control method (Li et.al.) [1] with PPO and compared its performance to our method in the YouShallNotPassHumans and KickandDefend environments. The experiment results are shared in the same link we show the videos of our experiments in the appendix. In YouShallNotPassHumans, our method outperforms [1]. In KickandDefend, [1] even harms the training method. This indicates that existing episodic control method may not be directly applicable to two-player competitive games.

3."The reward R′α obtained by the adversarial agent only includes the evaluations from a single game and does not contain evaluations from the historical experiences.". Does this imply that existing works on adversarial policy thus far are all on-policy?

"The reward R′α obtained by the adversarial agent only includes the evaluations from a single game and does not contain evaluations from the historical experiences." does not imply that existing works on adversarial policy thus far are all on-policy. As stated in Section 3.1, the “reward R′α” is the environmental rewards the adversarial agent received from the environment, which can be also stored and used in off-policy methods. This reward only contains the information for evaluating a single state transition in one game and what we want to do is to directly incorporate historical evaluations into such rewards for the adversarial agent to learn.

[1] Zhuo Li, Derui Zhu, Yujing Hu, Xiaofei Xie, Lei Ma, Yan Zheng, Yan Song, Yingfeng Chen, and Jianjun Zhao. Neural episodic control with state abstraction. arXiv preprint arXiv:2301.11490, 2023.

Thanks for your invaluable and enlightening review.

Weakness:

1.The proposed method lacks a theoretical guarantee and relies primarily on empirical results. The approach essentially applies reward adjustments by assigning intermediate rewards to specific states, which resembles a form of reward manipulation. It would be better if the author could give some bounds analysis regarding their propose method.

Our work does come from a relatively empirical insight, which is utilizing historical evaluations based on win-lose information to differentiate between good and bad states and encouraging the policy to prefer good states by reward revision. As we discuss in our paper, we implement this approach using episodic control, which we believe effectively embodies our insight. To the best of our knowledge, existing works on episodic control [1,2,3,4,5] do not contain clear theoretical insights but mostly rely on empirical insights and results, which may indicate that finding a theoretical insight for methods like ours could be a hard research question.

2.Clarification is needed on how patterns are selected during an episode. Are multiple patterns chosen, or is only a single pattern used per episode? Could you elaborate more on this?

We apologize for the unclear clarification. In Section 3.3.3, we state ‘In practice, we use sliding window’, which means we extract all patterns in an episode with sliding window for the episodic memory. To better clarify this, we will change ‘The episodic memory M is then trained to map patterns in P to the new cumulative reward R’ in Section 3.3.1 to ‘The episodic memory M is then trained to map all the patterns in P to the same new cumulative reward R’ .

3.It is unclear whether this method would succeed in more complex environments, such as StarCraft II or other sophisticated games. In these games, identifying successful patterns would likely be more challenging than in simpler environments like MuJoCo.

We implemented our method on PPO and conducted experiments in the StarCraft II environment. We use two agents that are the same to play against each other and train one of them as the adversarial agent. To our best knowledge, only the method from Guo et.al.[6] has done experiments on StarCraft II, so we compare our method with PPO and Guo et.al.. The results (“StarCraft II.png”) are available in the same link where we show the videos of our experiments in the appendix. From the results, we can see that our method enhances PPO's performance in adversarial training for StarCraft II. However, unlike its performance in Mujoco two-player competitive games, our method does not surpass the performance of the method proposed by Guo et al., which includes specific designs tailored for environments like StarCraft II which may not be strictly zero-sum. Since our method is scalable and can be integrated with various DRL algorithms, we applied our method to the approach proposed by Guo et al. The results indicate that our method slightly improves the sample efficiency of their approach in the StarCraft II environment.

We hope our responses and clarifications have addressed your key questions and concerns. If so, we kindly request that you consider revising your score to better reflect this. If there are any remaining questions, please let us know, and we will do our best to resolve them.

[1] Máté Lengyel and Peter Dayan. Hippocampal contributions to control: the third way. Advances in neural information processing systems, 20, 2007.

[2] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint arXiv:1606.04460, 2016.

[3] Steven Hansen, Alexander Pritzel, Pablo Sprechmann, André Barreto, and Charles Blundell. Fast deep reinforcement learning using online adjustments from the past. Advances in Neural Information Processing Systems, 31, 2018.

[4] Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International Conference on Machine Learning, pages 2827–2836. PMLR, 2017.

[5] Hung Le, Thommen Karimpanal George, Majid Abdolshah, Truyen Tran, and Svetha Venkatesh. Model-based episodic memory induces dynamic hybrid controls. Advances in Neural Information Processing Systems, 34:30313–30325, 2021.

[6] Wenbo Guo, Xian Wu, Sui Huang, and Xinyu Xing. Adversarial policy learning in two-player competitive games. In International Conference on Machine Learning, pp. 3910–3919. PMLR, 2021.

Weakness:

3.It is well known that the agents are not robust to adversarial attacks. It is more meaningful if it is attacking a robust agent. Or maybe at least test on more agents. For example, attacking agents that also have histories as inputs.

We apologize that we do not have enough time to train a robust agent for Mujuco two-player competitive game as the training method from "Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations" is for single-player game and no well-trained robust agents in two-player competitive game field are open for use. However, we believe this does not represent a significant weakness in our work, as our experimental setup aligns with the common practices in the field [1, 2, 3, 4] for adversarial training in two-player competitive games. Additionally, we conduct extra experiments in a more complex game environment, StarCraft II, and include the results in the same link where we provide videos of our experiments. These results demonstrate that our method remains effective in more challenging scenarios.

We hope our responses and clarifications have addressed your key questions and concerns. If so, we kindly request that you consider revising your score to better reflect this. If there are any remaining questions, please let us know, and we will do our best to resolve them.

[1] Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=HJgEMpVFwB.

[2] Wenbo Guo, Xian Wu, Sui Huang, and Xinyu Xing. Adversarial policy learning in two-player competitive games. In International Conference on Machine Learning, pp. 3910–3919. PMLR, 2021.

[3] Xian Wu, Wenbo Guo, Hua Wei, and Xinyu Xing. Adversarial policy training against deep reinforcement learning. In 30th USENIX Security Symposium (USENIX Security 21), pp. 1883–1900, 2021.

[4] The Viet Bui, Tien Mai, and Thanh H Nguyen. Imitating opponent to win: Adversarial policy imitation learning in two-player competitive games. arXiv preprint arXiv:2210.16915, 2022.

Thanks for your invaluable and enlightening review.

Weakness:

1.Using the history in robust RL is not new. With more information, it is not surprising that the results are better. "Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations"

We agree that using history features is not new in RL. As stated in Section 2.2, we implement our method using episodic control, which has been proved to effectively enhance sample efficiency by leveraging historical information in single-player games. Since adversarial training for two-player competitive games reduces the environment to a single-player setting, existing episodic control methods can be adapted for this task. To evaluate this, we implemented the SOTA episodic control method (Li et.al.) [1] with PPO and compared its performance to our method in the YouShallNotPassHumans and KickandDefend environments. We share the experiment results in the same link we show the videos of our experiments in the appendix. The experimental results demonstrate that our method outperforms [1] in YouShallNotPassHumans. Moreover, [1] appears to hinder adversarial training in KickandDefend, suggesting that existing methods may not be directly applicable to two-player competitive games. In contrast, our proposed method effectively utilizes historical information to improve adversarial training in two-player competitive games settings.

2.Need more justification for the design selection. For example, why not train a reward function that takes some history features as input. Or Why not use a transformer or a CNN instead of a LSTM?

2.1.Why not train a reward function that takes some history features as input?

We believe this approach effectively utilizes history features. In fact, the value functions of basic algorithms like PPO also leverage history features. However, as mentioned in Section 3.2, we do not directly use our reward signals (episodic feedback) as replacements for the actual rewards. Instead, we add these signals to the real rewards cumulatively. This may better differentiate between good states and bad states over time, which can more effectively highlight the good ones.

2.2. Why not use a transformer or a CNN instead of a LSTM?

The reason we use LSTM is to encode a pattern into an abstract vector which can be processed with MLP. While more complex models, such as transformers, could also fulfill this role and may have better performance, our method requires training an additional neural network alongside the primary one. This introduces challenges related to computational complexity and time cost. Implementing our method with simple neural networks like LSTM and MLP can reduce such burden, and the effectiveness of our experiments has shown that LSTM is sufficient for relatively good performance for Mujoco two-player competitive games.

[1] Zhuo Li, Derui Zhu, Yujing Hu, Xiaofei Xie, Lei Ma, Yan Zheng, Yan Song, Yingfeng Chen, and Jianjun Zhao. Neural episodic control with state abstraction. arXiv preprint arXiv:2301.11490, 2023.

We value the time and effort you've dedicated to reviewing our work.As it is close to the end of the discussion, we kindly request your feedback on the response to ascertain whether it effectively addresses your concerns.