Analyzing the Effects of Emulating on the Reinforcement Learning Manifold

Thank you for your well-considered and thoughtful review.

---

“While it may be tricky to produce, it may be useful to show what the perturbations used in the environment really look like”

Yes, we will add this information to the supplementary material. Thank you very much for the suggestion.

Thank you for the time you have allocated in providing feedback on our paper.

---

1. “In several sections, the paper mentions "with reward" and "without reward." However, their exact meanings are unclear. Does "without reward" refer to imitation and inverse RL?”

Yes, without reward refers to imitation and inverse RL.

---

2. ”Furthermore, "without reward" is commonly associated with unsupervised RL, where the agent does not receive a reward signal. Could you please provide clarification? “

Yes, as mentioned above without reward refers to imitation and inverse reinforcement learning.

---

3. “For instance, does “that focus on learning via emulating affect” refer to imitation learning?”

Yes, it refers to imitation learning.

---

4. “What exactly is meant by "learning from exploration"? Is it referring to regular reinforcement learning?”

Yes, learning from exploration refers to regular reinforcement learning.

---

5. “Could you clarify how the proposed method learns without a reward function?”

The method mentioned here is the inverse Q-learning algorithm which is also explained in Page 3 between the lines 12 and 19 of the Section titled “Learning without Rewards”.

---

6. “Throughout the paper, I find it challenging to link the various Lemmas, Propositions, Corollaries, and their proofs to the central message of the paper. For instance, the purpose and significance of Lemma 3.1 (and its proof) remain unclear to me. The necessity and meaning behind Propositions 3.3 and 3.5 are also ambiguous.”

Lemma 3.1, Proposition 3.3, and Proposition 3.5 together theoretically demonstrate that learning only from expert trajectories results in highly non-robust policies in a natural setting. In particular, Lemma 3.1 proves that states in expert trajectories have large projections onto a subspace implicitly defined by the actions leading to high reward. Proposition 3.3 proves that training on states from the subspace described in Lemma 3.1. results in a policy that is essentially untrained (i.e. takes actions no better than random guessing) outside this subspace. Proposition 3.5 then uses Proposition 3.3 to analyze a natural setup in which a policy trained in the subspace is highly non-robust. The non-robustness and vulnerabilities of these policies are further justified empirically in Section 4 and Section 5.

---

7. “The paper intends to analyze the reinforcement learning manifold. However, the meaning is not evident from the text. Could you clarify what this entails and how the theoretical and empirical analyses relate to it?”

The reinforcement learning manifold is represented by the graph of the function $J_{\pi}(s)$, and thus implicitly given by the policy $\pi$. Here $J_{\pi}(s)$ is the expected cumulative rewards obtained by policy $\pi$ starting from state $s$. The paper both theoretically and empirically analyzes how small perturbations from states $s$ on the learned trajectory yield large changes in $J_{\pi}(s)$ when $\pi$ is trained from expert trajectories, equivalently corresponding to the manifold represented by $J_{\pi}(s)$ having steeper slopes near the learned trajectory.

---

8. “What is the intent of Definition 4.1? How does it relate to the main objective of the paper?”

Definition 4.1 describes the algorithm and MDP independent adversarial direction setting which is a state-of-the-art adversarial framework for producing black-box adversarial perturbations in deep reinforcement learning.

---

9. “Both Table 1 and Table 2 mention the tasks Pong and Seaquest, yet the results display three different games. How were these results derived?”

The games in the first and fourth row of the Table 1 and 2 represent the MDPs in which the adversarial perturbations are computed. As already described in detail in definition 4.1. the algorithm and MDP independent adversarial direction $\mathcal{A}_{alg+\mathcal{M}}^{random}$ computes an adversarial direction independent from the MDP in which the perturbations are added to the state observations.

---

10. “The state-of-the-art imitation and inverse reinforcement learning policy is trained via the inverse Q-learning algorithm described in Section 2.” However, I do not see this detailed in the paper. How can both imitation and IRL be trained via Q-learning?”

Please see the supplementary material for the training details. Furthermore, see the response to bullet point 5.

---

11. “Implementation details appear to be missing in the paper. Which architecture is used to represent the policy? Which specific algorithms are utilized for IRL and imitation? Without these specifics, it's challenging to justify the performance presented.”

This is already currently present in the supplementary material. Please see supplementary material.

Thank you, authors, for the response. However, my assessment of the paper, particularly the weaknesses mentioned earlier, remains unchanged.

My concern regarding the implementation details and their impact on the reproducibility of the study persists. In point 11, the author refers to supplementary materials for specifics. While I can find the hyperparameters of the algorithms, there is no mention of the policy architecture. For instance, is a neural network used to represent the Q-network, and if so, how many layers does it have, and what are the dimensions of these layers? Since no code is provided, these implementation details are crucial for replicating the experiments. This is also emphasized by reviewer uuJm, who noted that "...the experimental protocol is not mentioned clearly in the main paper nor the appendix," indicating that the lack of detail presents challenges for reproducibility.

Could you please respond to the following question: “For Figure 1, which environment is being depicted? Is it an average of all three games (Seaquest, BeamRider, and Breakout)? The purpose and message of Figure 1 are unclear.”

In point 10, upon reviewing the supplementary materials, I was unable to locate this specific analysis. Could you mention it here or indicate where this detail can be found?

Dear Reviewer,

Thank you for your service. The authors have put effort to try to clear all doubts in their response and revision. Can you clarify in details why your assessment remains unchanged?

Best,

AC

Thank you for dedicating your time to provide kind feedback on our paper.

---

1. ”The comparisons between the vanilla policies and those derived from IRL are unfair as the vanilla policy is trained on hundreds of thousands of tranisitions whereas its IRL counterpart gets to see only few thousands of transitions.”

We would like to kindly highlight that this statement is false. The IRL counterpart experiences up to and more than a million transitions.

---

2. “The case of studied linear Q-function approximators in the theoretical part is quite restrictive. The assumptions used for Proposition 3.5 are very strong. A potentially more general direction would be to study smooth Q-approximators in the case of Lipschitz MDPs.”

An important point here is that the theoretical part is demonstrating a failure mode of training from expert trajectories. Typically this is done by producing a counter-example MDP in which the failure occurs. Thus, a theoretical demonstration of such a counter-example in a simple setting is a stronger result, as it demonstrates that the algorithm fails even in very simple MDPs. MDPs with bounded, linearly parameterized rewards and transitions are immediately Lipschitz, and thus our counter-example setting does indeed correspond to a Lipschitz MDP.

Furthermore, linear function approximation is currently at the frontier of research that develops a rigorous theoretical understanding of reinforcement learning algorithms [1,2,3,4,5,6,7,8]. Thus, our theoretical results are demonstrated in the most general setting currently amenable to provable mathematical analysis.

[1] Bilinear classes: A structural framework for provable generalization in RL. ICML, 2021.

[2] Reinforcement learning from partial observation: Linear function approximation with provable sample efficiency. ICML 2022.

[3] Provably efficient reinforcement learning with linear function approximation. Conference on Learning Theory. PMLR, 2020.

[4] Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. Conference on Learning Theory. PMLR, 2021.

[5] Learning near optimal policies with low inherent bellman error. ICML, 2020.

[6] Reinforcement learning in linear MDPs: Constant regret and representation selection. NeurIPS 2021.

[7] Nearly minimax optimal reinforcement learning with linear function approximation. ICML, 2022.

[8] First-order regret in reinforcement learning with linear function approximation: A robust estimation approach. ICML 2022.

---

3. “Another weakness is that the authors do not offer no solutions to the problem of lack of robustness. This could be in the form of better training methods for the policy like regularization or better strategies to collect demonstrations.”

Please note that in the adversarial machine learning literature there is a line of work dedicated solely to discussing the explicit and concrete vulnerabilities of the models [1,2,3,4,5,6,7], and there is a line of work dedicated to addressing these problems described in these studies [8,9]. We believe these two lines of research progress side by side, without the expectation on one single paper carrying both of these perspectives at the same time.

[1] On Adaptive Attacks to Adversarial Example Defenses, NeurIPS 2020.

[2] Label-Only Membership Inference Attacks, ICML 2021.

[3] Poisoning and Backdooring Contrastive Learning, ICLR 2022.

[4] Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness, AAAI 2023.

[5] EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples, AAAI 2018.

[6] Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs, AAAI 2022.

[7] Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning, AAAI 2020.

[8] Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations, NeurIPS 2020.

[9] Robust Deep Reinforcement Learning through Adversarial Loss, NeurIPS 2021.

---

4. “Could you clarify what you mean by "Manifold setting" in the introduction?”

Manifold settings here refers to the first paragraph of the introduction describing the current fields that reinforcement learning is being utilized in, such as autonomous vehicles, finance and large language models.

---

5. “Could you clarify what $\phi$ stands for in the inverse Q-learning objective?”

$\phi:\mathbb{R}\to\mathbb{R}$ is a concave function corresponding to a choice of distance metric for regularization in inverse Q-learning. For example, setting $\phi(x) = x$ corresponds to total variation distance, and setting $\phi(x) = x - x^2/4$ corresponds to $\chi^2$ distance. Please see [1] for more details.

[1] IQ-Learn: Inverse soft-Q Learning for Imitation, NeurIPS 2021.

---

6. “Relating to the weaknesses, could you provide plots for the evolution of the performance of the IRL policy under adversarial and or perturbation setting in an online setting where it can perform X transitions?”

Please see the response to bullet point 1.

---

7. ”I am aware this might not be possible for the current time window. It would've been interesting see a comparison of the robustness of policies learned from expert demonstrations and those learned by ranking of trajectories. As the latter case covers sub-optimal states it can be a potential solution to the robustness problem. Could you add such experiments?”

In the RLHF setting, the trajectories selected for ranking are the outputs of an LLM (e.g. GPT-4), and hence these trajectories analogously are the “expert trajectories”. Thus, the analysis of our paper extends to the setting of ranking trajectories. Furthermore, note that LLMs have had hard limits applied to the number of rounds of dialogue with users, due to the observations that unlimited conversations steer towards harmful behavior by the LLM (i.e. insulting users, manipulating, or lying) [1].

[1] New York Times. Microsoft to Limit Length of Bing Chatbot Conversations, 2023.

1. ”Indeed, the experimental protocol is not mentioned clearly in the main paper nor the appendix…How many expert trajectories and online samples are used?...The experimental section should be rewritten to better clarify the protocol used for the DQN and IQ-Learn baselines.”

This is indeed mentioned in our supplementary material. Please see Line 14 (i.e. item 9), and Line 18 respectively of the Section Titled “HYPERPARAMETER DETAILS AND ARCHITECTURES” of the supplementary material.

---

2. ”It is not clear how the analysis of the paper extends to rankings of trajectories. Also in a general RL setting like a locomotion task or Atari games, one can imagine having access trajectories of varying performance and their respective rankings. Such trajectories could potentially cover more effectively the state space of MDP and exhibit more robustness. Indeed, in the experimental section, it is only normal for the agent to fail once it visits a single sub-optimal state as it has never encountered such states during training.”

Ranking trajectories requires human feedback (as in RLHF training for LLMs). Thus, increased coverage of the state-space via ranked trajectories comes at a very significant human labor cost, this is a substantially different level of cost than only having a few expert demonstrations, as used in the IQ learn algorithm we study.