Solving robust MDPs as a sequence of static RL problems

1. In appendix. I, the author shows the training times for SAC and IWOCS in Mujoco control tasks. As shown in Table 6, IWOCS requires about twice time cost for training compared to SAC, leading to additional burdens in real applications. According to this phenomenon, please provide the following results to analyze and improve the new method:
   1. Provide a more detailed breakdown of where the additional computation time is spent in IWOCS (e.g. worst-case search, policy optimization, etc.). This would help identify specific areas for potential optimization.
   2. Provide potential methods or approaches to reduce the computational cost of IWOCS.
   3. Include training times for other baseline robust RL methods to provide more context for evaluating IWOCS's computational efficiency relative to the state-of-the-art.
2. This work focuses on the RL task with uncertain transition, such as robotics control with dynamic friction settings. This issue is also researched in Meta-RL domains, such as Meta RL methods with task inference. Such methods can adapt to new environment transitions with limited interactions. Please explain the relationship or difference of this work and such meta RL methods. A brief introduction of Meta-RL with uncertain transition should be included in the related work, with highlighting key similarities and differences between IWOCS and meta-RL approaches.
3. Some typos in the manuscript, such as line 085: denote -> denotes

The paper has two main weaknesses:

(1) The primary weakness is the lack of theoretical contribution, as most theoretical results presented are well-known. The application of established theories such as the lack of duality gap and the equivalence between static and dynamic stationary models, to the IWOCS setting seems trivial and adds limited novelty. A more in-depth discussion of how IWOCS provides advantages (i.e. scalability analysis, sample complexity) over existing robust MDP algorithms would strengthen the paper.

(2) The motivation of the paper appears somewhat contradictory. On one hand, it is stated that the static model is more challenging to solve than the dynamic model and is valuable for decision-makers seeking robustness to static transition model uncertainty. On the other hand, the authors note that, under mild assumptions, the static and dynamic models are equivalent. This positioning makes it seem as though the approach achieves similar outcomes to robust value iteration (RVI) but with added complexity. The paper could benefit from more detailed analysis of IWOCS against other robust algorithms beyond short discussion on "rationale to why IWOCS might be a time-efficient algorithm in large scale robust RL problems".

Minor Issue: Algorithm 1 missing close bracket in the if statement: $Q_i(s_0,\pi_i(s_0))$

• The main weakness of the paper is its originality. This is limited since prior work proposes a similar approach to estimate the worst-case MDP to compute a robust policy based on a single transition function (Gadot et al. 2024).

• The significance of the proposed approach also seems limited when we put it in perspective with the work from Gadot et al. (2024), since this method requires a specialized implementation of the SAC algorithm, while the method from Gadot et al. (2024) does not requires any changes on the underlying RL algorithm. Furthermore, their method already shows the scalability of deep RL algorithms to robust settings, which puts the empirical contribution of this paper in check. While the paper provides a considerable overview of the related work literature, it misses a key reference, as discussed above. The paper needs a clear discussion of such an approach and, if appropriate, an empirical comparison with it. Lastly, the presentation of the worst-case identification on the deep IWOCS is difficult to parse. Furthermore, the provided supplemental material does not provide any instructions about how to use the code. A pseudo-code detailing this procedure and instructions to reproduce the experiment would significantly increase the clarity and reproducibility of the paper.

• Gadot, U., Wang, K., Kumar, N., Levy, K. Y., and Mannor, S. (2024). Bring your own (Non-robust) algorithm to solve robust MDPs by estimating the worst kernel. ICML, 14408–14432. https://proceedings.mlr.press/v235/gadot24a.html

• It is hard to talk about soundness of the results because there is no precise problem statement.
• As such, it is impossible to prove a meaningful result, which is critical when dealing with uncertainties and their impact on performance and safety.
• The literature within which the paper places itself misses a large body of more relevant work on RL and IRL (which solves a series of RL problems as a subroutine) for robust/uncertain MDPs. This body of work originates from probabilistic formal methods and investigates how existing approaches for planning in robust/uncertain MDPs can be utilized in RL. Examples include the following. https://www.prismmodelchecker.org/papers/neurips22.pdf, https://arxiv.org/html/2312.06344v1, https://arxiv.org/html/2312.06344v1, https://ojs.aaai.org/index.php/ICAPS/article/view/19785
• The method is, at best, ad hoc. The implementation leaves many choices to the user.

• The primary sampling-based approach raises significant concerns. Typically, solving a max (or min) problem through sampling requires an exponential number of points to achieve the desired accuracy, making this approach generally intractable from a theoretical standpoint.

• There is no discussion on the number of transition samples needed to obtain a robust solution.

• The sampling-based approach appears overly simplistic and could be seen as a straightforward baseline for robust MDP. Combined with the lack of theoretical analysis (e.g., theoretical analyses on the impact of the number of samples on the convergence), the contributions feel incremental.

• The proposed algorithm does not seem scalable. The authors should consider comparisons with some model-free algorithms and test on larger benchmarking tasks.