First and foremost, we greatly appreciate the reviewer for pointing out the typos in the comments, as well as the constructive suggestions regarding paper writing. We will revise accordingly to further improve the paper’s readability and quality. In the following, we will elaborate in detail on the questions raised by the reviewer.

Questions on AnyMDP

Behavior policies

(To answer the reviewer's Question 1.)

Definitions for each policy

On page 20, we present Table 5 about "Correspondence of prompt IDs and the policies it represents". We thank the reviewer for pointing out the issue of lacking exact definitions for each policy, and will subsequently include them in the paper. For IDs 0 to 3, the policies are value iteration policies with different gamma factors, corresponding to short-term to long-term oracles. For the model-based reinforcement learner, we implement the RL solver algorithm based on Optimistic Thompson Sampling (OTS) for episodic reinforcement learning problems in unknown environments. It balances exploration and exploitation by dynamically adjusting exploration noise (incorporating UCB heuristics) to learn the optimal policy. The Tabular Q Learner is a model-free reinforcement learning algorithm that maintains a Q-table to store Q-values for each state-action pair, updates these values via temporal-difference learning using rewards and next-state information from environmental interactions.

Purpose of using different learning methods

While the Q-learning and model-based behavior policies are always optimal at the end of training, their learning methods and paths during training differ. In our data generation process, these two policies learn synchronously along the trajectory of the data, rather than using pre-trained models to generate data. Because what we aim for our OmniRL model to learn is how to converge online to a good policy in a new task through interaction with the environment. Thus, the learning processes of these two behavior policies are actually more important than the final optimal policy—after all, the oracle policy can be directly obtained via value iteration.

Random sampling with condition ensures both diversity and quality

(To answer the reviewer's Question 2.)

The sampling of transition matrix and reward matrix in AnyMDP is not uniformly random, to avoid generating trivial MDP problems (e.g., MDPs with optimal episode length=1). Uniform randomization would result in generating a majority of trivial Markov Decision Processes (MDPs), such as those with optimal episode lengths of 1 or MDPs where most states are irrelevant to the optimal solution. The policy only needs to remember which short path has the best reward and will not learn the trade-off of risk assessment. Eventually, the environment will be more inclined towards the bandit problem. In most complex RL problems, delayed reward is quite common. Usually, a large positive reward is only obtained when the goal is achieved. Sometimes, there may even be a stage of deduction before getting the positive reward.

Ergodicity is ensured by Theorem 1 in the appendix, which guarantees that a uniformly random policy induces a strictly positive probability of reaching every state from the initial state. At the same time, AnyMDP does not guarantee that the optimal policy is strictly ergodic—a trait absent in most existing benchmarks. However, the banded transition kernel and reward structure ensure that the optimal policy requires transitions across a substantial number of states.

Thus, via the "Banded transition kernel" condition specified in Line 525, we ensure the path to reaching high-value targets is longer. Following the procedure outlined in Algorithm 1 (page 15), we: randomly generate states with varying values (Line 2 of Algorithm 1); randomly select states that trigger reset or termination (Lines 3–4); randomly sample transition matrices under the "Banded transition kernel" condition (Lines 5–12); and randomly sample composite reward functions that satisfy the value condition (Lines 13–19). This approach ensures both task diversity and quality simultaneously.

Priori knowledge

(To answer the reviewer's Question 5.)

The reviewers raised questions regarding the inclusion of prior information in AD, AD𝜖, and DPT. To clarify, prior information introduces additional details about different behavioral policies—specifically, policy tags. Since the AD policy remains unchanged during generation, adding prior information has no theoretical effect. For AD𝜖, policy tags can be incorporated based on the noise level. Additionally, as DPT involves multiple policies, the inclusion of policy tags is feasible. We believe that adding policy tags is also highly likely to enhance performance for AD𝜖 and DPT.

Indeed, compared with DPT, our primary contribution lies in proposing the Step-wise Supervision method, which enhances the strength of supervisory signals and improves training efficiency. As for policy tags, they serve as an additional benefit—analogous to high-quality prompts in LLMs—by informing the model of the actual level/type of the current action, thereby facilitating strategy distillation during training.

Question on experiment settings

Testing task

(To answer the reviewer's Question 3.)

We chose the RL benchmark Gym for our out-of-distribution tasks. In most suitable games within it, state and action spaces are fixed, making it infeasible to conduct a comprehensive experiment on the number of states and actions. However, in Figure 3 on page 6, we compare performance across AnyMDP tasks with different numbers of states, where results show that difficulty increases as the state space grows.

Multi-agent test

(To answer the reviewer's Question 4.)

Both agents are independent OmniRL agents. They perform ICL in the same environment with distinct observations. Observations are partial: one agent can only observe the column number of the other agent. Thus, both agents operate dynamically. We were also pleasantly surprised that the model—trained on single-agent datasets with a single-agent architecture—could solve multi-agent problems. This also inspires us to explore the application of ICL in multi-agent scenarios.

On the Value of Discrete Space MDPs

(To answer the reviewer's Question 1.)

We appreciate suggestions to extend our framework to continuous MDPs, but we feel that the value of discrete MDPs is underestimated. At this stage, partially pursuing the task complexity risks undermining the core advantages of procedurally generated discrete MDPs. Below, we explain the challenges of preserving key properties in continuous domains and contrast our approach with contemporary benchmarks.

Challenges of Continuous MDPs in Preserving Structural Integrity

Extending procedurally generated MDPs to continuous spaces while maintaining low structural bias and sufficient diversity poses significant technical hurdles. At the current stage, efforts moving to continuous MDPs risk typically prioritizing "absolute task complexity" over qualitative diversity, introducing biases or homogenizing task structures—contrary to our framework's goals.

Inherent Long-Term Dependency and Qualitative Diversity

(To answer the reviewer's Question 3.)

The reviewer suggests conducting tests on tasks with varying complexity and different types of disturbances. In reality, existing continuous MDP benchmarks rely on continuously randomized hyperparameters to simulate "infinite tasks," prioritizing few-shot learning over long-horizon in-context learning (ICL), such as environments like maze navigation and open-world games. Our analysis reveals two insights:

• Task Diversity ≠ Numerical Scale: True diversity depends on qualitative variation in task structures, not just quantity.
• ICL efficacy depends on disparity between tasks in the training set, not individual task complexity. Existing continuous benchmarks tend to produce homogeneous task structures, limiting models to superficial pattern recognition via in-weight learning (IWL) rather than sustained reasoning. In contrast, discrete MDPs in AnyMDP are designed with procedural complexity and temporal coherence, enabling rigorous evaluation of a model’s ability to generalize across structurally distinct tasks and sustain reasoning over extended sequences.

Compatibility with Existing LLMs

Beyond being a novel benchmark, AnyMDP bridges a critical gap in assessing experience-driven, holistic reasoning capabilities of pre-trained LLMs. Existing benchmarks predominantly measure declarative memory and one-shot circuits, while AnyMDP emphasizes procedural memory and experience-driven learning.

Low-cost Access to Oracle Policy

This property significantly enhances the efficiency and scalability of ICRL training, making AnyMDP a scalable benchmark.

Empirical analysis across state-of-the-art architectures

We augment the empirical analysis with the training curves and ICL comparisons across state-of-the-art architectures, including Mamba2, RWKV-7, Gate Delta Net, and other competitive models. This will demonstrate AnyMDP’s utility as a rigorous benchmark for advancing long-context modeling capabilities.

• Table 1. Validation Loss of Different Models on AnyMDP Datasets | CL | 1 | 100 | 1000 | 10000 | 30000 | |-------------|------------|------------|------------|------------|------------| | Mamba2 | 1.417 ± 0.035 | 1.272 ± 0.051 | 0.822 ± 0.064 | 0.436 ± 0.053 | 0.294 ± 0.042 | | GSA | 1.406 ± 0.034 | 1.235 ± 0.053 | 0.765 ± 0.066 | 0.342 ± 0.050 | 0.235 ± 0.042 | | Gate DeltaNet | 1.356 ± 0.035 | 1.130 ± 0.051 | 0.599 ± 0.055 | 0.267 ± 0.049 | 0.173 ± 0.042 | | RWKV-7 | 1.371 ± 0.034 | 1.151 ± 0.052 | 0.627 ± 0.055 | 0.269 ± 0.042 | 0.175 ± 0.029 |

Overall, in our research on ICL, the benefits brought by the discrete MDPs far outweigh the drawbacks of its limited problem space. We have also started to explore ICL in the continuous space.

For future work in continuous settings, there are two directions. First, we will focus on specific continuous problems and prioritize in-domain adaptation. For instance, in frame prediction tasks, generalization manifests as adaptation to environments with distinct features and spatial structures. Similarly, in robot whole-body locomotion control, it manifests in adapting to different robot configurations (size, shape, weight). Second, we will continue exploring how to generate tasks with high diversity and low structural bias in continuous spaces. Undoubtedly, the first direction holds more practical value, while the second poses greater theoretical challenges.

The "Perturbation" in Question 4

The reviewer mentions "perturbations can lead to a decline in model performance." We believe there are no such claims in our paper. Two candidates appear in our manuscript, neither fitting this question:

• Domain/World randomization (Line 132): This creates task diversity; AnyMDP already exposes the learner to maximal variations, so no need to delve into specific impacts of perturbations.
• Stochasticity in the behavior policy (Line 620 & Table 5): This is necessary for data synthesis and seems irrelevant to the mentioned perturbation. Moreover, incorporating Domain/World randomization and stochasticity in the behavior policy has enhanced model performance, contradicting the reviewer's viewpoint.

About Long-Term Adaptation in Weaknesses

The reviewer views long-term adaptation as a disadvantage of our methods, but we believe there are misunderstandings. Our proposed methods, step-wise supervision (SS) and chunk-wise learning, aim to improve in-context sample efficiency, not intentionally extend context length. Our experiments reveal that "Long-Term Adaptation is a tax we must pay toward increasing the generalization scope for ICRL." Characterizing increased context length as an inherent limitation seems unjustified, even from the standpoint of whether trading generalization for long-context adaptation is worthwhile.

About Computational Costs in Question 5

In Figure 7 on page 17, we show the time consumption of the AnyMDP task generation on an Intel(R) Xeon(R) Platinum 8374C CPU. After generating the task files, the time consumption of generating the sequences accordingly is within hours, which can be ignored compared to the training cost. To train an OmniRL model with [16,128] state space and action space of 5, we generate 512K sequences with length 12K. The total training iterations are 0.2M, 8 seconds per iteration on 8 GPUs. We have also considered conducting the training across task complexity to assess the exact consumption, but the computing resources are tight. In general, training a model with a larger number of states or actions will bring greater resource consumption. After the problem becomes more complex, more sequences are needed to learn generalization, and each sequence has to be longer to obtain sufficient information for a single task.

Evaluated More ICRL Methods in Question 2

We have conducted experiments on well-known ICRL frameworks, including AD, AD𝜖, and DPT. Another technical direction is RL². However, with existing large-scale task sets, model scales, and context lengths, the computational cost of running RL is substantial. Using fewer tasks, smaller models, or shorter context lengths would render the conclusions lacking in comparative significance. Furthermore, methods like DPT have already been compared with RL²; thus, we believe the results from comparisons with DPT are sufficiently compelling.

First and foremost, we sincerely thank the reviewer for identifying the typos. We will revise accordingly to further improve the paper’s readability. In the following, we will elaborate in detail on the weaknesses and questions raised by the reviewer.

On the Value of Discrete Space MDPs

We appreciate the reviewer's interest in extending the task to continuous domains. Before discussing how to extend to continuous spaces, we would first like to engage in a deeper discussion with the reviewers about why we chose discrete MDPs at this stage, as well as the benefits this choice brings.

At this stage, partially pursuing the task complexity risks undermining the core advantages of procedurally generated discrete MDPs. Below, we explain the challenges of preserving key properties in continuous domains and contrast our approach with contemporary benchmarks.

Challenges of Continuous MDPs in Preserving Structural Integrity

Extending procedurally generated MDPs to continuous spaces while maintaining low structural bias and sufficient diversity poses significant technical hurdles. At the current stage, efforts moving to continuous MDPs risk typically prioritizing "absolute task complexity" over qualitative diversity, introducing biases or homogenizing task structures—contrary to our framework's goals.

Inherent Long-Term Dependency and Qualitative Diversity

Existing continuous MDP benchmarks rely on continuously randomized hyperparameters to simulate "infinite tasks," prioritizing few-shot learning over long-horizon in-context learning (ICL). Our analysis reveals two insights:

• Task Diversity ≠ Numerical Scale: True diversity depends on qualitative variation in task structures, not just quantity.
• ICL efficacy depends on disparity between tasks in the training set, not individual task complexity. Existing continuous benchmarks tend to produce homogeneous task structures, limiting models to superficial pattern recognition via in-weight learning (IWL) rather than sustained reasoning. In contrast, discrete MDPs in AnyMDP are designed with procedural complexity and temporal coherence, enabling rigorous evaluation of a model’s ability to generalize across structurally distinct tasks and sustain reasoning over extended sequences.

Compatibility with Existing LLMs

Beyond being a novel benchmark, AnyMDP bridges a critical gap in assessing experience-driven, holistic reasoning capabilities of pre-trained LLMs. Existing benchmarks predominantly measure declarative memory and one-shot circuits, while AnyMDP emphasizes procedural memory and experience-driven learning.

Low-cost Access to Oracle Policy

This property significantly enhances the efficiency and scalability of ICRL training, making AnyMDP a scalable benchmark.

Empirical analysis across state-of-the-art architectures

We also augment the empirical analysis with the training curves and ICL comparisons across state-of-the-art architectures, including Mamba2, RWKV-7, Gate Delta Net, and other competitive models. This will demonstrate AnyMDP’s utility as a rigorous benchmark for advancing long-context modeling capabilities.

• Table 1. Validation Loss of Different Models on AnyMDP Datasets | CL | 1 | 100 | 1000 | 10000 | 30000 | |-------------|------------|------------|------------|------------|------------| | Mamba2 | 1.417 ± 0.035 | 1.272 ± 0.051 | 0.822 ± 0.064 | 0.436 ± 0.053 | 0.294 ± 0.042 | | GSA | 1.406 ± 0.034 | 1.235 ± 0.053 | 0.765 ± 0.066 | 0.342 ± 0.050 | 0.235 ± 0.042 | | Gate DeltaNet | 1.356 ± 0.035 | 1.130 ± 0.051 | 0.599 ± 0.055 | 0.267 ± 0.049 | 0.173 ± 0.042 | | RWKV-7 | 1.371 ± 0.034 | 1.151 ± 0.052 | 0.627 ± 0.055 | 0.269 ± 0.042 | 0.175 ± 0.029 |

Overall, in our research on ICL, the benefits brought by the discrete MDPs far outweigh the drawbacks of its limited problem space. We have also started to explore ICL in the continuous space.

For future work in continuous settings, there are two directions. First, we will focus on specific continuous problems and prioritize in-domain adaptation. For instance, in frame prediction tasks, generalization manifests as adaptation to environments with distinct features and spatial structures. Similarly, in robot whole-body locomotion control, it manifests in adapting to different robot configurations (size, shape, weight). Second, we will continue exploring how to generate tasks with high diversity and low structural bias in continuous spaces. Undoubtedly, the first direction holds more practical value, while the second poses greater theoretical challenges.

Explanation of AnyMDP

How to generate high-quality MDPs

(To answer the reviewer's Weakness 1 and Question 1.)

The sampling of transition matrix and reward matrix in AnyMDP is not uniformly random, to avoid generating trivial MDP problems (e.g., MDPs with optimal episode length=1). Uniform randomization would result in generating a majority of trivial Markov Decision Processes (MDPs), such as those with optimal episode lengths of 1 or MDPs where most states are irrelevant to the optimal solution. The policy only needs to remember which short path has the best reward and will not learn the trade-off of risk assessment. Eventually, the environment will be more inclined towards the bandit problem. In most complex RL problems, delayed reward is quite common. Usually, a large positive reward is only obtained when the goal is achieved. Sometimes, there may even be a stage of deduction before getting the positive reward.

Ergodicity is ensured by Theorem 1 in the appendix, which guarantees that a uniformly random policy induces a strictly positive probability of reaching every state from the initial state. At the same time, AnyMDP does not guarantee that the optimal policy is strictly ergodic—a trait absent in most existing benchmarks. However, the banded transition kernel and reward structure ensure that the optimal policy requires transitions across a substantial number of states.

As we have presented above, AnyMDP can serve as a Long-Term Dependency Benchmark. For this reason, we must limit the step length—meaning that to reach high-value targets, the path cannot be short. As the number of states increases, the path to achieving a high score grows correspondingly longer. Thus, "Under a uniform random policy, the probability of reaching high-valued states remains greater than 0 but decreases at least exponentially with respect to $n_s$."

Behavior policies

(To answer the reviewer's Weakness 3.)

On page 20, we present Table 5 about "Correspondence of prompt IDs and the policies it represents". We thank the reviewer for pointing out the lack of clarity in this part, and will subsequently include them in the paper.

• For IDs 0 to 3, the policies are value iteration policies with different gamma factors, corresponding to short-term to long-term oracles ($\gamma$=0.5, 0.93, 0.994).
• The model-based reinforcement learner comprises two distinct components: an environment model and a reward model, both conditioned on the current interaction history. A Bellman update is applied at every step to compute the mean value function, which is subsequently augmented with Optimistic Thompson Sampling to balance exploration and exploitation.
• Tabular Q-learner with UCB is a model-free method that maintains a Q-table of state-action values. These values are iteratively refined via value iteration and adjusted by an upper-confidence-bound bonus derived from rewards and successor-state information observed during interaction.

Regarding the exact impact of the diversity of behavior policies in training data, DPT has demonstrated its benefit on model performance. Indeed, compared with DPT, our primary contribution lies in proposing the Step-wise Supervision method, which enhances the strength of supervisory signals and improves training efficiency. As for policy tags, they serve as an additional benefit—analogous to high-quality prompts in LLMs—by informing the model of the actual level/type of the current action, thereby facilitating strategy distillation during training.

Gymnasium environments and reward shaping

(To answer the reviewer's Question 2.)

We select the Gym envs based on two points: 1. The env should have a delayed reward to make it more difficult. 2. The observation and action space can match our model. We will also cite the Deep Sea environment, and compare the difference with the Gymnasium.

(To answer the reviewer's Question 3.)

Reward shaping is a common practice in reinforcement learning, and it is applied consistently across all baselines as well as our model—we consider this approach fair. The quality of such reward shaping is analogous to that of prompts in large models: it helps render the problem more intuitive. Notably, without reward shaping, both the context size required for our model’s online learning and the number of training iterations needed for baselines would increase.

First and foremost, we sincerely thank the reviewer for identifying the typos and providing valuable suggestions on the writing. We will revise accordingly to further improve the paper’s readability, especially on the explanation of ICL and Casual Blocks. In the following, we will elaborate in detail on the weaknesses and questions raised by the reviewer.

Garnet MDP and other related works

We appreciate the reviewer’s reference to Garnet MDP, an efficient MDP sampling methodology. Key parallels and distinctions include:

• Similarities:
  • Both prioritize MDP structural integrity
  • Both focus on stopping "fast mixing" MDPs. (In our case, theorem 1 also guarantees this point)
• Differences:
  • Garnet MDP controls the transitions under an optimal policy, whereas AnyMDP controls the transitions under the uniformly random policy. In our case, we guarantees that a uniformly random policy induces a strictly positive probability of reaching every state from the initial state (ergodic). However, AnyMDP does not guarantee that the optimal policy is strictly ergodic---a better property aligns with most existing benchmarks. Nonetheless, the banded transition kernel and reward structure ensure that the optimal policy requires transitions across at least a substantial number of states (nearly ergodic).
  • Garnet MDP is tailored for infinite-horizon MDPs, while AnyMDP supports both infinite- and finite-horizon settings (aligning with common RL benchmarks).

The paper, Natural actor–critic algorithms, presents four new natural actor–critic reinforcement learning algorithms, provides their convergence proofs, and improves upon prior work by being the first to offer such proofs and propose fully incremental algorithms for these methods. MICo, the learnt representations in MDP problems, demonstrated a gratifying performance improvement for DQN and SAC deep reinforcement learning agents. A Kernel Perspective on Behavioural Metrics defines a new metric equivalent to the reduced MICo distance, establishes new theoretical results, and demonstrates strong empirical effectiveness in deep reinforcement learning.

While our work also focuses on MDP problems, the emphasis differs. The AnyMDP we propose aims to generate tasks with both diversity and quality. Using datasets generated by AnyMDP, we train a generalizable OmniRL model to investigate the in-context learning (ICL) mechanism of large models.

We appreciate the reviewer's suggestion. We will include a proper comparison with prior work on procedurally generating MDPs in the related works section and cite these papers.

AD, AD𝜖, DPT, and our AnyMDP

Explanation of Figure 1

Figure 1 illustrates the differences between AD, DPT, and our AnyMDP. Blue arrows (Behavior) depict the trajectory fed to the model; yellow arrows (Reference) provide the corresponding supervisory signal. (Left) For AD, the reference action is identical to the behavioral action. (Middle) For a sequence of behaviors, DPT provides a reference action only at the end of the sequence. (Right) For our method, we provide a reference action at each step of the behavioral sequence, which in turn enhances the strength of the supervisory signal and improves training efficiency. The key contribution of Stepwise Supervision is that it tolerates step-level inconsistencies between behavior and reference, aiming at solving the issue of distribution shift in imitation learning.

Explanation of prior information

The reviewers raised questions regarding the inclusion of prior information in AD, AD𝜖, and DPT. To clarify, prior information introduces additional details about different behavioral policies—specifically, policy tags. Since the AD policy remains unchanged during generation, adding prior information has no theoretical effect. For AD𝜖, policy tags can be incorporated based on the noise level. Additionally, as DPT involves multiple policies, the inclusion of policy tags is feasible. We believe that adding policy tags is also highly likely to enhance performance for AD𝜖 and DPT.

Indeed, compared with DPT, our primary contribution lies in proposing the Stepwise Supervision method. As for policy tags, they serve as an additional benefit—analogous to high-quality prompts in LLMs—by informing the model of the actual level/type of the current action, thereby facilitating strategy distillation during training.

The prior information is accessible. During the training phase, the classification of prior information is manually designed, and we randomly mask some tags as "Unknown". In online testing, we allow the model to assess current performance with all tags set to "Unknown".

Experiment settings in Section 4.3

Section 4.3 aims to examine how the number of tasks influences generalization. While all tasks in our experiments are drawn from AnyMDP, we ensured sufficient diversity among them—and our results support this observation. Specifically, models trained on fewer tasks showed limited performance on other AnyMDP tasks, whereas satisfactory generalization to additional AnyMDP tasks was only achieved when the number of training tasks was sufficiently large. This aligns with the key point we intended to convey: a larger number of tasks can foster generalization.

Furthermore, our experimental observations suggest that if a model fails to generalize within AnyMDP, it is unlikely to generalize to Gym tasks either. We believe that out-of-distribution generalization is inherently more challenging than in-distribution generalization.

We sincerely appreciate the reviewer’s insightful feedback, and we carefully address the reviewer's concern as below. We will also include the necessary discussion and revision in the paper to answer those questions.

Q1: Compare AnyMDP with Existing Benchmarks

• (Archibald et al., 1995) outlines a three-step procedure for constructing MDPs: (1). Transition structure generation for the optimal policy,(2). Finalization of reward and transition data for the optimal policy (ensuring ergodicity), and (3). Incorporation of non-optimal policy connections. It prevents "fast mixing" of states and also ergodicity of state-transition under optimal policy. Theorem 1 in our paper establishes two key properties under a uniform random policy: (1). It precludes fast mixing (i.e., the stationary distribution decay at least exponentially), ensuring convergence is not overly rapid, which is critical for stable learning dynamics. (2). It guarantees ergodicity (the stationary distribution is strictly positive for all states), ensuring all states are reachable under this policy. An analysis of existing benchmarks reveals a critical trade-off: while transitions under a uniformly random policy are designed to be ergodic (as required for exploratory robustness), transitions under an optimal policy are not inherently ergodic.

• The Garnet MDP (Bhatnagar et al., 2007) represents another class of MDP samplers, parameterized as Garnet(n, m, b, σ, τ). Notably, Garnet supports non-stationary MDPs when τ != 0, with parameter b constraining state-action pair transitions. Upon closer examination—and excluding cases with absorbing states—AnyMDP can be interpreted as a subset of Garnet(n_s, n_a, b = b₋ + b₊, σ) （ b₋ , b₊ are the parameters in Algorithm1 in appendix）. However, Garnet does not inherently guarantee ergodicity (e.g., unreachable states may exist) or fast mixing, and more closely aligns with a variant of AnyMDP that lacks composite rewards and banded transition. In contrast, AnyMDP emphasizes irreducibility, longer-delayed rewards, and more challenging tasks, distinguishing it from standard Garnet formulations.

Importantly, neither the Archibald et al. (1995) framework nor Garnet MDPs explicitly account for absorbing states (e.g., terminal states requiring environment resets, such as pitfalls or goal states). This could also be a crucial limitation to generalize to existing benchmarks.

Q2: The contribution of prior (tag) and stepwise supervision

The core distinction between DPT and Stepwise Supervision (SS) lies in their data structuring and training efficiency

• DPT structures data as <Trajectory, Independently Sampled Single Query State, Reference Action> tuples
• Stepwise Supervision is theoretically identical to <Trajectory, Sequence of All Trajectory States as Query States, Sequence of Reference Actions> tuples.

For each trajectory, DPT trains only one state-action pair, whereas Stepwise Supervision leverages all state-reference action pairs across the trajectory. This makes Stepwise Supervision theoretically L-fold more training-efficient (where L = trajectory length). In our context (context length over 8K to 512K, DPT becomes extremely in efficient.

Our revised manuscript will emphasize that Stepwise Supervision offers significantly greater scalability and training efficiency compared to DPT. We will introduce the DPT+ prior in the final version to address these limitations. Figure 6 directly contrasts DPT with OmniRL (without prior knowledge), effectively isolating and demonstrating SS's advantages over DPT.

On the Value of Discrete Space MDPs

We appreciate the reviewer's positive feedback on the value of the seemingly toyishness of the discrete MDPs. At this stage, partially pursuing the task complexity risks undermining the core advantages of procedurally generated discrete MDPs. Below, we explain the challenges of preserving key properties in continuous domains and contrast our approach with contemporary benchmarks.

Challenges of Continuous MDPs in Preserving Structural Integrity

Extending procedurally generated MDPs to continuous spaces while maintaining low structural bias and sufficient diversity poses significant technical hurdles. At the current stage, efforts moving to continuous MDPs risk typically prioritizing "absolute task complexity" over qualitative diversity, introducing biases or homogenizing task structures—contrary to our framework's goals.

Inherent Long-Term Dependency and Qualitative Diversity

Existing continuous MDP benchmarks rely on continuously randomized hyperparameters to simulate "infinite tasks," prioritizing few-shot learning over long-horizon in-context learning (ICL). Our analysis reveals two insights:

• Task Diversity ≠ Numerical Scale: True diversity depends on qualitative variation in task structures, not just quantity.
• ICL efficacy depends on disparity between tasks in the training set, not individual task complexity. Existing continuous benchmarks tend to produce homogeneous task structures, limiting models to superficial pattern recognition via in-weight learning (IWL) rather than sustained reasoning. In contrast, discrete MDPs in AnyMDP are designed with procedural complexity and temporal coherence, enabling rigorous evaluation of a model’s ability to generalize across structurally distinct tasks and sustain reasoning over extended sequences.

Compatibility with Existing LLMs

Beyond being a novel benchmark, AnyMDP bridges a critical gap in assessing experience-driven, holistic reasoning capabilities of pre-trained LLMs. Existing benchmarks predominantly measure declarative memory and one-shot circuits, while AnyMDP emphasizes procedural memory and experience-driven learning.

Low-cost Access to Oracle Policy

This property significantly enhances the efficiency and scalability of ICRL training, making AnyMDP a scalable benchmark.

Empirical analysis across state-of-the-art architectures

We also augment the empirical analysis with the training curves and ICL comparisons across state-of-the-art architectures, including Mamba2, RWKV-7, Gate Delta Net, and other competitive models. This will demonstrate AnyMDP’s utility as a rigorous benchmark for advancing long-context modeling capabilities.

• Table 1. Validation Loss of Different Models on AnyMDP Datasets | CL | 1 | 100 | 1000 | 10000 | 30000 | |-------------|------------|------------|------------|------------|------------| | Mamba2 | 1.417 ± 0.035 | 1.272 ± 0.051 | 0.822 ± 0.064 | 0.436 ± 0.053 | 0.294 ± 0.042 | | GSA | 1.406 ± 0.034 | 1.235 ± 0.053 | 0.765 ± 0.066 | 0.342 ± 0.050 | 0.235 ± 0.042 | | Gate DeltaNet | 1.356 ± 0.035 | 1.130 ± 0.051 | 0.599 ± 0.055 | 0.267 ± 0.049 | 0.173 ± 0.042 | | RWKV-7 | 1.371 ± 0.034 | 1.151 ± 0.052 | 0.627 ± 0.055 | 0.269 ± 0.042 | 0.175 ± 0.029 |

Overall, in our research on ICL, the benefits brought by the discrete MDPs far outweigh the drawbacks of its limited problem space. We have also started to explore ICL in the continuous space.

Question about more complex domains and continuous settings

• For future work in continuous settings, there are two directions. First, we will focus on specific continuous problems and prioritize in-domain adaptation. Second, we will continue exploring how to generate tasks with high diversity and low structural bias in continuous spaces. Undoubtedly, the first direction holds more practical value, while the second poses greater theoretical challenges.
• If the second direction remains unaddressed, cross-domain generalization in continuous spaces will be difficult to achieve, but in-domain adaptation is highly feasible. For instance, in frame prediction tasks, generalization manifests as adaptation to environments with distinct features and spatial structures. Similarly, in robot whole-body locomotion control, it manifests in adapting to different robot configurations (size, shape, weight).
• Another more complex domain is Multi-agent problem. As agents need to collaborate and their performance cannot be guaranteed to be consistent, a new challenge arises: in-context adaptation across other agents' actions. If a single agent can observe and learn other agents' actions online, while adjusting its own real-time policy, it could potentially alleviate the fragility of existing multi-agent systems. We believe this scenario will uncover more unknown findings.