Uncovering Untapped Potential in Sample-Efficient World Model Agents

We thank reviewer DVfk for their time and effort in reviewing our paper and for their constructive and valuable feedback.

We appreciate that reviewer DVfk acknowledged the clarity of our presentation, rigorous evaluation practices, strong baseline comparisons, and the effective integration of known techniques, supported by insightful ablations.

Weaknesses:

W1:

The primary weakness of the paper is its limited originality from a methodological standpoint. ... the work does not introduce a new fundamental algorithm or mechanism. Since the method does methodically nothing novel, a slightly stronger experimental section would strengthen the paper ...

R1: Regarding the limited originality concern, while new fundamental algorithms or mechanisms are typical indicators of novelty, demonstrating novel insights using combinations of existing methods (e.g., that combining intrinsic motivation, prioritized world-model replay, and RaC significantly boosts the performance of world model agents in sample-efficiency settings, even in dense-reward environments) can be equally valuable and impactful, as such combinations often require significant effort and insight. A notable example is the influential Rainbow algorithm [1] (3238 citations), which integrates known techniques to achieve strong performance. Rainbow gained significant influence despite combining existing techniques and lacking algorithmic novelty. We believe that our work belongs to the category of "Rainbow-style" empirical contributions, and that the absence of new algorithmic mechanisms should not be considered a weakness, as the paper offers important empirical insights instead.

Regarding the stronger empirical section concern, similar published works in the world model agents literature (RL) evaluate their methods on 1-2 benchmarks, typically only Atari 100K or only continuous control [2][3][4][5][6][7]. Despite our compute limitations and disadvantage (see below), we managed to include 3 benchmarks. Hence, our empirical section is evidently stronger than the standard in the literature and we believe that it should not be considered as one of our work's weaknesses, but rather a strength.

In addition, when choosing benchmarks, several factors affected our decisions. First, the available baselines and compute. As a small and resource-limited lab, we conducted most of our research on a single RTX 4090 GPU. Full benchmark evaluations were performed only twice, toward the end of the project, on V100 GPUs. As a result, a major factor in our benchmark selection process was the availability of external baseline results, as we could not afford to evaluate other algorithms ourselves. In addition, we made our best effort to include as many benchmarks as we could afford within our budget, while also maintaining diverse benchmark modalities. As DreamerV3 is arguably the most significant baseline with existing results on DMC, and since DMC is already an established and popular benchmark, we decided to use DMC for continuous control tasks.

While exploring harder continuous control tasks can be valuable, we believe that it will not significantly strengthen the main contribution of our work which is the insight that the underexplored combination of intrinsic motivation, prioritized WM replay, and RaC is highly effective in sample-efficient world model agents, even in environments with dense rewards.

[1] Hessel, Matteo, et al. "Rainbow: Combining improvements in deep reinforcement learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

[2] Alonso, E., Jelley, A., Micheli, V., Kanervisto, A., Storkey, A. J., Pearce, T., & Fleuret, F. (2024). Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37, 58757-58791.

[3] Zhang, W., Wang, G., Sun, J., Yuan, Y., & Huang, G. (2023). Storm: Efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems, 36, 27147-27166.

[4] Cohen, L., Wang, K., Kang, B., & Mannor, S. Improving Token-Based World Models with Parallel Observation Prediction. In Forty-first International Conference on Machine Learning.

[5] Georgiev, I., Giridhar, V., Hansen, N., & Garg, A. PWM: Policy Learning with Multi-Task World Models. In The Thirteenth International Conference on Learning Representations.

[6] Micheli, V., Alonso, E., & Fleuret, F. Transformers are Sample-Efficient World Models. In The Eleventh International Conference on Learning Representations.

[7] Hansen, N.A., Su, H. & Wang, X.. (2022). Temporal Difference Learning for Model Predictive Control. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:8387-8406.

W2:

A key contribution is the modular multi-modality tokenization framework. However, of the three benchmarks used, only Craftax truly requires multi-modal input processing. To make the claim of general multi-modality more compelling, the paper would benefit from evaluation on additional multi-modal environments. The authors themselves note the scarcity of such benchmarks as a limitation with which I do not fully agree with. E.g. a simple robotic setup can be multi-modal where the agent gets the proprioceptive sensor data and RGB images as input (e.g. see ManiSkill).

R2: Regarding the second point, i.e., that a simple robotic setup can be multi-modal by including both the RGB and the proprioceptive signals, note that since each of these signals is sufficient for solving the task, an agent can essentially rely on one and ignore the other. Hence, positive results on such a benchmark are not truly indicative of multi-modality capabilities. For this reason, we believe that such a benchmark would not be adequate in our case, unfortunately. That said, we truly appreciate the creative direction and the reviewer’s intent to help us improve our multi-modality evaluation.

Regarding the first point, we agree that extensive evaluation on rich multi-modal environments remains a limitation, as we also acknowledged in our limitations section. However, our notion of a "multi-modality tokenization framework" also encompasses the ability to handle diverse uni-modal environments. Under our three-benchmark budget, already more comprehensive than the 1–2 benchmarks typical in this literature, we covered a wide range of modalities. Given that standard and well-established sample-efficiency benchmarks like Atari 100K and DMC are uni-modal, and since such environments are far more prevalent, demonstrating strong performance across them is arguably more important, and also provides concrete support for the framework’s multi-modality capabilities.

Therefore, while we recognize the limitation, we believe the lack of additional multi-modal tasks should not be considered a major concern.

Questions:

Q1:

The authors decided to discretize the continuous action space for the DMC tasks. Could you provide more insight into the potential limitations of this approach, especially for more complex robotics tasks that might require finer motor control than the ones tested?

A1: Discretizing continuous actions does not inherently limit controller precision, as this can be compensated by a shorter control interval or higher control frequency. However, there is a clear trade-off: large action spaces allow lower decision frequency but increase exploration cost, while smaller action spaces require more frequent decisions, increasing computational load. In our experiments, we did not encounter any issues related to action discretization.

A key insight in the context of world models is that the size of the vocabulary controls not only reconstruction quality but also the complexity of the dynamics modeling task, i.e., how difficult it is for the world model to learn and predict transitions. We found that using smaller vocabularies significantly improves sample efficiency, not only in continuous control but more broadly. While this may seem intuitive, we believe it is worth explicitly including in the appendix of our next revision, especially as it was also highlighted by other reviewers. We thank reviewer DVfk for raising this point.

Q2:

The authors write in the Appendix that the DMC experiments for Simulus take around 40h. Is this the run time for a single task? Compared to DreamerV3 which has a similar performance but takes only around 3h this is much longer.

A2: Training Simulus on DMC take around 40h on a V100 GPU, which is equivalent to roughly 20h on an A100 GPU, with which DreamerV3 was trained. In addition, the training of DreamerV3 on proprioception tasks was reported to last 0.3 (A100) GPU days, which is equivalent to 7.2 hours. Hence, the difference (20h vs 7.2h) is not as significant as mentioned.

Furthermore, as we mentioned in our limitations section, we believe that the current approach of discretizing individual features (adopted from existing literature) is inefficient, as it leads to excessive sequence lengths. Exploring more efficient solutions requires substantial efforts and is largely orthogonal to the contribution of our paper, hence we left it for future works. That said, we expect that more efficient tokenization strategies could substantially reduce the training cost of Simulus. Importantly, we believe this current inefficiency should not detract from the significance of our empirical findings.

Limitations:

The authors mention that the scarcity of multi-modal environments prevented them to evaluate further experiments with multi-modal environments. Many robotic tasks provide multi-modal observations; thus I do not fully agree with them.

We believe this concern has been addressed in R2 and kindly ask you to also consider our compute limitations and the supporting arguments outlined in the second paragraph of R1.

Training Simulus on DMC take around 40h on a V100 GPU, which is equivalent to roughly 20h on an A100 GPU, with which DreamerV3 was trained. In addition, the training of DreamerV3 on proprioception tasks was reported to last 0.3 (A100) GPU days, which is equivalent to 7.2 hours. Hence, the difference (20h vs 7.2h) is not as significant as mentioned.

The training times reported in the DreamerV3 paper are mixed up—the durations for visual control and proprioceptive control tasks should swapped. As a result, DreamerV3 requires only approximately 2.4 hours of training on proprioceptive tasks. Hence, the difference is still significant. Nonetheless, this reviewer appreciates the additional insights provided regarding potential possibilities for further training speed improvements.

A notable example is the influential Rainbow algorithm [1] (3238 citations), which integrates known techniques to achieve strong performance. Rainbow gained significant influence despite combining existing techniques and lacking algorithmic novelty. We believe that our work belongs to the category of "Rainbow-style" empirical contributions, and that the absence of new algorithmic mechanisms should not be considered a weakness, as the paper offers important empirical insights instead.

While I fully agree with the underlying point, the overall tone and presentation of the Rainbow manuscript differ from that of the authors’ submission. Nonetheless, the authors convincingly demonstrate that the integration of intrinsic motivation, prioritized experience replay, and RaC leads to performance improvements across a reasonable number of tasks. In light of this, I will adjust my score accordingly.

We thank reviewer nS4G for their time and effort in reviewing our paper, for their constructive and valuable feedback, and for the positive review.

We appreciate that reviewer nS4G acknowledged many of the strengths of our work, including the significant improvement in IQM over prior works (Atari 100K) and overall strong performance, the quality of our ablations study, and the importance of our results and insights to the community.

Weakness:

W1:

The computational cost of the model will be a problem. Just as the author said, "token-based world model agents remain significantly slower to train than other baselines in sample-efficient RL."

R1: While current token-based methods are indeed slower to train than some baselines (notably DreamerV3 and its variants), the overall training cost of Simulus remains reasonable. For example, DIAMOND, a recent baseline, requires nearly 3 days of training per Atari game on an RTX 4090 GPU, whereas Simulus completes training in approximately 11.9 hours under the same hardware. We observe that recent baselines generally incur higher compute costs, and we thank reviewer nS4G for prompting us to clarify this point.

In addition, we argue that this should not be seen as a major drawback of our work, for several reasons:

1. Most importantly, architectural inefficiency of TBWMs does not diminish our main findings: our ablations show that combining intrinsic motivation, prioritized world-model replay, and regression-as-classification substantially improves the performance of world model agents in sample-efficiency settings, even in dense-reward environments. These insights are valuable regardless of training efficiency.
2. Our implementation builds on REM with a naive RetNet, unlike the highly optimized RNN/CNN/Transformer baselines. There is substantial headroom for improving efficiency in TBWMs, though this lies outside the scope of this work.
3. While training may be slower, the resulting controller is lightweight and decoupled from the world model, enabling fast and cheap inference. This offers a practical advantage over methods like Dreamer, which require the full model at test time.

W2:

While this paper presents a system that demonstrates good performance and is empirically validated, its core contribution lies in the combined application of existing techniques within the TBWM framework. These components have been considered separately in prior work, and this work does not introduce entirely new learning objectives or architectural mechanisms. Thus, the novelty of this paper is systematic and empirical, rather than algorithmic.

R2: While new learning objectives or architectural mechanisms are typical indicators of progress, demonstrating novel insights using combinations of existing methods (e.g., that combining intrinsic motivation, prioritized world-model replay, and RaC significantly boosts the performance of world model agents in sample-efficiency settings, even in dense-reward environments) can be equally valuable and impactful, as such combinations often require significant effort and insight. A notable example is the influential Rainbow algorithm [1] (3238 citations), which integrates known techniques to achieve strong performance. Rainbow gained significant influence despite combining existing techniques and lacking algorithmic novelty. We believe that our work belongs to the category of "Rainbow-style" empirical contributions, and that the absence of new algorithmic mechanisms should not be considered a weakness, as the paper offers important empirical insights instead.

[1] Hessel, Matteo, et al. "Rainbow: Combining improvements in deep reinforcement learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

Questions:

Q1:

Can the authors quantitatively report the computational cost, such as GPU hours or FLOPs? The observed improvement may be because of the increased computation rather than the method itself, which could be unfair to the baselines. One way to ensure fairness would be to extend the baseline models' training steps to match the total computational budget.

A1: Simulus training times were reported in Appendix C (P. 26). In addition, we measured the run time of each component of Simulus, in each benchmark, using a RTX 4090 GPU, and also included a baseline where the intrinsic rewards and prioritized replay are disabled (No IR + No PR) to demonstrate the negligible overhead of these components.

Please note that some measurements differ slightly (shorter) from the reported values, possibly due to updated drivers. We will add the above runtime analysis to the appendix of the next revision of the paper.

To address the fairness concern, we note that our ablations clearly demonstrate that the observed improvements stem from the proposed components and their synergy. Disabling these modules leads to a significant performance drop, despite using the same compute and model size across all ablations.

For completeness, we also consider each of the benchmarks:

1. Atari 100K: the run times, model size (~35M), and compute of Simulus are standard and similar to those of the other baselines.
2. DMC: The only available baseline is DreamerV3, which used a 12M parameters model, and a replay ratio of 512. The replay ratio is the number of steps the model is trained on per collected environment step. In comparison, on DMC Simulus uses ~8.3M parameters and a replay ratio of 128 and 96 for the controller and world model, respectively. Hence, there are no fairness issues on DMC as well.
3. Craftax: As Craftax is a new benchmark (2024), the only world-model baseline we found was a closed-source concurrent work, which reports that their algorithm was trained on 8 A100 GPUs and performed ~978K world model update steps in total. In comparison, Simulus was trained on a single RTX 4090 GPU for approximately 4 days and performed ~975K world model update steps in total. Thus, if there are any fairness concerns on Craftax, they would arguably disadvantage Simulus, not favor it, as the baseline relied on substantially greater computational resources.

For transparency, as a small and resource-limited lab, we conducted most of our research on a single RTX 4090 GPU. Full benchmark evaluations were performed only twice, toward the end of the project, on V100 GPUs.

Q2:

Regarding prioritized replay, how sensitive is the model performance to the hyperparameter α? How is this hyperparameter determined?

A2: Due to compute resources limitations (see the last paragraph in A1 above), we could not extensively investigate the sensitivity of the hyperparameter α, unfortunately. The value we used was selected early in development based on satisfactory performance and was fixed thereafter. This stability suggests that the method is not overly sensitive to α; otherwise, it would have been unlikely to obtain strong results without tuning.

We thank reviewer KDoM for their time and effort in reviewing our paper and for their constructive and valuable feedback.

We appreciate that reviewer KDoM recognizes the significant improvement in sample-efficiency over REM and other baselines, and the quality of our ablations study.

Weaknesses:

W1.1:

Experiments are a bit suspect. Different environments are experimented with a different set of algorithms.

W1.2:

Especially problematic is that for craftax no Dreamer v3 is used! Noting that the original Crafter environment is designed for Dreamer v2 and the experiments shown in the Crrafter github repo author shows that Dreamer v2 clearly wins over any other algorithm.

R1.1: The reason for the differences in baselines between environments stems from the fact that except for DreamerV3, no other world model baseline is designed to handle multiple modalities. For example, the DIAMOND, STORM, and TWM baselines are all designed for image inputs and discrete action spaces. Hence, these baseline can not be evaluated on the other benchmarks.

The baselines we used for Craftax include one world model approach which was design specifically for Craftax, and a few other simpler baselines from the original Craftax paper which we included as additional reference points for the sake of completeness.

R1.2: While we would have liked to include DreamerV3 as an additional baseline for the Craftax benchmark, there are two limitations that prevented us from doing so:

First, as a small and resource-limited lab, we conducted most of our research on a single RTX 4090 GPU. Full benchmark evaluations were performed only twice, toward the end of the project, on V100 GPUs. As such, we prioritized benchmarks with publicly available baseline results, since we could not afford to evaluate other baselines ourselves.

Second, unlike Craftax, observations in Crafter are images. Hence, we preferred Craftax as (1) it offers both multi-modality and a native token modality which is absent from our other benchmarks, and (2) we already included an image-based benchmark (Atari 100K). While DreamerV2 and V3 have demonstrated strong performance on Crafter, the other baselines used for comparison are primarily model-free methods (e.g., PPO), which are known to be sample-inefficient. Since results for competing world model agents such as STORM or IRIS are not available, it remains unclear whether the Dreamer algorithms are truly superior on Crafter.

Overall, we have made an honest effort to include as many relevant and recent baselines in each of our results as possible within our limitations.

W2:

As far as I see authors have capped their intection budget ot 100k environment interactions and that itself is ok. It is ok to focus in sample-efficiency regime. But it would be also really good to see what happens model is let to use millions of steps. This kind of experiment(s) would give full disclosure to the reader.

R2: While we agree that large-scale experiments with significantly more data and compute would make for an interesting study, such experiments are unfortunately beyond our reach due to the strict compute limitations outlined in R1.2. It is worth noting that the interaction budgets in DMC and Craftax are 500K and 1M steps, respectively.

That said, we believe the core insights of our work remain broadly applicable and valuable, even in the absence of large-scale validation.

Questions:

Q1:

Have I undersrtood correctly that world model algorithms are typically considered to be on-policy? So then could you explain how prioritized replay buffer is used?

A1: Yes, current world model agents typically rely on on-policy reinforcement learning. While “prioritized replay” usually refers to sampling experience based on TD-error for RL updates, in our case it applies to the world model: we prioritize trajectory segments based on their world model prediction error. Concretely, during world model training, part of each batch consists of segments sampled according to this error.

Q2:

In my lab we have ran a multiple WM type experiments with atari benchmarks and we have found as are also found in Dreamer set of papers that Atari is actually sometimes quite hard. Reason is that in WM you represent state by a latent code and some Atari games have really small details. Especially hard is Breakout where many times difference in successive states is ball that is only one pixel wide. That details is very easily lost in WN type latent modeling. What about your proposed method? I see some examples in the appendices, but images are too small to see what is happening.

A2: The observations raised by reviewer KDoM are highly relatable. We have encountered similar challenges and devoted significant effort to addressing them. Based on our experience,the following strategies have proven effective in mitigating these issues, especially in low-data settings:

1. A modular design effectively mitigates optimization interference issues similar to those observed in RSSM-based methods (e.g., Dreamer), as demonstrated in Appendix B.1. Empirically, we found that such interference often leads to "blurred" reconstructions, an effect also noted in prior work, which exacerbates the difficulty of preserving fine details in the learned representations.
2. Unlike continuous auto-encoders, which offer limited control over the complexity of learned representations, discrete methods allow explicit control via the vocabulary size. We found that restricting the vocabulary is particularly beneficial in low-data regimes. This insight extends beyond image observations, for example, we observed similar benefits in DMC environments. Moreover, smaller vocabularies affect not only the reconstruction quality but more importantly simplify the dynamics modeling task, i.e., how difficult it is for the world model to learn and predict transitions.
3. We also conjecture that discrete methods enhance the separability of learned representations. This may be due to the inherently fixed and well-separated latent codes (e.g., in FSQ [1]) or due to regularization mechanisms in methods like VQVAE. Such separability can further aid the model in preserving fine details, such as those found in Atari games like Breakout.

We thank reviewer KDoM for raising this question. As these insights could be valuable to the community, we propose to include them in the appendix of the next revision of the paper.

[1] Mentzer, F., Minnen, D., Agustsson, E., & Tschannen, M. (2023). Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505.

We thank reviewer cNHU for their time and effort in reviewing our paper, for their constructive and valuable feedback, and for the positive review.

We appreciate the reviewer’s recognition of Simulus’s modular design, the effective combination of underused techniques, and its state-of-the-art sample efficiency across diverse benchmarks. They also valued our extension of TBWMs beyond vision-only tasks, the clarity and insight of our ablations, especially on intrinsic rewards, and our commitment to transparency through detailed documentation and open-sourcing.

Weaknesses:

W1:

Training is expensive: tokenizing continuous vectors blows up sequence length, and the authors admit TBWMs (including Simulus) train much slower than standard baselines. Even if modularity helps inference, the upfront cost is a real barrier for groups without large compute.

R1: While the training cost of current TBWMs is indeed higher than some baselines (notably DreamerV3 and its variants), it remains comparable to or lower than others. For instance, the DIAMOND baseline requires nearly 3 days of training per Atari game on an RTX 4090 GPU, compared to ~11.9 hours for Simulus. Moreover, as relatively recent methods, TBWMs still offer considerable room for optimization, as we also discuss below (A4).

Crucially, we believe that the architectural inefficiency of current TBWMs should not detract from the core insights of our work. Our ablations demonstrate that combining intrinsic motivation, prioritized world-model replay, and regression-as-classification significantly boosts the performance of world model agents in sample-efficient settings, even in dense-reward environments. These findings are meaningful regardless of implementation efficiency.

For transparency, as a small and resource-limited lab, we conducted most of our research on a single RTX 4090 GPU. Full benchmark evaluations were performed only twice, toward the end of the project, on V100 GPUs.

W2:

Multi‑modality isn’t fully stress‑tested: the lack of rich multi‑modal RL benchmarks limits how broadly they can validate the approach. Craftax is a start, but more varied tasks would make the claim stronger.

R2: We agree that extensive evaluation on rich multi-modal environments remains a limitation, as we also acknowledged in our limitations section. However, our notion of a "multi-modality tokenization framework" also encompasses the ability to handle diverse uni-modal environments. Under our three-benchmark budget, already more comprehensive than the 1–2 benchmarks typical in this literature, we covered a wide range of modalities. Given that standard and well-established sample-efficiency benchmarks like Atari 100K and DMC are uni-modal, and since such environments are far more prevalent, demonstrating strong performance across them is arguably more important, and also provides concrete support for the framework’s multi-modality capabilities.

Therefore, while we recognize the limitation, we believe the lack of additional multi-modal tasks should not be considered a major concern.

W3:

While the appendix discusses interfering objectives in RSSM optimization and provides preliminary results on PWM vs. PWM-decoupled, this section feels somewhat tangential to the core contributions of Simulus itself. While interesting, it could be refined to more directly link how Simulus's design avoids or mitigates such interference, rather than just observing it in a related model.

R3: These results were included to serve two main purposes:

1. To support the claim made in the introduction that separate optimization objectives reduce interference, by first demonstrating that such interference exists, and then showing that decoupling the objectives effectively mitigates it.
2. To motivate the modular design of Simulus.

We assume this behavior holds independently of the underlying architecture (RSSM in this case), as the interference arises from coupled optimization rather than architectural specifics. This directly supports the idea that Simulus’s modular design helps avoid similar optimization issues. We thank reviewer cNHU for this thoughtful observation. In the next revision, we will make the connection between these results and Simulus’s design more explicit to improve clarity.

Questions:

Q1:

Can you quantify the wall‑clock training cost of Simulus relative to key baselines (e.g., GPU hours per environment) and identify which modules dominate that cost?

A1: Simulus training times were reported in Appendix C (P. 26). In addition, we measured the run time of each component of Simulus, in each benchmark, using a RTX 4090 GPU, and also included a baseline where the intrinsic rewards and prioritized replay are disabled (No IR + No PR) to demonstrate the negligible overhead of these components.

Please note that some measurements differ slightly (shorter) from the reported values, possibly due to updated drivers. For comparison, on Atari, REM runs for ~ 11 hrs, and on DMC, DreamerV3 runs for ~7.2 hrs on a A100 GPU where Simulus would run for ~20 hrs on the same GPU (assuming that V100 is twice slower than A100). Note that Simulus is not highly optimized, and as stated in the limitations, vector tokenization is highly inefficient as it leads to excessive sequence lengths. On Craftax, the only world model baseline we found (concurrent work) uses 8 A100 GPUs for training. Hence, it is not directly comparable, as such setup requires significantly more memory.

Q2:

How sensitive is performance to the tokenization granularity for continuous vectors, and have you explored strategies to cap sequence length without hurting returns?

A2: Discretizing continuous actions does not inherently limit controller precision, as this can be compensated by a shorter control interval or higher control frequency. However, there is a clear trade-off: large action spaces allow lower decision frequency but increase exploration cost, while smaller action spaces require more frequent decisions, increasing computational load. In our experiments, we did not encounter any issues related to action discretization.

A key insight in the context of world models is that the size of the vocabulary controls not only reconstruction quality but also the complexity of the dynamics modeling task, i.e., how difficult it is for the world model to learn and predict transitions. We found that using smaller vocabularies significantly improves sample efficiency, not only in continuous control but more broadly. While this may seem intuitive, we believe it is worth explicitly including in the appendix of our next revision, especially as it was also highlighted by other reviewers. We thank reviewer cNHU for raising this point.

Regarding the second part of the question, we did not explore strategies to cap sequence length, as this would require substantial additional effort and is largely orthogonal to our paper’s main contributions. Our insights are independent of architectural (in)efficiencies, and we leave this line of work to future research. That said, we propose a simple direction in A4 below.

Q3:

Given the scarcity of multi‑modal RL benchmarks, do you have plans (or preliminary results) for additional domains beyond Craftax that stress different modality combinations?

A3: While we made sincere efforts to devise creative solutions to this challenge, we were ultimately unable to identify an adequate benchmark or evaluation protocol. One preliminary idea involved constructing a multi-modal environment by running two uni-modal environments in parallel. However, we found this approach difficult to justify, as it does not reflect real-world settings and can be effectively handled using separate uni-modal controllers.

We hope future benchmarks will build on modern video games where audio conveys crucial information. In these scenarios, agents that can leverage such cues gain a clear advantage.

Q4:

Where does Simulus underperform or fail to scale (e.g., very high‑dimensional observations or extremely long horizons), and what do you see as the most promising path to reduce the compute burden?

A4: While we have not explored large scale training due to compute limitations, we observed that in environments with continuous (vector) observations the scaling would be particularly inefficient, as mentioned in our limitations section, due to the inefficient tokenization that leads to excessive sequence lengths.

We believe this compute burden can be significantly reduced with improved design. In particular, architectural changes that better balance intra-token and inter-token information, e.g., by aggregating fixed-size groups of latents (akin to “patching”), offer a simple and promising direction. Furthermore, our implementation uses a naive RetNet variant, and we expect substantial efficiency gains from sequence model optimization.