PoE-World: Compositional World Modeling with Products of Programmatic Experts

Thank you for your positive feedback! We are glad the reviewer finds our Montezuma's Revenge result impressive. We address the raised concerns below:

Concern 1: My biggest concern is that while the authors evaluate on Atari games, they end up working on a simplified state space, not on the direct pixel space. So part of my concern here is: how different is working on this "simplified" Atari setting from working on a complex grid world? Is it the dynamics? The number of objects? It would be good to get some sense here in order to understand how big of a step the authors are taking away from toy grid worlds.

OCAtari is much “noisier” than gridworlds, and much higher-resolution, involving fine-grained motion and physics. It is also partially observed. Handling partially observed stochastic environments with fine-grained physics is an important step toward making these techniques applicable in the real world.

To illustrate the complexity of OCAtari, we include below two example trajectories from Pong. Here is a trajectory (displaying just the Pong paddle’s velocity y and the action) from a real Pong gameplay (Note that NOOP stands for NO OPeration):

paddle velocity = 0 
-> action = RIGHT 
paddle velocity = -6 
-> action = RIGHT 
paddle velocity = -8 
-> action = NOOP 
paddle velocity = -15 
-> action = NOOP
paddle velocity = -2 
-> action = NOOP
paddle velocity = -2 
-> action = NOOP 
paddle velocity = 0

And here is a trajectory in the same gameplay but at a different timestep:

paddle velocity = 0 
-> action = RIGHT 
paddle velocity = -3 
-> action = RIGHT 
paddle velocity = -18 
-> action = NOOP 
paddle velocity = -8 
-> action = NOOP
paddle velocity = -6 
-> action = NOOP
paddle velocity = -1 
-> action = NOOP 
paddle velocity = 0

Notice how the two trajectories lead to different sequences of velocities despite the same action sequences. In fact, in the first trajectory, performing NOOP when velocity = -2 can lead to two possibilities: velocity = 0 or -2. Performing NOOP also does not immediately stop the paddle from moving; the paddle only stops moving (velocity = 0) after a few timesteps.

Concern 2: One of the biggest concerns I have is that a lot of the core parts of building the programmatic world model is left to the appendix. I understand the authors would do this for the planning part which is not core to the contribution, but the way to construct the programs that are used as experts is completely missing in the main body of the paper. This feels strange as it is core to the method.

Thank you for pointing this out. If accepted we will use our extra page to move those details to the main text.

Concern 3: Some things that could further improve the quality of the paper:

1. I understand getting this method to run even on a single Atari game requires substantial work. That being said, only two Atari games does feel on the low side. Maybe the authors could add a few environments from the Atari-5 subset
2. In table 1, there are no baselines that also use the demonstration that is used for PoE. It would be good if the authors could include a behavioral cloning baseline as well as maybe one offline RL baseline that uses this demonstration.

We agree with the reviewer. We were not aware of Atari-5 when we wrote the paper, and it would have been a good addition to the paper. In table 1, WorldCoder also uses the same demonstration that is used for PoE-World, but yes, we use an on-policy algorithm (PPO) as a model-free RL baseline, so this baseline could not take advantage of the demonstration. An offline RL baseline may help with Pong, as the demonstration shows the player scoring once, but for Montezuma’s Revenge, the demonstration never shows positive score, so we do not expect off-policy RL to help in this case. We believe that the experimental results will convey the same message even with an off-policy RL baseline that uses the demonstration.

Concern 4: I found the paper lacking in clarity in the following (sometimes more minor) places:

1. Line 105: does "projection of $o$ onto feature $f$" mean "indexing $o$ with $f$"? From what I understood, $o$ is a vector of state attributes (e.g. velocity) and so $f$ just serves as an index into this vector? If this is the case, it might make sense to reword this to improve clarity.
2. I found the paragraph starting on line 106 a bit confusing. Could the authors make the argument being made here a bit more precise / formal? Do the authors mean instead of conditioning on $o_{1:t}$, using some latent variable $z_t$ to represent this?
3. The hard constraints paragraph starting on line 114 doesn't feel expansive enough to understand what's really going on or why the hard constraints are necessary. Including an example of a hard constraint here would go a long way.
4. Are the multi-timestep predictions being discussed in equation 4 used anywhere? If not, I'd recommend leaving this part out and use the space it provides to address some of the other comments.
5. Line 144: "We then add noise to the distributions ...". What kind of noise?
6. Why does the equation on line 166 not use the same independence assumptions as the one on line 4?
7. Line 172: how exactly are the demonstrations created? Are they random agents? And if not, how much work went into creating these demonstrations? The details here could reflect on the practicality and assumptions of the method.
8. Figure 3: it would be helpful here if screenshots of the original environments were shown as well so it's easy to see the differences.
9. Are the numbers after +- in the various tables standard errors? Or standard deviations? Or something else? And how many seeds were used to compute these? And does each seed regenerate the programmatic experts?
10. In Table 2 and 3, how are the prediction accuracies defined? Are they exact match?
11. Table 4 reports "planning successes" but as far as I could tell this is never defined. What does "planning successes" mean?

1. Yes, we will reword it.

2. Yes, The POMDP formulation would condition on a latent variable $z_t$ instead of $o_{1:t}$. The issue is that $z_t$ would be an output of a learned component; it would be an output of $z_t = f_\phi (o_{1:t-1}, a_{1:t-1})$ where we are learning $f_\phi$. Since $f_\phi$ would be changing, $z_t$ would also change, and so all $p^{expert}_i$, which takes $z_t$ as input, needs to change correspondingly

3. We display the learned constraints in Table 10. An example is something like player_obj.bottom_side == conveyer_belt_obj.top_side

4. It is used in the PoE-World algorithm evaluated in the paper.

5. Small epsilon value: 1e-6.

6. Line 166 is meant to formalize how world models can be used in general (not specific to our work), so we did not include the independence assumption we use in equation 4 there.

7. They are not random agents. We do a few iterations of demonstrations to make sure it covers all the causal rules of the environments. The important thing here is that the demonstration shows examples of the causal rules needed to do well in the environment (for example, the player falls when its feet are not supported, touching the skull kills the player, etc.). Creating this demonstration is not hard for humans, but a random agent would not be sufficient.

8. Thank you, that is a good point. We will try to include the original screenshots in Figure 3 if there is enough space, if not, we will put them in the appendix.

9. They are standard errors. We use three seeds. Each seed does regenerate the programmatic experts :)

10. Yes, they are exact match

11. Thank you for pointing it out, we will fix it. Planning successes for Montezuma’s Revenge mean successfully grabbing the key.

Q1: In the definition of $Z_f$ in the equation after line 105, is it tractable to iterate over $o_f$? I guess the authors are using some discretized values with some range for velocity etc? And if not, how are continuous values handled?

In OCAtari, the velocities are discrete, corresponding to per-pixel movement

Q2: On line 132, how expensive is it to repeat step 1 - 3 every time the observed trajectories get updated?

Not expensive – we optimize the code so that it only synthesizes experts for newly added observations, and also we skip synthesizing experts for observations that are already well-explained by the current world model.

Q3: Line 140: "x/y velocity and visibility". Do the observations not include bounding boxes / positions of objects? And what does visibility mean?

The world model takes as input an observation with bounding boxes and positions, and so it only needs to predict velocities and visibility (whether the object gets deleted). Object sizes do not change in Pong and Montezuma’s Revenge, so world models do not need to predict bounding boxes.

Q4: In Table 2 and 3, why are the prediction accuracies typically better on the alt environments? This feels a bit counterintuitive.

The prediction accuracies are on random frames (generated by random actions). It is possible that random actions in the original environment may lead to unseen/unusual situations easier than in the alternative environments.

Q5: On line 226, are the 4000+ lines of code "active" or is most of it pruned? And also, how many experts does this typically consist of? It would be great if the authors could give some insight / statistics like this on the resulting world model programs.

They are all active (the pruned programs are deleted from the world model and do not count words in the 4000+ line of code). There are usually around 300-500 experts.

Thank you for your feedback! We address the raised concerns below:

Concern 1: The main goals should be to demonstrate (1) how well the world model reflects the actual dynamics of the environment… However, I interpret Figure 4 primarily as evidence of data efficiency

We believe Table 2 (Next observation prediction accuracy vs baseline) and Table 3 (Next observation’s object attributes prediction accuracy vs baseline) is what the reviewer is looking for.

Concern 2: (2) how the world model supports the learning process of downstream models… Figure 5 presents interesting results relevant to research question (2), but it is limited in scope

We consider a broader range of ways world models can support decision-making: Both learning (RL) AND planning. Our main experiments, Table 1 and Figure 4, analyze planning, while Figure 5 shows how the world model supports downstream learning via RL.

We focus more on planning than RL learning because planning is usually seen as having stricter requirements on the accuracy of world models, as errors can accumulate when an agent plans a long sequence of actions before acting. Pre-training model-free RL on world models (like Dreamer [1]) is usually easier to get to work and see gains of world models.

Concern 3: Further, the Pong game and Montezuma’s Revenge exhibit very different behaviors in terms of learning: the PPO performs better in Pong, whereas in Montezuma’s Revenge the other methods fail to learn.

This is because Montezuma’s Revenge is much harder than Pong. There is a consensus among RL researchers that Montezuma’s Revenge is one of the hardest Atari games for RL. Go-Explore, Nature 2021 [2] calls it a grand challenge. Many papers explicitly mention Montezuma’s Revenge in their title/abstract [3, 4]. Reviewer LjAF notes that the Montezuma’s Revenge result is impressive.

Model-free RL generally fails on Montezuma's Revenge, unless trained on an exorbitantly large number of environment interactions.

Concern 4: program synthesis could be an important building block. However, applying this work to a general case could be challenging on how do people know the objects, actions, and potential interaction results? Can you elaborate further on how to get these building blocks in a real world scenario?

All these building blocks (perception, planning/acting, and world modeling) have their own dedicated research areas, and one paper cannot do everything at once. We believe that building AI agents that can learn, reason, and adapt flexibly in the real world requires progress on all of these building blocks, but in this work, we target the world modeling problem.

Then, to get each of these building blocks to work in a real world scenario, we think researchers should gradually work on more and more complex domains over time. Previous work in symbolic world models only considered gridworlds and text worlds. We go beyond that and work on non-gridworlds like OCAtari: Easier than raw Atari, but much more complex than gridworlds. Ultimately, these games serve as testing grounds for research, and this work should be seen as a stepping stone toward more general agents.

Concern 5: Can you elaborate more on how the expert programs are synthesized? Specifically, how is the world domain defined and passed into the large language model? What is the prompt strategy?

We have put details on how programs are synthesized and how input is turned into natural language to be passed into LLMs in Appendix A.1.

Q1: What does the discrepancy [between Pong and Montezuma’s Revenge] indicate for the general applicability and limitations of the approach?

Montezuma’s Revenge is a long-horizon sparse-reward problem (see our reply to concern 3). The discrepancy indicates that model-free RL struggles in such settings (relative to planning). Although this is well known, it strengthens the argument for learning a world model: Only with such a model is planning possible.

References:

[1] Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2025). Mastering diverse control tasks through world models. Nature, 1-7.

[2] Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2021). First return, then explore. Nature, 590(7847), 580-586.

[3] Salimans, T., & Chen, R. (2018). Learning Montezuma's revenge from a single demonstration. arXiv preprint arXiv:1812.03381.

[4] Aytar, Y., Pfaff, T., Budden, D., Paine, T., Wang, Z., & De Freitas, N. (2018). Playing hard exploration games by watching youtube. Advances in neural information processing systems, 31.

We thank the reviewer for engaging in this discussion! We clarify the reviewer's questions below:

Besides table 2 and table 3, I wonder whether there is a more direct way comparing the ground truth rules against the actual rule in the model? For example, the ground truth has 300 rules, and among the 3000 rules that is generated by the LLM, what is the precision and the recall?

There is a ground truth source code written in Assembly (as opposed to Python like our world model) for Atari 2600's Montezuma's Revenge (we are told not to paste links in rebuttal, but it is pretty easy to find this on Google search if the reviewer is curious). The program is 4721 lines long. It is unclear, however, how to count how many "rules" there are, as the source code is written in a single Assembly program. so we do not know if there is a good way of calculating rule precision and recall.

In addition, even if the ground truth source code could somehow be structured into rules, it is also possible that a single ground truth rule could be broken down and represented by multiple rules generated by LLMs. For these reasons, it is hard to have an apples-to-apples comparison, and so we stick to next prediction accuracy and downstream learning metrics as an indicator of how good the learned world model is for this paper.

The major concern I have here is about the potential scaling problem. "World models produced by PoE-World consists of 4000+ lines of code for Montezuma’s Revenge compared to WorldCoder’s less than 100 lines. " Compare to the ground truth program, how much is the redundancy? Do you think scalability could be a problem, for example, extending this experiment to minecraft?

For world models produced by PoE-World, redundancy is handled by the following mechanism: once a behavior is well-explained by some experts, the algorithm would not synthesize more experts to explain that behavior. Thus, there might be redundant experts if similar experts are all synthesized at the same timestep. However, in the next timesteps, the corresponding behavior would be well-explained, so our algorithm would not try to synthesize more experts to explain that behavior

It is possible that one can try to manually code up a world model that contains less number of lines than PoE-World's world model, just like how a software engineer can refactor code to make it more compact. Directly removing experts in PoE-World's world models, nevertheless, will cause a drop in the likelihood value of observed trajectories. This is the result of how redundancy is handled as explained in the paragraph above -- PoE-World only adds experts if they would help increase likelihood of observed trajectories.

In terms of scalability, the runtime of the algorithm is linear with respect to the number of experts, so PoE-World should be able to scale well. To build model-based agents for Minecraft, we suspect that other components of the agents, apart from world modeling, would present big obstacles: perception, decision-making, etc. We leave these other components for future work.

Please do not hesitate to ask any follow-up questions!

Thank you for your positive feedback! We are glad the reviewer finds this work interesting. We answer the reviewer’s questions below:

Question 1: Are there cases where decomposing the world model as done in this paper is not a good idea? Even in classical control theory, when the world is modeled as, say, a linear system, dx/dt = Ax + Bu, where x is a vector of state variables, the world model is decomposed into constituent parts since each x_i is predicted separately using a different equation; however, the predictions are still conditioned on the current estimate, x, of the full state of the system. What is the input to the Python program -- I think it is a history of past observations, but I am not sure how long a history. I also understand that the input does not include the current estimate of the state of the system. Is that right?

We think there are two different questions here. First, whether decomposing the world model is always a good idea; and second, whether conditioning predictions only on the past history (but not the other state variables) is too strong of an assumption.

For the first question, we think decomposing the world model should never really hurt – having a single program as a world model is a special case of PoE-World (with number of experts = 1).

For the second question, this relates to a research direction we’re currently exploring. Factoring the state such that each state variable does not depend on the current values of other state variables works in this work, because we’re given the ground truth state variables (x/y position/velocity and visibility of each object).

However, if we do not have access to these ground truth state variables and we have to learn the state variables, then state variables can become entangled (like “player y-velocity” correlates with “player’s feet supported by platform”), so the factored state assumption no longer holds. In our ongoing work, we’ve relaxed the independence assumption and added experts that model correlations between state variables. This gives interesting technical puzzles, as we cannot easily normalize the final probability distribution, so we need to use training techniques from the area of energy-based models (EBM) [1]; we’re currently exploring this direction in follow-up work.

Question 2: I am not surprised that hard constraints do not affect next observation prediction -- I can see that -- but it is surprising that they contribute to planning success. Is the planner only using the learned model (learned using the hard constraints), or is it using the learned model plus the hard constraints separately? In other words, are the hard constraints used in two different places in the 2nd row of Table 2?

The planner does not use the hard constraints separately; it only uses the world models, whose outputs are affected by the hard constraints. We also find this somewhat surprising, and we suspect that hard constraints tend to play an important role when the world model has to generate a multi-timestep sequence of observations (as opposed to one-step prediction) where errors can accumulate.

Question 3: (1) Is the summary of the paper above accurate, or have I misrepresented the work?

And the same question as (1), but for the evaluation part of the review (titled the strengths and weaknesses).

The written summary is a good and concise summary of our work :)

The evaluation part is also accurate. We were not aware of the connection to older systems theory work, but the learning algorithm you described does sound very similar to what we have in this paper. We see this work as relating back to many older works but done in the modern age where LLMs can be used as powerful program synthesizers, and we demonstrate how such methods can work on a domain more complex than what symbolic world models have previously considered.

References:

[1] Song, Y., & Kingma, D. P. (2021). How to train your energy-based models. arXiv preprint arXiv:2101.03288.

Thank you for your feedback! We address the raised concerns below:

Concern 1: There is a lack of novelty of this approach. I see its promise but at its core its WorldCoder but with multiple smaller synthesizers that gets aggregated together.

Quick clarification before the actual reply: it is the programs that get aggregated, not the synthesizers, but this is probably a typo from the reviewer. The actual reply is below:

We believe our novelty lies not in the idea of aggregation itself, as we indeed cite many works advocating the idea of composing smaller parts to obtain a sophisticated whole. Our novelty is in how the aggregation is done: we turn small, deterministic programs into simple distributions and aggregate them stochastically to obtain sophisticated, stochastic world models. We highlight key differences between WorldCoder and PoE-World in a table below:

Concern 2: Even though compositionality is a major motivation behind the approach, there were no formal definitions for compositional generalization and experiments really emphasizing that.

By “compositional generalization”, we mean two things:

1. Generalization to object combinations not seen during training: what Cosmos, ICLR’24 [1] calls “entity composition”
2. Factorization of the world model into rules conserved across object types: What Cosmos calls “relational composition”, and what RIMS [2] calls “independent mechanisms”.

Experiments in Table 1 and Figure 4 evaluate both forms of compositionality: The Alt environments evaluate entity composition, and the comparison with the WorldCoder baseline evaluates relational composition. We will revise the paper to clarify this point about compositionality.

Concern 3: This approach would extend well to domains with large amounts of simple rules that are object-centric. However, when the rules cannot be broken down into such modular, small components, when the scene is not as distinct in terms of its labeled objects, or when the size of the rule set is less than m, I am unsure of how this approach will perform.

Although the experts in this work are relatively simple, they could be more complex: we can in fact even use WorldCoder wholesale to generate experts for PoE-World. So even if the environment cannot be described by modular rules, our approach can just fall back to WorldCoder. We think though that many environments are mostly modular, and note that experts can model inter-object interactions.

Concern 4: There were mentions to partial observability but no justification or expansion on it. This is a key point that got completely overlooked

OCAtari, the domain we use in this work, is partially observed: the current scene and the current action alone cannot describe the next scene. This is true even for Pong. Looking to the future, the real world is partially-observed, so handling this complexity is an important step for program world models. We will revise the paper to clarify that OCAtari is partially observed.

References:

[1] Sehgal, A., Grayeli, A., Sun, J. J., & Chaudhuri, S. (2024). Neurosymbolic Grounding for Compositional World Models. ICLR.

[2] Goyal, A., Lamb, A., Hoffmann, J., Sodhani, S., Levine, S., Bengio, Y., & Schölkopf, B. (2019). Recurrent independent mechanisms. arXiv preprint arXiv:1909.10893.

Thank you for your feedback! We address the raised concerns below:

Concern 1: The reviewer is concerned that we assume an object detector and a planner, limiting generalization to other domains. The reviewer wonders how hard it is to satisfy these assumptions. (See concern 1.1 and 1.2 for specific assumptions)

We agree with the reviewer’s concerns on the limitations of used object detectors and planners. It is indeed hard to obtain these assumed components. We note, however, that these assumptions are not specific to our work and should be seen as generic limitations of symbolic world models line of research. We elaborate on each assumption below:

Concern 1.1: The paper assumes an existing object detector

Object detectors are an important assumption, and one often made in robot decision-making. All our baselines use the same structured input representation, rather than pixels. Under this comparison, we make significant progress relative to previous symbolic world models (Worldcoder NeurIPS 24 [1], Code World Models NeurIPS 24 [2]). While WorldCoder and CWM, both published at NeurIPS 24, work on gridworlds and text domains, we work on OCAtari (Atari + Object detector), which is much more complex.

Other works learn symbolic world models from pixels [3], but do not scale to dynamics as complex as those we consider. We hope that publishing our paper paves the way for combining our method with such works, which would help address this concern.

Concern 1.2: seems like the planner might be hard to generalize across domains

Planning is a very hard problem with its own research community. The planner used is not part of our method/contribution: we use the same planner for both our work and the baseline (WorldCoder) we are comparing against. In the paper, we denote our agent as PoE-World + Planner to emphasize this point. We further note that a planner is not always needed: As figure 5 shows, we can run model-free RL (PPO) in the world model and see performance gains. We acknowledge that we use a more domain-specific planner, but this was necessary to check that our world models were accurate enough for long-range planning.

Concern 2: Whether it (this work) could bring gains in cases where perception is not necessarily a significant challenge (but world modeling can still be, regardless)

Thanks for pointing this out: We focus on world modeling because it still presents a significant challenge even when perception and planning are taken as given, as evidenced by the poor performance of the baselines. Our main experiments (Table 1 and Figure 4) serve to demonstrate that given reliable perception, PoE-World outperforms other methods, even in the presence of fine-grained physics, stochasticity, and partial observability.

Q1: Are there other existing environments with less challenging perception where PoE-World would be able to scale better than WorldCoder in terms of the complexity of the world model?

Yes! Robot decision-making, specifically Task and Motion Planning (TAMP), is a prime example. We think Atari is adequate for a first paper, though: Montezuma’s Revenge has served as the centerpiece of several well-known papers [4, 5].

Q2: How large does the world model get on Pong? ... Is the main advantage here against WorldCoder the fact that the learning method is compositional, even if the total size of the world model might not be as large as in MR?

Pong has surprisingly complex physics, requiring a world model with ~3000 lines of code. Compositional factoring allows learning independent programs for each object, and even breaking the dynamics of a single object up into several programs.

To illustrate the complexity of Pong, we include below two example trajectories from Pong. Here is a trajectory (displaying just the paddle’s velocity y and the action) from a real Pong gameplay (Note that NOOP stands for NO OPeration):

paddle velocity = 0 
-> action = RIGHT 
paddle velocity = -6 
-> action = RIGHT 
paddle velocity = -8 
-> action = NOOP 
paddle velocity = -15 
-> action = NOOP
paddle velocity = -2 
-> action = NOOP
paddle velocity = -2 
-> action = NOOP 
paddle velocity = 0

And here is a trajectory in the same gameplay but at a different timestep:

paddle velocity = 0 
-> action = RIGHT 
paddle velocity = -3 
-> action = RIGHT 
paddle velocity = -18 
-> action = NOOP 
paddle velocity = -8 
-> action = NOOP
paddle velocity = -6 
-> action = NOOP
paddle velocity = -1 
-> action = NOOP 
paddle velocity = 0

Notice how the two trajectories lead to different sequences of velocities despite the same action sequences. In fact, in the first trajectory, performing NOOP when velocity = -2 can lead to two possibilities: velocity = 0 or -2. Performing NOOP also does not immediately stop the paddle from moving; the paddle only stops moving (velocity = 0) after a few timesteps.

Q3: How often do the synthesized experts overlap in behavior? Is there any pressure for them to be orthogonal? What happens if the same (useful) expert is generated multiple times - do all of its copies get kept by the weight-based pruning procedure, since they'd have similar weights?

All copies do get kept, so there might be redundant experts if similar experts are all synthesized at the same timestep. However, in the next timesteps, the corresponding behavior would be well-explained, so our algorithm would not try to synthesize more experts to explain that behavior.

Q4: I'm confused about the "Multi-timestep predictions" description. The text describes the idea here of assuming independence to phrase Multi-timestep predictions as a product of next-timestep experts. But Equation 4 contains $p^{expert}(o_{t+k}|o_{1:t}, a_{1:t})$. When $k > 1$, this is not a next-timestep expert anymore. Don't you need to also condition on an accumulating sequence of observations and actions (up to $t+k-1$)?

Yes, for multi-timestep experts, we make an extra assumption, so we do not condition the predictions on the sequence of observations and actions between time $t$ and $t + k$.

Q5: Did the authors observe any experts that become too slow to compute as the trajectory grows? Was the unpredictability of running times ever a problem for the scalability of the learned world model?

Runtime is linear with the number of experts, so it is quite predictable. The experts tend to be quite simple, so they run fast. Appendix A5 includes runtime.

Q6: In (3) (L119), is the aggregation of hard constraints not supposed to be a conjunction (and) as opposed to disjunction (or)? If it is really a disjunction, why are they "hard" constraints if you only need one of them to be satisfied? (for instance, leaving the screen downwards wouldn't necessarily be a "hard" constraint unless all the constraints prevent it?)

As discussed in appendix A.1, in Equation (3), we choose to use a disjunction rather than a conjunction because the physics in video games can be peculiar and unrealistic—a player might have their body overlap with a platform when climbing down a ladder attached to that platform. The constraints are still hard in the sense that each of them is deterministic, not probabilistic. In certain situations, we agree that a conjunction could also be a reasonable choice.

References:

[1] Tang, H., Key, D., & Ellis, K. (2024). Worldcoder, a model-based llm agent: Building world models by writing code and interacting with the environment. Advances in Neural Information Processing Systems, 37, 70148-70212.

[2] Dainese, N., Merler, M., Alakuijala, M., & Marttinen, P. (2024). Generating code world models with large language models guided by monte carlo tree search. Advances in Neural Information Processing Systems, 37, 60429-60474.

[3] Liang, Y., Kumar, N., Tang, H., Weller, A., Tenenbaum, J. B., Silver, T., ... & Ellis, K. (2024). Visualpredicator: Learning abstract world models with neuro-symbolic predicates for robot planning. arXiv preprint arXiv:2410.23156.

[4] Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. (2016). Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29.

[5] Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2021). First return, then explore. Nature, 590(7847), 580-586.

We thank the reviewer for engaging in this discussion, and we are glad that we have helped clarify some of the reviewer's concerns!

This is indeed surprising. Does "required" just mean that this is the size of the expert that PoE-World synthesized, or do the authors think that if you were to hand-write a similar world model as a monolithic program it would have a similar size? (it would be helpful if there was a "ground truth" implementation of the world model alone to compare to, but I assume there's none for Atari games).

There is in fact a ground truth implementation for Atari 2600's Montezuma's Revenge that we found online (we could not find it for Pong). It is written in Assembly, and the total line count is 4721 lines (we are told not to paste links in the rebuttal, but it is pretty easy to find this on Google if the reviewer is curious).

We do suspect that one can manually hand-write a similar world model with a smaller size, just like how a software engineer can refactor code to make it more compact. At the same time, we do note that for PoE-World's world models, directly removing an expert from a world model would cause a drop in the likelihood value of the observed trajectories. At each point, PoE-World only adds experts if it helps increase the likelihood value of the observed trajectories (as explained in our reply to Q3)