Worldcraft

• The proposed method is limited in both theory and application.
  • Theoretically, the proposed method makes a very strong assumption that the parameter of the decoding process is known prior to the hierarchical modeling of the entire 3D world. This is an assumption that is too strong to be true technically. As for 3D scene generation, the decoding process always indicates the process that decodes the underlying representation to pixels, which is the primary focus for the majority of papers in the field of 3D generation. However, in this paper, the authors assume the decoder is known (at least through the demonstration on top of Minecraft, the decoder is known).
  • Application-wise, the authors use Minecraft as their proof-of-concept for hierarchical world modeling. Yet, the proposed method still requires lots of data for 3D scene generation. However, the data distribution in Minecraft is bounded by its procedural generation process. As a decoder-free method, there is no way for the proposed method to generate 3D scenes beyond the procedural generation. In other words, I do not see the motivation to use procedural generation itself for both quality and efficiency on Minecraft as a special case. Furthermore, whether the proposed method is applicable to other scenarios and platforms is also unknown.
• The evaluation cannot support the paper’s claim.
  • There is no proper evaluation or comparison against other methods or even some simple baselines to better demonstrate the effectiveness of the proposed method. For example, evaluations against procedural generation is needed as well as ablation studies for the number of hierarchies, the effect of errors on world-level pixel maps, etc.
  • As a generative model (as claimed), metrics like FID are recommended to measure.
• The related work is not properly discussed.
  • The authors seem to omit a majority of papers related with 3D scene generation. There are prior works (like GANcraft and SceneDreamer) working on Minecraft-style world translation and generation. Moreover, there are a couple of works (CC3D, GAUDI) focusing on layout-guided scene generation which shares a similar idea with the proposed method in hierarchical modeling.

1. Overall, the entire paper was difficult for me to understand. The authors claim to have three main contributions, but there is a lack of necessary explanations and experiments to support these claims. Using the DDPM diffusion model to generate MineCraft maps does not support the authors' proposed concept of worldcraft. Additionally, using hierarchical generation models in scene generation tasks is also a quite common strategy.

2. This paper lacks necessary comparative experiments and ablation studies to show the effectiveness and the importance of the proposed method. For comparative experiments, it can directly compare with traditional procedural modeling methods, or compare by generating height maps. For ablation

3. I think the authors have omitted some relevant references on related works regarding 3D scene/world generation like GANCraft, SceneDreamer, InfiniteGen and etc.

[1] Zekun Hao, et. al., Unsupervised 3D Neural Rendering of Minecraft Worlds, ICCV 2021

[2] SceneDreamer, et al., SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections, TPAMI 2023

[3] Raistrick, Alexander, et al., Infinite Photorealistic Worlds Using Procedural Generation, CVPR 2023

• The Conclusion argues that the method may be adopted for other, even continuous worlds, but the paper makes little effort to show how this would be done or that it is even possible.
• The use of "unprecedented scale and speed [...]" in the abstract must be specified. Clearly it is not entirely unprecedented as Minecraft itself is already capable of generating such worlds. So what exactly does unprecedented refer to?
• Some of the images are very dark (Fig. 1 Center Image and two Images in Fig. 8).
• Fig. 7 could make use of less examples while giving each example more space to allow for larger images.
• Fig. 6 uses a hard to differentiate color-coding, this hampers accessibility.
• Fig. 6 also contains a large generation artifact in the form of an entirely lava chunk. This remains entirely unaddressed in the paper.
• Fig. 4 shows too small images.
• Fig. 5 uses a primarily white image on a white background. This limits the comparison. The figure also lacks image-to-caption labels.
• Chapter 3 "Worldcraft" mixes statements of previous works (e.g., General Representation Theorem), descriptions of properties (e.g., Exchangeability) and definitions (e.g., their Implicit Method in one large description block).
• The citation style is too verbose.
• The paper fluctuates between using world in cursive, as to the own definition of the authors and "world" without cursive. It is however not always clear which is meant when. Perhaps a different term should have been chosen or the paper should follow more carefully the chosen formatting.
• The paper is unscientific in it's choice of words at times: Line 64: "In this paper, we take the concept of world seriously". What does "taking it seriously" mean?; Line 150: "This is shown [...]" when there is just an example/demonstration of Exchangeability and no proof of any sorts; Line 213: "We realise p_phi(Phi^2) with a Denoising [...]" Generate? Predict?; Realise is an odd choice of word here;
• The title "Worldcraft" is overly vague and all encompassing. It is followed by a general throughline of hubris in the paper, notable examples including: Line 64: "In this paper, we take the concept of world seriously"; Line 69,70: "Thus richly complex worlds rather than mere scenes are achievable [...]"; Line 75: "At the core of our technical contribution [...] this paradigm shift [...]"; Line 423: "This potentially revolutionary set-based approach [...]".
• Line 303 claims a chunk is a 16x16x16 block. This appears to be incorrect: a chunk extends from the bottom to the top of the world, making it a 16x16x384 block.