We appreciate your comments, and address your concerns as follows:

1. Q: The key insight of the proposed method is similar to the recently released paper UrbanArchitect [A]. Although two works can be seen as concurrent works, I hope the authors can include a discussion.

We appreciate the reviewer highlighting this related work. We were unaware of UrbanArchitect [A], which was released just before the submission deadline. We will include a discussion of this concurrent work in our revision.

We would like to clarify that outdoor scenes are not necessarily more complex than indoor scenes.

• Indoor scenes feature dense, overlapping bounding boxes with fine-grained categories, while outdoor scenes typically have sparser, non-overlapping boxes with broader categories. Indoor scene generation methods must therefore handle object overlap and reason about relationships among numerous objects of varied categories.
• Outdoor scene generation allows for greater geometric approximation. As evidenced in Fig.17 of [A], while street views are reasonably generated, peripheral elements like houses and cars lack precise construction. Indoor scenes, conversely, demand accurate geometry for all objects within a closed room.
• Our distillation pipeline is optimized for efficiency and reduced memory usage. In contrast, [A] employs VSD and a two-stage generation process, incurring higher time and memory costs.
• As noted by HiFA [71] and mentioned at L124, the SDS used by [A] is essentially equivalent to the IN2N we employ in DreamScene, with only slight differences in loss calculation. Building on IN2N, we introduce a novel distillation pipeline that removes artifacts through periodic migration and enhances texture consistency via specialized loss design (Sec. 3.3, L165-184), achieving superior 3D consistency.

In conclusion, indoor and outdoor scene generation present distinct challenges and cannot be directly compared. Methods developed for outdoor scenes may not be directly applicable to indoor scene generation tasks.

2. Q: Visualization: it would be better to provide video results for better illustration.

Thank the reviewer for the suggestion. Please refer to the general response for our provided video visualization.

3. Q: Evaluation: authors should provide quantitative results.

We appreciate the reviewer's suggestion. We have provided the results in the general response.

4. Q: Training details: Some training details are missing (requirements for memory and time)

We thank the reviewer for the suggestion. We have provided the training details in the general response.

5. Q: Since the layout prior is used, would it be desired to use decomposed 3D representations for further editing?

While layout-controlled scene editing is indeed an intriguing concept, we believe it falls outside the scope of our current manuscript. Our focus, in line with existing indoor scene generation methods [11,24,55], has been primarily on the generation task. Given the time constraints of the rebuttal phase, we have not explored this avenue thoroughly. However, we acknowledge the potential of this idea and consider it a promising direction for future research.

6. Q: It would be better to include a method for automatic scene layout creation. E.g., Ctrl-Room utilizes a diffusion model [A]. LLM can also be used [B].

We would like to clarify that our method is designed to be layout-agnostic, accepting scene layouts as input regardless of their source. This flexibility allows for layouts to be pre-defined, user-generated, or even LLM-generated. Hence, our framework is compatible with various layout generation methods, including LLM-based approaches like Ctrl-Room [16]. This means that a text-guided layout generation module could be seamlessly integrated into our system to support fully text-guided scene generation.

Given the time constraints of the rebuttal period, we have not implemented this extension. However, we recognize its potential and plan to include it in our final revision. This addition would further enhance the versatility and user-friendliness of our system without altering its core functionality.

7. Q: 3DGS has been widely used in 3D generation models. Have you tried to use 3DGS to replace NeRF?

As mentioned in our introduction, our method is designed to be agnostic to the underlying scene representation, allowing for compatibility with various approaches. Our choice of NeRF was based on its proven stability in handling large-scale indoor scenes. However, we concur that integrating 3DGS could potentially offer comparable performance with improved convergence time. Due to the limited time in the rebuttal phase, we will try 3DGS in our revision.

8. Q: Is there a limit to the complexity of the scenarios a method can handle?

Thank the reviewer for considering this problem. We provide a discussion around failure cases and in the general response.

9. Q: Text2Room uses SD for missing parts inpainting. Have you tried to replace the SD model in Text2room with your finetuned SD model for inpainting for controllability?

We appreciate the reviewer's insightful suggestion.

• Indeed, our diffusion model could potentially replace SD in Text2Room for inpainting tasks, as it is specifically designed for indoor scene generation. Moreover, this substitution could introduce layout conditioning capabilities to the Text2Room framework, potentially enhancing its controllability.
• However, it's important to note that Text2Room relies on a depth-based naive 3D reconstruction approach. This fundamental aspect of its architecture would likely still result in challenges related to inappropriate or incomplete geometry, regardless of the diffusion model used for inpainting. Consequently, we anticipate that the results may not achieve the same level of quality as our current method.
• Due to the time limitation, we consider this investigation as future work.

We appreciate your comments, and address your concerns as follows:

1. Q: Overall, the scenes look more synthetic compared to MVDiffusion and text2room.

We'd like to address this concern of the reviewer by providing context for our approach:

• Prioritizing 3D Consistency: Our primary focus in this work is achieving 3D consistency in generated rooms, which we consider the most crucial and fundamental factor in 3D generation tasks, especially for complex room layouts. While this approach may result in some loss of detail or texture simplification, it consistently produces rooms with reasonable geometry that accurately matches the specified layout.
• Differences in Scene Representation:
  • The two methods mentioned have different scene representations from ours. MVDiffusion and Text2Room use RGB images from diffusion models directly and reconstruct scenes using depth estimators.
  • Our method employs Neural Radiance Fields (NeRFs), offering a different approach to scene representation.
• Dataset Choice and Impact:
  • Our diffusion model is primarily fine-tuned on the synthetic Hypersim [45] dataset, chosen for its high-quality layout data. This choice aligns with other recent works like ControlRoom3D [48].
  • In contrast, MVDiffusion uses a version of Stable Diffusion fine-tuned on the real-world ScanNet dataset, while Text2Room uses the original Stable Diffusion model.
• Flexibility for Improvement: The perceived quality of our results can be significantly enhanced by adjusting prompt instructions, distillation parameters, and camera trajectories. One example is provided in the General Response.

In summary, while our current results may appear more synthetic in some aspects, this is not an inherent limitation of our method. Rather, it reflects our current focus on geometric consistency and layout fidelity. Our approach offers significant flexibility for quality improvements while maintaining its core strengths in 3D consistency for complex scenes.

2. Q: In order to really demonstrate the uniqueness of this work, authors should provide more challenging qualitative results (especially complex scenes).

In our main paper, we aimed to show the diversity (i.e., different room types and styles) and layout-controllability of our model with the results. And considering the results shown in our paper, we believe those adequately demonstrate the superiority over the sota baselines [11,24,55].

Following your suggestion, we provide more results on complex scene generation in the rebuttal PDF and the linked video, as detailed in the General Response.

3. Q: Evaluation: authors should provide quantitative results.

We appreciate the reviewer's suggestion for quantitative results. We have provided the results in the general response.

4. Q: “our DreamScene achieves … and complicated indoor scenes, …”. This sentence seems confusing. Authors should clearly mention that (being able to generate) complicated indoor scenes is the key difference to previous works.

We thank the reviewer for pointing this out. We will revise and clarify this in our revision.

5. Q: Why couldn't we just use a random camera trajectory?

It is important to note that generating a reasonable camera trajectory, particularly for non-square rooms with complex layouts, is far from trivial. There are several key considerations:

• Complexity of Trajectory Selection: In complex room layouts, a random camera trajectory may intersect with objects or fail to cover all scene elements adequately. Even in 2D scenarios, collision-free trajectory generation remains a challenging research problem in robotics. There are dedicated papers investigating this problem in the problem of text-to-3D generation [A, B], suggesting its complexity.
  • [A] E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness, ECCV 2024
  • [B] CameraCtrl: Enabling Camera Control for Video Diffusion Models, arXiv 2024
• Consistency with Prior Work: Our approach aligns with existing methods in the field. For instance, Text2Room also requires user-provided camera trajectories. Other baselines that use panoramas have a built-in camera trajectory (rotation around a center point), but this limits them to simple square rooms with minimal occlusions.
• Advantages of current setting: By allowing for specified camera trajectories, we enable more precise control over scene exploration and visualization, which is particularly valuable in complex room layouts. This approach provides flexibility for future integration with advanced trajectory generation methods as they develop.

6. Q: Typo at line 134?

We will revise the manuscript and correct this typo.

7. Q: Training details: Lack of runtime data?

We have provided the training details in the general response.

8. Q: The method is limited to small scenes. It would be nice to provide results on a large outdoor scene, which does not anyway diminish the work's contribution.

We appreciate the reviewer's observation. Regarding the generation of large outdoor scenes:

• This is not a limitation of our method. While large outdoor scenes might intuitively seem more challenging due to their scale, our work prioritizes 3D-consistent, complex layout-controlled generation — a challenging aspect independent of scene size.
• Outdoor scenes generally present simpler conditions: fewer object categories, sparser object placement, and more predictable camera trajectories. For instance, most outdoor scene views follow a common paradigm (road and sky in the center, vehicles and buildings on the sides), simplifying the diffusion process.
• Our focus on dense indoor scenes aligns with the problem setting of several prior methods [11,24,55]. We consider outdoor scene generation beyond the scope of our current model, but we acknowledge it as an intriguing direction for future research.

We appreciate the reviewer's feedback on the need for more complex scene examples.

• During the limited rebuttal period, we have tried our best to generate two new complex scenes, as shown in this anonymous link: https://github.com/DreamScene3D/DreamScene/blob/main/rebuttal-visualization.jpg?raw=true. The last two rows showcase these new results.
• We would also like to point out that “complex scenes” are not just limited to non-square shaped ones – square scenes can also be complex so that the baselines will fail. In our Fig.3, the BBS-A/B/C contains many objects that are not adjacent to the walls and corners, which will also create occlusions in panoramas and make the depth map non-continuous and hard to predict. Therefore, these scenes should also be regarded as “complex scenes” where our DreamScene is the only work that can work well.

We hope these results could establish our DreamScene’s ability to generate rooms based on various complicated layouts, and address your concern about this. We commit to including these new examples, along with additional complex scene generations, in our future revision.

We appreciate your comments, and address your concerns as follows:

1. Q: The proposed method might not be very novel. The contribution of the paper seems mainly to be fine-tuning the diffusion model for semantic-guided image synthesis. The distillation stage is mostly similar to the existing methods.

We would like to highlight the novelty and contributions of our work:

• Pioneering Free Multi-View Trajectories: Our primary contribution lies in being the first to operate on free multi-view trajectories during scene generation. This approach contrasts sharply with baseline models [16,48,55], which rely on panoramas as intermediate states. As a result, we can now generate complex non-square rooms (e.g., L-shaped rooms, as demonstrated in supplementary Figure B and rebuttal PDF Figure.1), a capability beyond the reach of prior work.
• Other than the novel non-panorama-based framework, our DreamScene has the following technical contributions, as also summarized in L58~L64:
  • Bounding-Box Scene (BBS) Layout Condition. We introduce this novel layout condition, which offers greater versatility compared to layout conditions in previous work, inpainting-based approaches, and composition-based methods.
  • We support text-conditioned generation to generate rooms with different styles, with our fine-tuned diffusion model which produces multiview semantic-rich guidance, which is not different from prior works using panorama guidance.
  • We propose a novel NeRF distillation procedure that could remove the flocs via periodical migration and consolidate textures through unique loss design, as described in Sec. 3.3 Line 165-184. Additionally, we introduce multi-timestep sampling to enhance both semantic guidance and generation consistency.
• Unlike existing layout-guided methods, we commit to releasing our code upon acceptance, fostering transparency and reproducibility in the research community.

These contributions, particularly our non-panorama-based framework, represent significant advancements in the field of semantic-guided image synthesis and scene generation.

2. Q: Evaluation: authors should provide quantitative results.

We appreciate the reviewer's suggestion for quantitative results.

• Our initial decision to prioritize qualitative evaluation was based on the challenges in defining fair, accurate, and comprehensive metrics for evaluating 3D scene generation methods across all aspects. The development of such metrics remains an open challenge in the generation community.
• However, to address the reviewer's concern, we include quantitative comparison with baseline methods [11,24,55] on CLIP Score (CS), Inception Score (IS), and a user study on 3D scene consistency and overall visual quality. The experiment design follows previous work [48]. The detailed quantitative evaluation results are provided in General Response.

3. Q: Visualization: authors should provide videos to better show the 3d consistency in addition to multiview images.

Thank the reviewer for the suggestion. Please refer to General Response for details about our provided video visualization.

We appreciate your comments, and address your concerns as follows:

1. Q: Evaluation: Authors should provide quantitative results.

We appreciate the reviewer's suggestion for quantitative results.

• Our initial decision to prioritize qualitative evaluation was based on the challenges in defining fair, accurate, and comprehensive metrics for evaluating 3D scene generation methods across all aspects. The development of such metrics remains an open challenge in the generative AI community.
• However, to address the reviewer's concern, we include quantitative comparison with baseline methods [11,24,55] on CLIP Score (CS), Inception Score (IS), and a user study on 3D scene consistency and overall visual quality. The experiment design follows previous work [48]. The detailed quantitative evaluation results are provided in General Response.

2. Q: Overall quality: While the proposed method seems to perform better than the chosen baselines, the quality of the generated scenes seems to be relatively lower overall (especially Fig.4 second example).

We appreciate the reviewer's observation regarding the quality of generated scenes. We'd like to address this concern and highlight the strengths of our approach:

• Pioneering Complex Scene Generation: Our primary contribution lies in achieving 3D-consistent complex scene generation with flexible layouts and free camera trajectories. We are the first to generate non-regular room shapes with complex layouts, as demonstrated by the L-shaped rooms in our supplementary Figure B and the new T-shaped 1-bedroom apartment scene provided in the rebuttal (PDF and video). This capability surpasses baseline models [16,48,55] that rely on panoramas as intermediate states and cannot achieve such non-regular room generation with precise layout control.
• Geometric Consistency: While the second example in Figure 4 may have some limitations in color and texture, it maintains reasonable and consistent geometry that accurately follows the layout condition. This geometric fidelity is a crucial aspect where our method outperforms others (e.g., Text2Room, Set-the-Scene, MVDiffusion) that struggle to achieve comparable geometry quality or adhere to layout constraints.
• Improved results: We have found that the quality of generated scenes can be significantly enhanced by using more stylish and thematic descriptions. In the rebuttal PDF, we've included two variants of the same layout with improved text prompts. Our experiments show that using stylish descriptions like "ocean-themed" or "Van Gogh style" yields much better results compared to using specific words like colors to define the scene.

These improvements demonstrate the potential of our method to generate high-quality scenes when provided with appropriate prompts, while maintaining its unique capabilities in complex layout generation and geometric consistency.

3. Q: Lack of failure cases: successful and failure cases should be provided and discussed.

We thank the reviewer for the suggestion. Here, we provide a summary over the common failure cases of our model. Analysis shows that two failure cases of our model are Extremely Complicated Layout and Unmatched Prompts. For more details and visualization please refer to the General Response and figures in our rebuttal PDF.

4. Q: Training details: Lack of runtime data.

We thank the reviewer for the suggestion, and will clarify these points in our revised manuscript.

• As described in suppl. Sec. A, we use a duo-GPU training scheduling.
  • We use 2 A6000 GPU to conduct all our experiments and generally, each well-generated scene costs around 3-4 hours, the first GPU holds the process of diffusion model and the memory cost is ~6GB in FP16 with an image size of 512×768, another GPU holds the process of scene representation, and the memory cost is ~28GB (used Nerfacto from NerfStudio).
  • For NeRF training, we use a constant learning rate of 1e-2 for proposal networks and 1e-3 for fields. We will add those to the main paper.
• Note that some concurrent works cost more time, ShowRoom3D costs ~10h to produce a single scene (Tab.3 of their paper), UrbanArchitect costs ~12h and 32GB to produce a single scene (Sec.4.1 of their paper). Therefore, our DreamScene has higher efficiency than the baselines.

Dear Reviewers, here is the link to the anonymous video with our generated scenes. https://github.com/DreamScene3D/DreamScene/raw/main/rebuttal_video.mp4