Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraints

Thanks for your comments. We address the weakness and questions below:

• Variety of text prompt and generated room type: Many thanks for your valuable suggestions. Since our layout diffusion model is only trained on the annotated livingroom and bedroom with the fixed style text prompt, the layout distribution varies from different kinds of room, our framework could not generalize to the common prompt style and other types of room directly. We leave the generation of layouts with free-style text prompts for future work. For the original noisy Structured3D dataset, we have worked with three professional annotators for 800 hours to annotate 5,064 bedrooms and 3,064 living rooms, getting nearly 50K and 69K accurate 3D bounding boxes across 25 object categories, respectively. We plan to release our annotated living room and bedroom datasets soon, and are also working on expanding to other types of rooms. Hopefully, these datasets would facilitate more research in this direction.

• Text prompts are strange: Labeling text descriptions for high-quality complex scenes with a large number of objects is difficult, so we use the same rule as SceneFormer~\citep{wang2021sceneformer} to produce the text prompt for each room. The text prompt only describes a partial scene configuration, including several walls, objects, and their spatial relationships. Please refer to Section 1.2 in the appendix for more information on how to produce the text prompt. We acknowledge that extending our method to open-vocabulary textual input would be an interesting direction for future research.

• Counting wall numbers in generated layout: Since most furniture in the room is associated with the walls, using different wall numbers in the text prompt should generate rooms with different layouts. In Figure.3 of the revised Appendix, we provide additional visualizations of the generated scene layout in the format of 3D semantic bounding boxes. Our results demonstrate that users can control the number of walls or the layout of the room by changing the input text.

• Missing Citations: Thanks for your reminder. We have added these works to our main paper.

• Measure the variety of layout generations.: In the initial submission, we did not include a quantitative comparison for our first stage method, since our method considers both walls, doors, windows, and other furniture, while DiffuScene only considers furniture. Furthermore, DiffuScene has not released the text-conditional training code, making it difficult to compare our method with it on the Structured3D dataset. In hindsight, we agree with the reviewer that this comparison is relevant and include such a comparison on the 3D-FRONT dataset in the Appendix Sec. 1.1. Table.1 shows a quantitative comparison with DiffuScene on 3D-FRONT, indicating that our method achieves results comparable to DiffuScene on the text-to-layout task. DiffuScene's results are slightly better because it uses an additional network to learn a shape code for each furniture, facilitating the inference process to retrieve a more accurate CAD model for each furniture and improving its quantitative metrics.

Thanks for your comments. We address the weakness and questions below:

• Relying on 3D annotation: We appreciate the constructive feedback provided by the reviewer. As mentioned in the third response to reviewer eF7g, we fully concur that one of the key to ensuring a reasonable distribution of generated rooms lies in the layout generation component. Generating a sensible room layout without the introduction of annotated data as priors has been a challenge, as existing methods tend to suffer from the ``Penrose Triangle" problem (i.e. the multi-view consistency or the layout consistency problem), resulting in repetitive textures or objects in the generated rooms. To address this issue, we propose a framework where we first generate the layout and then generate 3D room content based on the generated layout. In order to ensure that the generated room layouts are both realistic and reasonable, we invested considerable effort in annotating a substantial amount of data (Three data annotators worked for 800 hours to annotate 5,064 bedrooms and 3,064 living rooms, getting nearly 50K and 69K accurate 3D bounding boxes across 25 object categories, respectively.). While our current experiments have primarily focused on living rooms and bedrooms, as pointed out by the reviewer, we plan to expand the datasets to various room types in the future. Moreover, we are committed to making our code and all associated datasets publicly available. We hope that our contributions will prove beneficial for future research endeavors and further advancements in this field.

• Does not support random view rendering: We greatly appreciate the valuable feedback from the reviewers. As we have mentioned in our main paper and supplementary materials, our current method is designed to generate panoramic views and does not support the generation of multiple perspectives. In the context of text-based 3D room generation, we believe that the primary challenge lies in overcoming the Penrose Triangle problem to generate rooms with realistic and reasonable layouts. Therefore, our paper introduces a two-stage framework to separate the generation of layout and appearance. Based on the generated results, our approach demonstrates the ability to produce more realistic and plausible outcomes compared to existing methods. We fully acknowledge and appreciate your suggestion. However, we believe that this is not the primary focus of the current paper. We plan to explore the generation of realistic freeform 3D rooms in future work, building upon the foundation laid out in this paper.

• Varify the effectiveness of the first stage: In the initial submission, we did not include a quantitative comparison to those layout synthesis works(e.g. ATISS\citep{paschalidou2021atiss}, DiffuScene\citep{tang2023diffuscene}) for our first stage method, since our method considers both walls, doors, windows, and other furniture, while DiffuScene only considers furniture and cannot represent a closed indoor scene. Furthermore, DiffuScene has not released the text-conditional training code, making it difficult to compare our method with it on the Structured3D dataset. In hindsight, we agree with the reviewer that this comparison is relevant and include such a comparison on the 3D-FRONT dataset in the Appendix Sec. 1.1. Table.1 shows a quantitative comparison with DiffuScene on 3D-FRONT, indicating that our method achieves results comparable to DiffuScene on the text-to-layout task. Although DiffuScene's results are slightly better than our qualitative results, this can be attributed to its use of an additional network to learn a shape code for each furniture. This allows DiffuScene to retrieve a more accurate CAD model, improving its qualitative metrics. We have also provided additional visualization results for our first stage, which can be found in Figure 3 in supplymentary. From Figure 3, it can be observed that the layouts we generated are reasonable and almost conform to the distribution of real-world scenes.

Thank you for your comment. We want to further response on the two points raised by the reviewer.

1. currently applies only to living rooms and bedrooms: As we stated earlier, we labeled 5K bedrooms and 3K living rooms by hiring 3 annotators working for 800 hours. This labeling process is expensive and time-consuming. Thus, we have only labeled living rooms and bedrooms. This dataset is prepared to verify the effectiveness of our proposed algorithm. It is not our focus to build a larger dataset with all kinds of rooms. Our dataset can be easily expanded and our method can be directly generalized to cover other types of rooms.

2. restriction to generating panorama images: We want to highlight that generating high quality panorama images for indoor scenes is not a simple task. As we demonstrated in the paper and the rebuttal, the state-of-the-art method MVDiffusion still suffers from the 'Penrose Triangle' problem, which violates the room layout constraints and generates multiple beds in a bedroom and multiple cabinets in a living room. Our key contribution is to introducing a stage for layout synthesis before the panorama (or 3D scene) generation to ensure high quality results.

Thanks for your comments. We address the weakness and questions below:

• Computational Cost: We appreciate the reviewer's suggestion, and as a response, we have added a table to compare our method with existing approaches in terms of runtime, parameter count, and GPU memory usage. As shown in Table.2 in the Appendix, our method achieves significantly faster runtime for generating a 3D room compared to existing methods. The GPU memory consumption and model parameters of ours are similar to MVDiffusion's. Despite having two diffusion models, our layout generation stage only needs to produce high-level room layouts and furniture arrangements, resulting in low computational costs and model complexity.

• Geometry Comparison in Figure5: In Figure 5 of the main paper, certain views may appear geometrically incorrect due to depth estimation errors and heavy occlusion in the generated panorama. In the newly added Figure.10 in Appendix, we present an alternative view using mesh rendering and shaded geometry to compare with the results of MVDiffusion. From the results, it is evident that MVDiffusion results in an entire room filled with cabinets, whereas our approach is capable of generating a realistic room layout with various pieces of furniture, such as a TV, a table, and a sofa, all arranged sensibly in accordance with the input textual instructions. This highlights the primary issue this paper aims to address: achieving the generation of genuinely realistic and reasonable 3D rooms through a two-stage process of layout generation followed by appearance generation based on the layout. Furthermore, refer to Reviewer Bzvn.3, while MVDiffusion fails to generate walls with the correct number, we can recover the required walls approximately. This highlights the effectiveness of our approach in generating more accurate room layouts.

• Layout Consistency: Thank you for your valuable feedback. It is indeed important to ensure consistency in the layout generation, since the final result is guided by this layout. To achieve this, we have adopted a diffusion-based generation method, which has been proven effective in capturing distributions and generating reliable layouts in other domains, such as 2D image generation. Please note that our layout diffusion model is trained on the scene code space, which consists of 3D furniture bounding boxes. To train this diffusion module for layout generation, we invested substantial efforts in annotating the 3D bounding boxes based on Structured3D dataset (three data annotators worked for 800 hours to annotate 5,064 bedrooms and 3,064 living rooms, getting nearly 50K and 69K accurate 3D bounding boxes across 25 object categories, respectively.). By training our layout diffusion model on such a finely annotated dataset, we are able to generate room layouts that closely adhere to real-world distribution patterns, as supported by our experimental results in Tabel.1 in the Appendix. Our method achieves a comparable result to the state-of-the-art method on text-to-layout generation task. On the other hand, during the training of our layout generation network, we have implemented a physical regularization loss to ensure that different objects do not cross overlap. This regularization penalty helps to maintain spatial relationships among objects, preventing undesired overlaps in adjacent areas of the rooms. We have provided more in-depth details in our revised version, please refer to Sec.1.1 in supplementary for a comprehensive understanding of our physical regularization loss.

Thanks for your comments. We address the weakness and questions below:

• Lack of novelty : As we described in the second paragraph of the Introduction, the focus of this paper is to solve the multi-view consistency or layout consistency in previous text-to-room methods, which we referred to as the Penrose Triangle Problem. As shown in Fig 1 (a), previous methods often generate repetitive objects (e.g. multiple beds or multiple cabinets) in their results. Our method can effectively solve this problem and further allows intuitive editing of the generated model as shown in Fig 1 (b). Our method is simple and effective. No previous text-to-room methods have adopted a similar approach. While some text-to-objects generation methods also use a two-stage design, our method is rather different from these methods. Our shape generation stage involves diffusion in an abstract scene code space, and the appearance stage employs a ControlNet with the guidance of semantic segmentation maps. Quantitative and qualitative comparisons demonstrate the superiority of our method over previous text-to-room generation approaches.

• Artifacts and free-form object surface: We greatly appreciate the detailed and insightful feedback from the reviewer. As discussed in limitation section (Appendix Sec 1.7), due to performance of the panoramic depth estimation network we are using, there may be some artifacts present in the 3D results we generated. Again, the focus of this work is to solve the Penrose Triangle Problem, e.g. the multi-view consistency or layout consistency. The 3D result can be improved with a more advanced depth estimation module, or as the reviewer suggested adding "another stage to convert it into freeform 3D surfaces".