InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior

Thank you for your insightful and valuable feedback. It is really inspiring to know that you think our work is meaningful and well-designed, our experiments are extensive, and the writing is easy to follow. Below are our clarifications for your concerns. We value your input and would love to hear any follow-up you have to the response.

1. Can you confirm how iRecall is computed?

It is right that “iRecall is the proportion of the required triplets in the generated scenes and the ground truth in instruction triplets”.

However, the triplets are not extracted from instructions. Instead, instructions in our dataset are derived from 1~2 randomly selected triplets. Thus, we have the ground-truth triplets of corresponding instructions for iRecall evaluation. More details about the dataset are provided in Appendix A.

2. Can you comment on what will happen if we give “arbitrary instructions”?

While the curated instructions in our proposed dataset are derived from predefined rules, we believe that our model exhibits generalizability to a broader range of instructions. We use instructions automatically generated in a certain pattern only for evaluation convenience.

The generalizability owes to the pretrained CLIP text encoder, which is adapted to extract semantically meaningful features of text inputs, instead of using a language parser like the early studies of 3D scene synthesis methods [1, 2] that can only handle sentences in limited expressions. It means the text features that we used from conditioning would be close in the feature space if the input texts have similar meanings.

We utilize instructions in various sentence patterns in the stylization task, such as "Let the room be wooden style" and "Make objects in the room black," as illustrated in Appendix Figure 8. We also try some meaningless “arbitrary instructions”, such as “a cat on the ground” and “a dog wearing a santa hat”, and our method can still generate meaningful and aesthetic 3D scenes like those generated unconditionally shown in Appendix Figure 11. When the inputs are some sentences with vague location words, like “Put a chair next to a double bed.”, our method will generate corresponding objects in all possible spatial relations (e.g., “left”, “right”, “front” and “behind”).

Nevertheless, our model still faces limitations in comprehending complex text instructions. For instance, (1) handling instructions with more required triplets, like 4 or 5, poses a challenge. (2) Additionally, identifying the same object within one instruction, such as "Put a table left to a sofa. Then add a chair to the table mentioned before" is also a difficult task. (3) Furthermore, InstructScene struggles with abstract concepts such as artistic style, occupants, and functionalities that do not occur in the curated instructions. Given the rapid development of large language models (LLMs), we believe the integration of LLMs into the proposed pipeline is a promising research topic.

A viable approach to improve the quality of current instructions involves employing LLMs to refine entire sentences in the proposed dataset or using crowdsourcing to make the dataset curation pipeline semi-supervised. We hope the proposed dataset and creation pipeline could serve as a good starting point for creating high-quality instruction datasets.

We will add this discussion to the revision of this work.

---

[1] Learning Spatial Knowledge for Text to 3D Scene Generation. EMNLP 2014.

[2] Text to 3D Scene Generation with Rich Lexical Grounding. ACL 2015.

5. Is there an assumed viewpoint from all scenes?

The term "view-dependent" means that spatial relations are respected by the users who provide instructions. These spatial relations will change in accordance with the users' viewpoints. There is no need to specify the camera's perspective.

6. The proposed method generates a single bed while the instruction mentions a double bed.

Thank you for the careful assessment. Although we have significantly promoted the controllability of instruction-driven 3D scene synthesis, the iRecall scores of our method are still not 100%. Therefore, it is possible to observe incorrect triplets in the generated scenes. We hope follow-up works could further improve the performance.

7. Missing reference of LayoutGPT.

Thank you for the kind reminder. We will add this reference in the revision of this work.

Thank you for your valuable feedback. We are happy to know that you regarded the curated dataset as useful, and that the proposed method is effective and versatile. Below are our clarifications for your concerns. We value your input and would love to hear any follow-up you have to the response.

1. The novelty of the proposed method.

While acknowledging the proposed method relies on existing techniques, such as the diffusion model first introduced in 2015 [1] and vector quantization first proposed in 1984 [2], the novelty and contribution of our work stem from three key aspects:

(1) we propose a new formulation of an instruction-driven scene synthesis pipeline, which integrates a semantic graph prior and a layout encoder, and notably improves the generation controllability; (2) our disentanglement design of semantic graph prior and layout decoder enables various useful applications in a zero-shot manner; (3) we curate a high-quality dataset to promote the research of language-driven 3D scene synthesis, which could be considered as a timely contribution that aligns with the current trends.

Meanwhile, the basic idea behind the proposed method is to introduce structural priors for complex language-driven content generation. The proposed semantic graph prior is a general concept and can be directly applied to other tasks, such as 2D poster design. We hope the exploration of 3D scenes could inspire more research on other domains.

2. Some claims about using VQ-VAE require further substantiation.

OpenCLIP features indeed encode characteristics like colors and materials. However, these attributes are entangled within a single vector of 1280 dimensions. This high dimensionality makes it challenging to model these features together with other low-dimensional scene attributes, such as transitions and scales, which only have 3 dimensions each.

The use of VQ-VAE is not for efficiency. Instead, it serves to simplify the learning task. Indexing discrete variables from a codebook can significantly reduce the burden of network optimization. Our preliminary experiments also suggest that both ATISS and DiffuScene face difficulties in modeling high-dimensional semantic feature distributions. To ensure fair comparisons in our experiments, we enhance these models to generate quantized features.

3.1. The diffusion preliminary section is too abbreviated and disjoined.

Making the preliminary diffusion models an independent section is because we apply two kinds of diffusion models in this work, i.e., discrete and continuous ones. Sec. 3 provides a general introduction to the core idea of diffusion models, which is agnostic of diffusion kernels, ranging from Gaussian to uniform categorical transitions and various other choices. By establishing this foundation in Sec. 3, we create a context for introducing the proposed semantic graph prior and layout decoder in a unified expression of diffusion models.

It's important to note that both conditional diffusion models in this work, one for instruction-driven graph generation and the other for graph-conditional layout generation, are specifically tailored for respective tasks. They differ from the original ones proposed for images or texts, and are not suitable to be treated merely as a preliminary study.

3.2. The annotation is overcomplicated and hard to follow.

We apologize for any confusion caused by the annotation. We try our best to keep the annotations rigorous and consistent in the manuscript. We use lowercase characters like $c$ and $f$ for attributes of one object, and use calligraphic fronts like $\mathcal{C}$ and $\mathcal{F}$ for attributes of one scene. $\mathcal{C}_i$ and $\mathcal{F}_i$ refers to the specific $i$-th scene. We use superscripts to denote different scenes and subscripts to denote different objects in the same scene. Variables with omitted super-/sub-scripts mean a non-specific sample.

We will try our best to keep the annotation concise in the revision of the manuscript. Thank you for bringing that to our attention.

4.1. What is the CLIP-aligned feature in Fig. 2?

The “CLIP-aligned feature” in Figure 2 refers to the output vector of OpenShape [3], which is a pretrained RGB point cloud encoder that is contrastively trained to be aligned with the output space of OpenCLIP ViT-bigG/14.

4.2. How is the CLIP-aligned feature vector applied to the VQ-VAE encoder?

As described in the first paragraph of Sec. 4.4, the encoder of $f$VQ-VAE is a stack of cross-attention modules. $n_f$ learnable tokens are employed as queries, and the CLIP-aligned feature vector is served as the key and value.

---

[1] Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015.

[2] Vector Quantization. IEEE Assp Magazine 1984.

[3] OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding. NeurIPS 2023.

Thank you for your insightful and valuable feedback. It is really inspiring to know that you think our formulation of scene generation is novel, effective and versatile, the curated dataset is a timely contribution, and the paper is clear and well-written. Below are our clarifications for your concerns. We value your input and would love to hear any follow-up you have to the response.

1. More details about the dataset creation can be provided.

The training data of instructions are very similar to those provided in the first column of Appendix Figure 5~7, which are for scenes in the test split, instead of the training split. The details about the data creation are provided in Appendix A. We also provide some examples below.

(0) Object categories are provided by the original dataset: e.g., {id1_category: "dressing_table"}. (1) Spatial Relation Extraction: e.g., (id1, “closely left”, id2). (2) Object Caption: e.g., {id1_caption: "a 3D model of a wooden desk with a mirror on top in a white background"}. (3) Caption Refinement: e.g., {id1_refined_caption: "a modern wooden dressing table with a mirror on top"}. (4) Instruction Generation: e.g., {“Place”, “Add”, “Put”, …} + id1_refined_caption + {“closely to the left of”, “to the close left of”, “to the near left of”, …} + id2_refined_caption. + {Then, Next, And, ...} + ...

Our dataset and scripts for creation will be publicly available after the review period.

2. Are the both directions of the instructions needed to be created? If so, how to determine which direction to use during training?

We randomly pick one direction to generate instructions during training, e.g., “a chair is left to a table” and “a table is right of a chair”, where both instructions should correspond to the same relation in a scene. To incorporate this inductive bias, we did some tricks in the code. In particular, the final logits for loss computation are derived by averaging the upper triangular part and the symmetrized lower triangular part of the relation matrix.

Our code will be publicly available after the review period.

3. What is the runtime of the proposed method compared to baselines?

In the default settings (T=100+10), our method takes about 12 seconds to generate a batch of 128 living rooms by our method on a single A40 GPU. In comparison, ATISS (Paschalidou et al., 2021) takes 3 seconds, and DiffuScene (Tang et al., 2023) requires 22 seconds.

It's noteworthy that our method can be significantly accelerated by reducing the number of diffusion time steps. For instance, setting T=20+5 reduces the runtime to 3 seconds without a noticeable decline in performance. The impact of diffusion time steps is investigated in Sec. 5.5.1. We believe with more advanced diffusion techniques and in more complex scenes, diffusion models can be more effective and efficient than autoregressive models.

4. Can these rules cover the majority of the instructions compared to a human-annotated dataset? What are the limitations of the instructions in the current dataset?

While the curated instructions in our proposed dataset are derived from predefined rules, we believe that our model exhibits generalizability to a broader range of instructions. For example, in the stylization task, we utilize instructions in various sentence patterns such as "Let the room be wooden style" and "Make objects in the room black", as illustrated in Appendix Figure 8. We also experiment with instructions containing vague location words, like "Put a chair next to a double bed," wherein our method generates corresponding objects in all possible spatial relations (e.g., "left," "right," "front," and "behind").

Nevertheless, our model still faces limitations in comprehending complex text instructions. For instance, (1) handling instructions with more required triplets, like 4 or 5, poses a challenge. (2) Additionally, identifying the same object within one instruction, such as "Put a table left to a sofa. Then add a chair to the table mentioned before" is also a difficult task. (3) Furthermore, InstructScene struggles with abstract concepts such as artistic style, occupants, and functionalities that do not occur in the curated instructions.

These limitations are attributed to the CLIP text encoder, which is contrastively trained with image features and tends to capture global semantic information in sentences. Given the rapid development of large language models (LLMs), we believe the integration of LLMs into the proposed pipeline is a promising research topic.

A viable approach to improve the quality of current instructions involves employing LLMs to refine entire sentences in the proposed dataset or using crowdsourcing to make the dataset curation pipeline semi-supervised. We hope the proposed dataset and creation pipeline could serve as a good starting point for creating high-quality instruction datasets.

We will add this discussion to the revision of this work.

4. A user study would have been useful.

Thank you for this suggestion. However, we found applying a user study for evaluation could be tricky and potentially biased. Instead, we introduce two new quantitative metrics: (1) iRecall, measuring the proportion of the required triplets occurring in synthesized scenes to all provided in instructions and (2) $\Delta\coloneqq \frac{1}{N}\sum_{i=1}^{N}\text{CosSim}(\mathbf{f}_i,\mathbf{d}^{\text{style}}_i)-\text{CosSim}(\mathbf{f}_i,\mathbf{d}^{\text{class}}_i)$, designed to quantitatively assess if the generated scenes matches the required styles.

Acknowledging the subjective nature of fidelity and aesthetics, we have presented comprehensive qualitative results in the appendix and leave it to readers to determine their preferences.

5. Discussions on how qualitatively the proposed method works better than prior works.

Thank you for this suggestion. In Appendix C.2, we discuss how our proposed method qualitatively outperforms prior works. We find that the autoregressive model ATISS tends to generate redundant objects, resulting in chaotic synthesized scenes. DiffuScene encounters challenges in accurately modeling object semantic features, often yielding objects that lack coherence in terms of style or pairing, thereby diminishing the aesthetic appeal of the synthesized scenes. Moreover, both of these baseline models frequently struggle to follow the provided instructions during conditional generation. In contrast, our approach demonstrates a notable capability to generate highly realistic 3D scenes that concurrently adhere to the provided instructions.

We will add this discussion to the main paper in the revision of this work.

6. Examples on how the proposed method can generalize & generate a diverse set of rooms from a single prompt.

Thank you for this suggestion. We provide examples of a diverse set of scenes generated from a single prompt and the same semantic graph in the supplementary material, showcasing the diversity of the proposed generative method.

We will add these visualization results in the revision of this work.

7. Discussion on how to incorporate fuzzier instructions.

While the curated instructions in our proposed dataset are derived from predefined rules, we believe that our model exhibits generalizability to a broader range of instructions. For example, in the stylization task, we utilize instructions in various sentence patterns such as "Let the room be wooden style" and "Make objects in the room black", as illustrated in Appendix Figure 8. We also experiment with instructions containing vague location words, like "Put a chair next to a double bed," wherein our method generates corresponding objects in all possible spatial relations (e.g., "left," "right," "front," and "behind").

Nevertheless, our model faces challenges in comprehending abstract concepts such as artistic style, occupants, and functionalities that do not occur in the curated instructions. This limitation is attributed to the CLIP text encoder, which is contrastively trained with image features and tends to capture global semantic information in sentences. Given the rapid development of large language models (LLMs), we believe the integration of LLMs into the proposed pipeline is a promising research topic.

A viable approach to improve the quality of current instructions involves employing LLMs to refine entire sentences in the proposed dataset or using crowdsourcing to make the dataset curation pipeline semi-supervised. We hope the proposed dataset and creation pipeline could serve as a good starting point for creating high-quality instruction datasets.

We will add this discussion to the revision of this work.

Thank you for your insightful and valuable comments and suggestions. It is really inspiring to know that you think our work is solid for both design choices and creating a dataset, and good quantitative and qualitative performance. Below are our clarifications for your concerns. We value your input and would love to hear any follow-up you have to the response.

1. Applicability of the proposed method in other domains.

The basic idea behind the proposed method is to introduce structural priors for complex language-driven content generation. The proposed semantic graph prior is a general concept and can be directly applied to other tasks, such as 2D poster design. We hope the exploration of 3D scenes could inspire more research on other domains.

2. Discuss other previous graph-/text-based works that are not trained on 3D-FRONT.

Thank you for the kind reminder. There are plenty of great works on both graph-based and text-based 3D scene generation. We try our best to include them in the related work section (Sec. 2) and discuss the drawbacks of existing works.

Two methods, namely ATISS (an autoregressive model) by Paschalidou et al. (2021) and DiffuScene (a diffusion model) by Tang et al. (2023), are selected as baselines in our work. That is because both of them are recently proposed (many papers in 2023 are still using the same architecture as ATISS [1, 2]), representative (autoregressive and diffusion models currently stand as two of the most popular generative models), and showcase state-of-the-art performance in various metrics. Other remarkable works, such as FastSynth [3], SceneFormer [4], DepthGAN [5] and Sync2Gen [6], despite their significance, demonstrated notably inferior performance compared to ATISS and DiffuScene, and most of them are unconditional generative methods, so they were excluded from our evaluation.

We acknowledge that we may overlook some important works and would appreciate any suggestions. We will add them to the revision of this work.

3.1. Would manually changing semantic feature of an object result in notable differences in the final scene? Can the model pick up stylistic information from the training data? Do objects that frequently occur together in the training data behave similarly in the generated scenes?

We conduct two additional experiments to further validate our method: (1) masking the semantic feature of one object and utilizing a pretrained semantic graph prior for recovery; (2) masking semantic features of all objects except one and again using the pretrained semantic graph prior for recovery. Instructions for both experiments are set to none. Visualization results are presented in the supplementary material submitted during the rebuttal period.

These results indicate the diversity of our method and highlight that semantic graph prior could effectively capture stylistic information and object co-occurrences from the training data. Our method trends to generate style consistent and thematic harmonious scenes, e.g., chairs and nightstands in a suit, and matched color palettes and cohesive artistic style. Other visualizations in the appendix also support this claim, especially with Figure 8 showcasing the generation of all semantic features based on both instructions and other scene attributes.

3.2. Is there any noticeable degradation in scene layout quality if semantic features are not encoded or if they are encoded with a smaller feature space?

We observed a significant decline in the appearance controllability and style consistency of generated scenes when semantic features were omitted. We include these degraded visualization results in the supplementary material.

It arises from the fact that, without semantic features, the generative models solely focus on modeling the distributions of layout attributes, i.e., categories, translations, rotations, and scales. This exclusion of semantic features results in generated objects whose occurrences and combinations lack awareness of object style and appearance, which are crucial elements in scene design.

We will add this discussion to the revision of this work.

---

[1] MIME: Human-Aware 3D Scene Generation. CVPR 2023.

[2] Learning 3D Scene Priors with 2D Supervision. CVPR 2023.

[3] Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models. CVPR 2019.

[4] SceneFormer: Indoor Scene Generation with Transformers. 3DV 2021.

[5] Indoor Scene Generation from a Collection of Semantic Segmented Depth Images. ICCV 2021.

[6] Scene Synthesis via Uncertainty-driven Attribute Synchronization. ICCV 2021.