SceneFunctioner: Tailoring Large Language Model for Function-Oriented Interactive Scene Synthesis

We thank you for your valuable comments on our work. Here we provide our point-by-point reply.

We look forward to your response and appreciate it if you could share your ideas.

1.I appreciate the conducted user study, however the validation of the approach appears to be more on the lightweight side.

We understand the concern about the comparison experiments, but we believe they are appropriately done. Evaluation of scene quality (functionality, aesthetics, etc.) is inherently subjective and lacks reliable quantitative metrics. Without reference images and other data, computing the collision rates of objects and the CLIP score was what we could do. We also supplemented three criteria (function, practicality, and aesthetics), evaluating them with both a GPT evaluator and human participants.

2.While the use of zones seems to generate interesting results, it is unclear if the zones can be easily adapted for more complex concepts and layouts (e.g. mixed-use spaces).

The motivation for proposing ‘zones’ is handling more complex concepts/layouts and multi-use spaces, since it helps to decompose the concepts and functions, lightens the burden for the LLM, and enhances understanding of the task. It is still possible that ‘zones’ could fail in extremely large and complex spaces, but we believe they have the generalizability to most interior scenarios in real life. If you have more insights or questions, please discuss them with us.

3.The current approach does not consider the aesthetics of the generated furniture layouts (which has also been highlighted as limitation).

The current framework has some aesthetical limitations as noted in the paper. However, we have instructed the LLM to consider aesthetics (e.g., balance among objects and coherence in layout) through prompting in each step, and our experiments have demonstrated that the results are aesthetically appealing.

4.How does SceneFunctioner handle function conflicts in multi-functional zones?

We appreciate that you have raised an interesting point. Currently, we would say that no mechanism is implemented to explicitly solve functional conflicts, but we expect the first step of our framework to never provide such cases. A ‘zone’, as described in our paper, should ensure consistency of the functions within it. The LLM is explicitly informed of this and carefully instructed to coordinate the functions. Still, we acknowledge that understanding and handling relationships among functions are worth studying, rendering it for our future work.

5.How would an LLM perform better for irregular room layouts (e.g. how would it need to be trained to achieve better performance)?

In our paper, the LLM is instructed to decide the zone sizes while the framework automatically places them within the room. For better performance in irregular spaces, the LLM could be provided with a ‘reference’ rectangular division plan. For example, an L-shaped room could be divided into two or three rectangles. Giving the references as prompts can reduce the possibilities of zone collision errors, but may likely lead to zone sizes that are similar to the division plan (which is undesirable). Therefore, currently we only provide these prompts when encountering continuous failures.

Another solution is to dispatch an LLM agent that focuses only on room division when determining the zones. However, overlooking whether the functions can fit in these zones may eventually lead to worse results.

6.How would this framework adapt to advancements in LLM capabilities (e.g. what features would be required to improve upon the current results)?

This is another interesting perspective. We would expect major advancements and adjustments in our framework when provided with more capable LLMs: (1) Ideally, the LLM could directly suggest the zone positions and shapes (could be irregular) in the first step so no postprocessing is required. The functions will also be allocated more appropriately. (2) The second and the third steps can possibly be combined if the LLM can handle all furniture relations and layouts in a single shot. More complicated relations and layouts may also be possible. (3) More factors (e.g., lighting, windows and doors, and human interactions) may be more easily included into the framework as long as they are appropriately prompted for the LLM to understand them.

7.How adaptable is SceneFunctioner to user feedback after an initial scene is generated (if a user wants minor adjustments, would the framework require a complete re-synthesis)?

Currently, our framework would need a complete re-synthesis for adjustments. For minor adjustments, the user may want to directly interact with the generated scene using our 3D scene platform. Of course, we appreciate the idea of enabling adjustment features and will investigate a stepwise and context-based ‘editable’ framework in the future.

(due to the character limit, we will add another comment)

We have just uploaded a rebuttal revised version of our paper, with all changes marked in blue for your convenience. This revision file supplements the related work in Section 2.1, as you suggested in #9. We would greatly appreciate it if you could review the updated manuscript and let us know if it addresses some of your concerns.

We thank you for your valuable comments on our work. Here we provide our point-by-point reply to your questions. We will post the additional results in the revision file when ready.

We look forward to your response and appreciate it if you could share your ideas.

1.In L231-232, the author mentions appending sample inputs and outputs to the context. How are these samples selected/formed? LayoutGPT selects the closest samples for in-context learning using room dimensions & type for generating random layouts of specific room types. However, synthesizing scenes based on room functions is a more specific task than generating random plausible layouts. Therefore, for a fair comparison, I suggest providing LayoutGPT with in-context samples that are closest in terms of room functions. Sample inputs/outputs to our work are manually designed.

Regarding the suggestion on LayoutGPT, we appreciate your valuable insight and agree that function-oriented examples could be fairer. However, the challenge lies in defining the ‘distance’ of functions. As far as we know, embedding and representation of functions in interior 3D scenes is still an unsettled problem (and this paper is investigating it). The results may vary significantly with different standards for deciding the examples. Additionally, sample inputs also need to be modified to suggest the functions. We are actively working on this and may post the results to the revision file. We also look forward to your response if you have any ideas on it.

2.The author mentions that the wrong results produced by LLMs increase the retries and the overall generation time. I am curious about the quantitative analysis of the feedback mechanism. How much time do the retries take when compared to the actual generation time? How many retries are required on average per-stage?

Thank you for your interest. We will soon list these retry-related statistics in the revision file.

3.While the paper presents intriguing qualitative results, it would be interesting to see how the method performs on larger rooms (dimensions more than 5 meters) or smaller rooms, different room types (such as bedrooms) to better evaluate the method's generalizibility.

Thank you for your interest. Our framework can be generalized to such configurations, and we will soon post the results to the revision file.

We have just uploaded a rebuttal revised version of our paper, with all changes marked in blue for your convenience. Specifically, Section A.2 provides statistics for all steps in our framework, including generation time and retry counts. Section A.3 showcases our framework's generalizability to smaller/larger rooms and a different room type (bedroom). We are actively working on addressing LayoutGPT's function-based examples, as you suggested. We would greatly appreciate it if you could review the updated manuscript and let us know if it addresses some of your concerns.

We thank you for your valuable comments on our work. Here we provide our point-by-point reply. We understand your concerns, but we respectfully disagree with you on some points, especially concerning your misunderstanding of our framework in #1 and #2.

We look forward to your response.

1.The only contribution is a stepwise human-in-the-loop prompting technique to use an LLM (GPT-4o) for scene synthesis. Such a framework does not necessarily address the issue of generating functionally plausible and usable 3D scenes. While there are some merits to the study in the paper in terms of helping LLMs adapt to 3D scene generation tasks, the overall impact and utility is limited.

We have to emphasize that our framework is not a human-in-the-loop technique. The user acts before the generation starts, and can interact with the scene or restart after the generation finishes. The postprocessing and feedback mechanisms are automatically performed. We will soon provide more technical details on these steps in the revision file, and we think that these mechanisms also contribute to the method. Also, we think the proposed ‘zone’, as the bridge for abstract representations (function) and spatial representation (layout), is also inspiring for the research field of scene representation.

Regarding the utility, the experiments demonstrated our framework generally outperforms LayoutGPT and I-Design and achieves decent quality. The user study suggested that our framework saves much time for interior design. Overall, we believe that our framework appropriately addresses generating functionally plausible and usable scenes.

2.In addition, the experiments presented in the paper are performed on a small dataset (500 scenes) that is manually generated for this framework.

The ‘manually generated’ is another misunderstanding, since all scenes for all three methods are automatically generated, with random-sampled function-specified prompts. We also believe that a sample number of 500 is sufficient to show our framework’s relative performance.

3.Did the users have a pool of scene functions to choose from? If not, the presented approach is going to see high input-entropy. How does the LLM handle that? If yes, how many total number of scene functions are provided to the users to choose from? I do see in Line 366 which mentions that each scene is configured with randomly selected 6 functions. From how many available functions are these chosen from?

Yes, there is a pool containing 16 functions to choose from.

4.Were there any quantitative metrics used to verify/validate the correctness of scenes generated at each stage? For example, zone allocation accuracy for stage 1, furniture grouping coherence for stage 2, and placement accuracy for stage 3. Such a quantitative evaluation for every stage can potentially provide more insights into the framework on where things are better/worse.

Thanks for your valuable insight. Although we cannot quantify whether a step output is ‘proper’ or ‘good’, we can judge whether it ‘succeeds’ or results in ‘violation’, as it is what the postprocessing mechanism does. We will soon provide a revision file supplementing these statistics.

5.Finetuned LLMs have shown to provide good generalization capability. Current framework uses the GPT-4o LLM as is. It would be interesting to see the results when the LLM is finetuned for scene synthesis task and then used to synthesize a new scene. I was actually hoping to see some sort of finetuning, followed by an contextual analysis. That would have been more interesting.

Thank you for your suggestion. Due to our limited time and budget, we found it difficult to collect (construct and label) sufficient data for effective finetuning but instead focused on improving the prompting and sample inputs/outputs. Nevertheless, we agree that finetuning is a promising approach in our problem setting that may further enhance the generation quality and efficiency.

6.The paper uses the words “scene” and “room” interchangeably. Is the method proposed in paper for individual room synthesis or scene-level (entire house) synthesis? Please adhere to one convention: either use scene or room everywhere. Better yet, good to provide a clarifying statement in the Intro and/or methodology section about this.

Our paper targets individual room synthesis and both ‘scene’ and ‘room’ refer to this. We will clarify such terms and avoid ambiguity.

We have just uploaded a rebuttal revised version of our paper, with all changes marked in blue for your convenience. Specifically, Section A.1 includes supplementary technical details for postprocessing steps. Section A.2 provides statistics for all steps in our framework. Additionally, we add a sentence in the Introduction Section to clarify 'scene' and 'room'. We would greatly appreciate it if you could review the updated manuscript and let us know if it addresses some of your concerns.

We thank you for your valuable comments on our work. Here we provide our point-by-point reply. We understand your concerns, but we respectfully disagree with you on some points.

Your reviews #3 and #5 contradict each other, where ‘a niche problem’ seems controversial to ‘extensively studied’. We are confused about the reviews.

We look forward to your response.

1.This paper reads more like a technical report of a user application and prompt engineering rather than addressing fundamental research problems.

Our paper addresses a research problem of ‘tailoring user preferences to scene representation and generation’. It may not be ‘fundamental’ as you suggest, but we consider it a valuable topic for deepening the understanding of representation for 3D scenes. Also, our contributions are not limited to prompt engineering. We believe our coarse-to-fine framework and the proposed ‘zone’ representation provide new insights into 3D scene representation and related research.

2.The proposed solution also seems tentative, as demonstrated by several limitations

We acknowledge that our solution requires improvements. However, some limitations mentioned are not critical or can be addressed by expanding our framework:

Limited zone shapes: Flexible shapes can be supported since we can simply ask the LLM to support them. However, as noted in the paper, the LLM’s performance may significantly reduce. Thus, we find it more reliable to support flexible zone layouts rather than the shapes themselves.

Generation inefficiencies: As long as there are LLM errors, extra efforts are needed. For example, I-Design[1] relies on a ‘layout corrector’ to fix such errors. We have implemented postprocessing mechanisms to handle them and minimize their impact on the generation time. Also, the average time for generating each scene is less than 30 seconds, which is acceptable for live user interaction. For comparison, I-Design requires about two minutes to generate each scene.

Floor plan structures: These factors can be easily addressed by expanding each step to include them.

[1] Çelen, A., Han, G., Schindler, K., Van Gool, L., Armeni, I., Obukhov, A., & Wang, X. (2024). I-design: Personalized llm interior designer. arXiv preprint arXiv:2404.02838.

3.The paper tackles a niche problem that may not resonate with a broad ICLR audience.

Our research falls into general topics like ‘applications of 3D scenes’ and ‘scene representation and understanding’, which are appropriate topics for ICLR. The research problem is worth studying (as you also suggest in #5 that extensive prior research exists in this area), regardless of whether it is ‘broad’.

4.The use of GPT, a closed-source model, means the solutions are less reliable and more tentative due to the lack of understanding and access to the model’s workings.

First, our research does not emphasize the mechanisms of LLMs. Instead, we emphasize how LLM understands and yields 3D scenes. Second, we do not think that a close-source model means the method is less reliable. This statement lacks factual evidence and theoretical basis. According to our experiments and existing research, the result scenes are reliable.

Please refer to our experimental sections.

5.The spatial understanding of LLMs has been extensively studied in prior research, and this paper does not provide new insights that could inspire or motivate future work in this area.

How we address the functions of the scene distinguish our work from prior research. We have proposed using ‘zones’ as a bridge for abstract representations (function) and spatial representation (layout), which is novel. Additionally, we believe the coarse-to-fine framework provides instructive inspiration in addressing more scene representation and generation problems.

Please refer to our ‘related works’ section and our supplementary video.

Thank you for your reply.

“Functionality preference” is a niche problem primarily relevant to very specific user applications. The general spatial understanding of LLMs (e.g., whether LLMs have a notion of relative positioning, collisions, etc)—problems shared by a wide range of downstream applications (not limited to digital design, robotics, SLAM, and more)—has been extensively studied. However, “functionality preference” does not contribute significantly to the broader spatial understanding of LLMs that is relevant to larger ICLR communities. It resembles more of a product pain point rather than a research problem.

I disagree with the assertion that the mentioned limitations are not critical. From a product pain point perspective, issues such as limited zone shapes, zone border inconsistencies, generation inefficiencies, and floor plan structures can negatively impact the user experience just as much as functionality preference. It is not fair to claim that only the addressed problems are critical.

These problems are not as readily addressable as suggested by the authors. As noted by the authors, accommodating flexible shapes could reduce the LLM’s performance. Additionally, the current pipeline requires multiple retries to avoid collisions. If the proposed method enables LLMs to generate valid layouts (e.g., no collisions, no blocked pathways) in a single attempt without relying on post-processing, I would champion the paper without hesitation. Clearly, this is an important and non-trivial problem shared by a much larger community. If the authors genuinely believe these issues can be addressed, it is strongly recommended to resolve them for resubmission.

Regarding the use of GPT-4o, the issue is not its usage but rather the sole reliance on it for validation. What happens if OpenAI updates their model checkpoint? Would the claims and experiments presented in the paper remain relevant? It is strongly recommended that the authors demonstrate their approach on multiple LLMs, preferably using open-source models.

Unfortunately, all of my concerns remain unaddressed.

I agree that not every paper needs to address a general problem, and not every solution must be free of limitations. However, these factors collectively influence the contribution of the paper.

I maintain my position that the technical contribution of this paper is highly incremental, given the narrow scope of the targeted problem, the limited solution, and, equally importantly, the insufficient validation, which leads to irreproducibility.

If OpenAI updates their model checkpoint, since no one has access to the original model, all the claims and experiments presented in the paper will become irrelevant. This seriously affects the reproducibility of the method. Evaluation solely on a closed-source model is far from sufficient.