SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent

Thank you for the valuable feedback and suggestions.

Q1: More participants in human evaluations.

We improve the user number to 20 and increase the evaluated scene number to 200, each of them is shown 40 samples. We show them the same instructions used as prompt in the LLM reflector for the participant to evaluate the scene carefully. We statistic the results into the following table, which indicates similar results with the previously shown Table 3 in the main paper.

Q2: GPT for both generation and validation

We understand your concern. To evaluate the generation system, human study stands for a fair choice, but it is very expensive to scale. The LLM provides a cheaper, faster and more stable (as shown in the user-LLM alignment) choice to evaluate. Taking LLM for both generation and validation seems conflict, while here we consider it as acceptable, since the LLM score is easy to acquire during generation and does not result in overfitting. We could also add other scanffoldings, but they may not provide a better evaluation. We can also apply GPT to filter the results of other methods such as ATISS, but they will still returns lower score due to the limitation of the method itself. For example, ATISS will be limited to the data distribution of the training dataset, trapped in limited categories of large furniture and simple layout of only 3 room types. Therefore, we emphasize that, given an accessable LLM model to guide the generation system, the key point is how to utilize the guidance and find a good way to improve the scene quality.

Q3: More controllable prompt

We have shown some samples of complex prompts in Figure A2 in the supplementary material. It shows the controllability of objects’ category (basket, washtub, etc.), number (”A laundromat with 10 machines”), position (”a car in the center”), and precise components (decoration on the wall, small objects). We will release more results in the final version and web page, since the rebuttal can not show additional images and external links.

Q4: Use SceneWeaver to refine scenes generated by baseline methods

That is a good idea. Due to time constraint in rebuttal, here we refine scenes generated by two baseline methods, PhyScene and LayoutGPT. (IDesign and Holodeck will take longer time in format conversion.)

Results shows our method could greatly improve the baseline method. Actually the baseline method serves as a initializer tool in this setting, and we can convert each baseline method to a new tool and merge them into the extensible tool cards.

Q5: “Iterative” examples to improve each of the metrics

The overall is self-adaptive rather than coarse to fine object placement. The agent in our pipeline will find the biggest problem in each iteration according to the reflection score and choose related tool to solve the problem. For example, the teaser serves as an obvious example. Here we show the metric score after executing each step:

The agent will choose the lowest score to improve. Specifically, the step 1 has lowest score in completion, which is 4. Then in the step 2, the agent chooses the implementer to add objects in the shelf and improve the completion score from 4 to 6. Now the lowest score is the layout, which is 5, owing to the illogical location of bathroom sink. So in step 3, it uses modifier to remove the bathroom sink, improving the layout score from 5 to 6. So now the lowest score consist of both layout and completion. In step 4, the agent focuses on the crowding area and decide to use modifier to rearrange the dining tables to ensure adequate space for movement and clear walkways. Then the score of layout becomes 8. The lowest score come to the completion again. So the agent choose implementer again to add objects on each table to improve the completion score.

You can see the realism and functionality will also be impoved while we focus on optimizing other metrics. And the agent can fix the problem caused by previous steps, such as removing the bathroom sink generated in previous step. And it can utilize the same tool for multiple times to improve the quality untill it fit the demand.

Q6: Sampling strategy

The Sampes in Figure 4 and Figure A2 is randomly choose from the generated scenes. Figure 4 also show 2 samples to show variance of our method. And the results in the tables are statistic on all the generated results with the related prompts.

And we further implement a user study to confirm the stability and diverisity of our method. We increase the generated scene (randomly chosen) number from 1 to 3 for each method in the second comparison setting in order to 1) reduce individual variance and 2) check diversity of different generation system. We ask the participants to choose:

• Which method do you prefer?
• Which method has greater diversity?

Results shows our methods has greater diversity while also stand for higher preference, showing the stability of our method.

Thank you for the valuable feedback and suggestions.

Q1: User Study

User Number and Sample Size

We enhanced the study's robustness by scaling to 20 participants evaluating 200 scenes (40 samples each), using identical LLM reflector's instruction of metrics to ensure consistent evaluation criteria. Similar to Infinigen-Indoors' approach of focusing solely on furniture layout's realness without texture and rendering, our original instructions (shown in Table A2) also emphasize to evaluate realness by excluding texture and lighting considerations. This unified evaluation framework, combined with our Objaverse-based pipeline, minimizes platform/asset biases. ）The results remain consistent with Table 3's findings.

Score Alignment and Validity of Evaluation

Given that human and LLM evaluations naturally differ slightly, we assessed their consistency by converting scores to rankings and computing Kendall's Tau across all four metrics. This confirms the stability of LLM's justification.

Results demonstrate strong agreement between users and LLM across metrics, with higher user-LLM alignment than user-user (τ=0.55 vs 0.40 on Completion metric), confirming LLM's superior evaluation stability. These findings validate our LLM justification approach in Tables 2/3.

Diversity and Preference

We increase the scene number from 1 to 3 for each method in the second comparison setting in order to 1) reduce individual variance and 2) check diversity of different methods. We ask the participants to choose:

• Which method do you prefer?
• Which method has greater diversity?

Results shows our methods has greater diversity while also stand for higher preference, surpassing other methods in a large degree.

Q2: GPT for both generation and validation is invalid. need more scanffolding.

We understand your concern. To evaluate the generation system, human study stands for a fair choice, but it is very expensive to scale. The LLM provides a cheaper, faster and more stable (as shown in the user-LLM alignment) choice to evaluate. Taking LLM for both generation and validation seems conflict, while here we consider it as acceptable, since the LLM score is easy to acquire during generation and does not result in overfitting. We could also add other scanffoldings, but they may not provide a better evaluation.

Using GPT to filter the results of ATISS will still returns lower score due to the limitation of the method itself, trapped in data distribution of the training dataset with limited categories and room types. Therefore, given an accessable LLM model to guide the generation system, we emphasize how to utilize the guidance and find a good way to improve the scene quality.

Q3: “Functionality” could have a geometric definition such as accessibility and interactivity.

This is a good suggestion. The “functionality” you mentioned is related to precise calculation for the physical interactivity, which is hard for LLM to reply. Here we refer to PhyScene and SceneEval and add additional physical metric accessibility.
Due to the time limits, here we randomly choose 3 bedrooms of each method and check the percentage of large furniture (such as bed, wardrobe, and nightstand) that can be reached in the front.

Our methods shows high accessibility since the agent is able to check the walkable area. For example, the step 4 in Figure 1 shows the agent has recognized the crowding area and rearranged the dining tables to ensure adequate space for movement and clear walkways. Note the Holodeck shows relatively high accessibility. That’s because its room has larger size than other methods and objects are closed to the wall.

Q4: Physical Metrics

Validity of #OB and #CN

Here we do not use shape packing or floating placement for higher physical score. The size of assets is decided by each tool in reasonable process. And each object has as least one relation (contact closely with out collision) with the room (floor/wall) or supporting objects (table/shelf/…), ensuring that nothing is floating in the room. Thus table 2/3's #Obj, #OB and #CN metrics are solid.

New Metric：Shift in simulation

Thanks for your suggestion. To assess object stability in simulation, we measured:

• Shift thresholds: percentage of objects moving >0.1m or >0.01m in 3s
• Average displacement: Mean shift distance (meters)

The lowest shift of our mthods verify our great performance on the original physical metrics #OB and #CN. Obove all, our generated scene remains the most stable in simulation with the largest object number, which further confirm the great ability of our method’s physically suitable position.

Q5: Cost

The average time for a single iteration is 8.6 minutes, and the average time for generating a complete scene is 64 (min:35, max:130) minutes. The average iteration is about 7. The average API cost is about 0.5 dollar per scene. And Holodeck report its cost as 0.2 per scene. IDesign and LayoutGPT does not release the cost. Since other baseline does not provide CPU&GPU cost, we think the comparison might not be precise.

The relatively higher cost come from well-designed agentic framework, making the great results on both physical and visual & semantic metrics is irreplaceable by other methods. More diffusion steps or longer infinigen procedure will not change the difference.

Moreover, here are some choices to reduce the cost by:

• simpler requirements and smaller room size.
• reduce the usage frequency of the physical optimization.
• remove some time-consuming tools or replace with other effective tools.
• reduce the iteration number.

Note reducing iteration number will not affect the final result too much, since the scene achieves high score quickly in the first few iterations. The rest iterations aim to fix corner cases and improve the score slightly. For example, the GPT score for bedroom is 8.6, 9.3, 7.5, 7 when we limited the steps to 3, still better than the baseline methods.

Q6: Implementation of the tools

We list the implementation of each tool.

During the planning process, the agent is provided with brief description of each tool, shown in A3-A13 in supplementary material and use the function calling to choose a single tool for each step. Then it will run the tool itself and go through the executor to update the scene.

Q7: Implementation of the executor

Note we not only contribute on the tool itself, but also modify the infinigen to :

• update iteratively
• interactive in blender in realtime with socker
• fit each tools rather than generate scene procedurelly.
• Physical optimization
• Add 3D marks and 2D top-down rendering
• …

Q8: How to choose assets dataset

In this project, we choose different asset according to the usage of tool. And you can also choose any of them to fit your own requirements.

• MetaScenes: For tool using Dataset such as MetaScenes, we employ its assets directly, since each scene contains several assets with delicated mesh and layout information.
• 3D FUTURE: For tool using Model such as PhyScene/DiffuScene/ATISS, we employ 3D FUTURE, since the model is trained on this dataset.
• Infinigen: For other tools, we use Infinigen's asset generation code to generate standard assets in common categories, such as bed, sofa, and plate. The asset will be generated in a delicated rule procedure in the scene generation process.
• Objaverse: For those catrgories that are not supported by Infinigen, such as clock, laptop, and washing machine, we employ open-vocabulary Objaverse dataset.

We thank reviewer DJFe for the response.

Here we list the detailed scores of 10 scenes generated by ATISS, which can be regarded as sampling ATISS ten times （10 is also the maximum iteration number of the agent in SceneWeaver).

We calculate the best score of each metric, which can be approximately regraded as taking the max over 10 trails using GPT-4.

Results shows the best score of ATISS still falls short compared to ours.

Moreover, due to the limited dataset, ATISS is in lack of scene diversity, taking the best over multiple trails may make it worse. Meanwhile, ATISS has no small or open-vocabulary objects, the #Obj is low, and the shift in simulation is high.

Also, according to the refinement results of PhyScene (more details shown in Q4 for Reviewer dp6k), we believe our method can further refine ATISS's result to a higher score, which can not be achieved by repeated sampling and max-selection with GPT-4. (Here PhyScene and ATISS are similar methods)

Above all, although using GPT to filter ATISS can make some improvement, there still exists a big gap between ATISS and our method.

Thank you for the valuable feedback and suggestions.

Q1: More participants in user study.

We enhanced the study's robustness by scaling to 20 participants evaluating 200 scenes (40 samples each), using identical LLM reflector instructions to ensure consistent evaluation criteria. The results remain consistent with Table 3's findings.

Q2: accessibility of the furniture

This is a good suggestion. We think it is a good point to add this aspect into a new physical metric. Here we refer to PhyScene and SceneEval and add additional physical metrics “accessibility”.
Due to the time limits, here we randomly choose 3 bedrooms of each method and check the percentage of large furniture (such as bed, wardrobe, and nightstand) that can be reached in the front.

Our methods shows high accessibility since the agent is able to check the walkable area. For example, the step 4 in Figure 1 shows the agent has recognized the crowding area and fix it. Note the Holodeck shows relatively high accessibility. That’s because its room has larger size than other methods and objects are closed to the wall.

Q3: Release more details and limitations such as diversity of results and anti-physical content.

Diversity The diversity of our method is high, due to the extendable tool cards and generation ability of foundation model. We implement the user study to evaluate the diversity of scenes.

Specifically, we increase the generated scene number from 1 to 3 for each method in the second comparison setting in order to 1) reduce individual variance and 2) check diversity of different generation system. We ask the participants to choose:

• Which method do you prefer?
• Which method has greater diversity?

Results shows our methods has greater diversity while also stand for higher preference, surpassing other methods in a large degree.

More Limitations

• Asset retrieval may return poor results due to non-standard open-vocabulary formats and language embedding corner cases.
• For assets not generated by Infinigen's procedures, supporting surfaces are estimated from geometry. If the geometry is irregular, this can result in tilted surfaces and unusual placement of child objects.
• The previous tool produced sparse layouts inside containers, unable to densely arrange objects because it couldn't accurately detect supporter shapes. To address this, we extended the tool to enable denser placements, such as adding 30 well-arranged books on a shelf.

Q4: More practical metrics：Shift in simulation

Thanks for your suggestion. To assess object stability insimulation, we measured:

• Shift thresholds: percentage of objects moving >0.1m or >0.01m in 3s
• Average displacement: Mean shift distance (meters)

The lowest shift of our mthods verify our great performance on the original physical metrics #OB and #CN. Obove all, our generated scene remains the most stable in simulation although with the largest object number.

Q5：generation with more complex prompts for robot task

This is a good point. The robotic task context can be achieved by adding specific description prompt design. Here we show some generated samples of complex prompts in Figure A3. Our method is capable of following complex requirements in different context. It shows the controllability of objects’ category (basket, washtub, etc.), number (”A laundromat with 10 machines”), position (”a car in the center”), and precise components (decoration on the wall, small objects).

Q6: Evaluate the impact of a single tool

Table 5 shows that adding different types of tools (Initializer, Refiner, Implementor) greatly improves results. To assess the effect of a single tool, results can be compared with and without that tool using identical prompts, which requires multiple generations for reliable statistics. Due to time constraints, we do not provide precise quantitative results here. However, we observed that without the “Update Rotation” tool, the model is less sensitive to invalid rotations, reducing the “layout” score. Meanwhile, the “Add Crowd” tool lets supporters be densely filled with child objects (e.g., a shelf packed with books), which improves scene realism. If there is multiple tools in similar function, adding/removing a single tool will not make big difference. And if the tool has negative effect, the agent will recognize it during iterations and avoid using the tool.

Q7: Room structure & single room scale

We remove windows and doors to simplify the generation process, though our method can handle such structures and update the scene accordingly. Due to the format limitation of rebuttal, we can show some examples in the final version.

For room scale, we focus on single rooms to improve scene quality as prompted by the user. The process can be repeated to handle multiple rooms. We could also generate multi-room scenes at once with some coordinate transformation.

Q8: time limits

The average time for a single iteration is 8.6 minutes, and the average time for generating a complete scene is 64 (min:35, max:130) minutes. Simpler and smaller room need less generation time. Here are some choices to reduce the time cost:

• reduce the usage frequency of the physical optimization
• remove some time-consuming tools, such as digital cousins. Although the results will be influenced, it will not cost too much drop.
• reduce the iteration number.

Q9: unrealistic scaling/placement for functional items

That is a good point. The problems comes from the non-standard open-vocabulary dataset in two points. On one side, the object in open-vocabulary dataset such as Objaverse has no standard size, for example, a toy can be 10 meters in length. Thus resize is neccessary for those assets to eliminate the gap and fit the realness. On the other side, the front direction of the asset is not provided in the dataset, thus we utilize VLM to predict the front direction, which we find with high success rate, but it still failed in some cases since sometimes the front direction of the asset can not be defined uniformly, such as the elliptical bike, our expected front direction is mistaken as the side direction, leading to unreasonable orientation and size. How to balance the standardization and generality for the open-vocabulary dataset is an unsolved problem, but it still provide a feasible and promising view for general scene synthesis.

Q10: Articulations and squashing.

That’s a good point. We will pay attention to the articulation and squashing in the future work.

Q11: limited complexity of relationships

The relationship is taken from the Infinigen with 3 types of obj-room relations and 7 types of obj-obj relations. This covers most of the common relationship. For more "well aligned" content, the code is also easily extendable for more complex and detailed relation.

Q12: Implementation of the tool & executor

We list the implementation of each tool.

During the planning process, the agent is provided with brief description of each tool, shown in A3-A13 in supplementary material and use the function calling to choose a single tool for each step. Then it will run the tool itself and go through the executor to update the scene.

Note we not only contribute on the tool itself, but also modify the infinigen to :

• update iteratively
• interactive in blender in realtime with socker
• fit each tools rather than generate scene procedurelly.
• Physical optimization
• Add 3D marks and 2D top-down rendering
• …

Q13: prompt engineering

We appreciate your insightful observation regarding the prompt engineering efforts in our work. We follow the structure of Octotools and design prompts for planner, tool cards, executor, and reflector, which makes the framework accurate, stable, and easily extensible. Other baseline methods (IDesign, LayoutVLM, Holodeck) also design several prompt for each step, which is a common choice for using the LLM and agent for generation task.

Q14: Typos error

Thanks for pointing out these problems. We will fix the typo error in the final version.

Thank you for the valuable feedback and suggestions.

Q1: Explanation of physical metrics

We refer to PhyScene to evaluate out-of-boundary objects #OB and collided object pairs #CN, which indicate the reasonbale layout from two aspects. Although there is a tie between these 2 metrics, we could also achieve the best performance for both of them, indicating the effectiveness of a well-designed agent framework. Meanwhile, we refer to IDesign and evaluate the number of objects #Obj. A higher number of objects means more objects to interact for embodied AI and also leads to higher diversity of the generated scene.

Q2: Increase sample size and inter-annotator agreement in user study.

User Number and Sample Size

In the main paper, we following prior work LayoutVLM for the user study, thus we only recruited 5 students. To make the user study more solid, we improve the user number to 20 and increase the evaluated scene number to 200, each of them is shown 40 samples. We show them the same instructions used as prompt in the LLM reflector for the participant to evaluate the scene carefully. We statistic the results into the following table, which indicates similar results with the previously shown Table 3 in the main paper.

Score Alignment and Validity of Evaluation

Note the evaluation results of human-evaluation and LLM-score can not be exactly the same, thus a slight difference between them is a normal phenomenom. To further confirm the stability of these justification method, we check the ailgnment between different users as well as LLM. We convert the users' and LLM's scores on different scene to rankings and calculate Kendall’s Tau for the 4 metrics:

Results show strong user-user agreement and user-LLM agreement on different metrics. And we find that user-LLM has higher alignment score than user-user, which indicates that LLM is more stable than user in evaluation. These alignment results also demonstrate the effectiveness of using LLM for justification in Table 2/3 of the main paper.

Diversity The diversity of our method is high, due to the extendable tool cards and generation ability of foundation model. We implement the user study to evaluate the diversity of scenes.

Specifically, we increase the generated scene number from 1 to 3 for each method in the second comparison setting in order to 1) reduce individual variance and 2) check diversity of different generation system. We ask the participants to choose:

• Which method do you prefer?
• Which method has greater diversity?

Results shows our methods has greater diversity while also stand for higher preference, surpassing other methods in a large degree.

Q3: stronger citations for VLM judgement

Thanks for this suggestion. We have checked the preprinted citations in our submission, and the LayoutVLM and FirePlace has been accepted to CVPR 2025, which utilize VLMs for 3D spatial reasoning. Additionally, GPTEval3D[1](accepted by CVPR 2024) proves that VLM can serve as a Human-Aligned Evaluator for Text-to-3D Generation. SceneCraft[2] (accepted by ICML 2024) also utilizes VLM to critic and revise the scene with Blender script. We will update the citation in the final version.

[1] Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation. In CVPR, 2024.

[2] Hu Z, Iscen A, Jain A, et al. Scenecraft: An llm agent for synthesizing 3d scenes as blender code. ICML, 2024.