We are grateful to Reviewer4 for the thoughtful assessment and positive evaluation of our work. We appreciate your recognition of 3D-SynthPlace, the two-stage DPO framework, and the clarity of our presentation. We now address your questions and concerns in detail.

R4Q1. How is the object retrieval module implemented, and how will it influence final scene quality?

Response:

We follow a purely text-based retrieval approach, inspired by HOLODECK (Yang et al., 2024c), to match the objects mentioned in the user instruction to assets in our 3D object database:

1. Let $t'$ be the target object’s name from the wrapper, and $t$ be the annotation of a candidate 3D asset from the re-annotated the Objervase dataset.

2. We compute cosine similarity between their sentence embeddings using SBERT (all-mpnet-base-v2):

   $$T(t, t') = \cos\big(\mathrm{SBERT}(t),\,\mathrm{SBERT}(t')\big)$$

3. The asset $o$ that yields the highest similarity is selected as the retrieved item for placement.

This ensures semantic coherence even without explicit spatial constraints, making our retrieval robust against variations in descriptive text.

Retrieval Performance:

This retrieval approach has been validated within both Holodeck and LLPlace, and we have conducted dedicated tests on our retrieval module:

• Explicit type queries (98% accuracy): When the input query explicitly specifies an object category (e.g., “sofa”, “desk”), the system returns the correct category as the top result 98% of the time.
• Rich, free‑form descriptions:
  • Top‑1 retrieval accuracy (86%): In 86% of cases, the single highest‑scoring asset exactly matches the ground‑truth asset.
  • Top‑3 retrieval accuracy (94%): In 94% of cases, the ground‑truth asset appears somewhere among the three highest‑scoring candidates, providing a small ranked shortlist for follow‑up selection.

Therefore, we believe the retrieval step has minimal impact on overall performance. Our 3D asset library is based on Objaverse, with refined and precisely annotated object labels, and is easily extensible beyond the 3D-SynthPlace dataset. The wrapper filters and standardizes user instructions, extracting clear room and object specifications. Retrieval then uses these parsed descriptions to match suitable assets. As long as the object type is clearly stated, retrieval remains reliable. For details on the wrapper, please see our response to R3Q1(2).

R4Q2. How is consistency between the generated layout and user instructions evaluated?

Response:

We fully understand your concern. In response to R3Q1(2), we have provided a detailed explanation of how the Qwen2.5-7B wrapper is used to parse natural language instructions into a standardized JSON template, and extract information. This design helps eliminate redundant or irrelevant information and ensures a consistent input format. As further discussed in R4Q1, our text-to-asset retrieval strategy reliably maintains semantic alignment between the input and the generated layout. We also evaluated the model’s instruction-following ability under both highly specific and vague prompts (see R3Q4), assessing its robustness to varying instruction granularity and complexity.

R4Q3. Is semantic reasoning in SFT necessary, given the small performance gain shown in main paper's Table 6?

Response:

Thank you for this insightful question. It is true that the performance gap between SFT with and without semantic reasoning appears small in terms of OOR. As shown in Table 6, the object overlap rates are nearly identical, and scene usability improves by only about 5 percentage points with reasoning. This is largely because, in the SFT-only setting, any object overlap renders the scene unusable. As a result, OOR dominates usability, and semantic reasoning has limited impact on the OOR metric itself. However, the reasoning step provides high-level spatial priors—descriptions of relative object positions—that guide the model toward more coherent and human-aligned layouts. While OOR may remain similar, this reasoning significantly improves overall scene structure and layout plausibility, leading to higher usability. To further clarify, we include below the quantitative comparison of SFT+DPO1+DPO2 with and without semantic reasoning:

Table 8:Compare the performance between with/without semantic reasoning

These results show that although the OOR only slightly drops without reasoning, scene usability drops noticeably. This confirms that semantic reasoning helps the model produce more usable and human-preferred layouts, even if the overlap metric is not significantly affected.

R4Q4. How is the layout editing task implemented, and how is the dataset constructed and formatted?

Response:

Thank you for your question. To enable layout editing using the same generation model, we constructed a dialogue-style editing dataset inspired by LLplace. Each original scene with more than four objects was converted into a two-turn dialogue:

1. In the first turn, we randomly removed a subset of objects (40% probability per object, excluding essential items like beds, tables, and sofas) to create a corrupted layout.
2. In the second turn, we either added or removed these objects depending on the task type. For the add task, we provided the corrupted scene and the list of removed objects as input (add=True), expecting the model to reconstruct the original full scene. For the remove task, we gave the original scene and the target objects to remove (remove=True), expecting the model to generate the edited scene. Each sample includes structured object attributes (category, position, rotation) and uses <|eot_id|> to separate dialogue turns. In total, we created 5000 editing samples (3000 add, 2000 remove). The input format mirrors our original generation format, with added flags and object lists to indicate the editing intent. To adapt the model for editing, we further trained it for 5 additional epochs on this dataset, continuing with the original model. This improves the model’s ability to handle scene edits in a conversational context.

R4Q5. More ablation study results with GPT4o and FID.

Response:

We provide a complete ablation study that includes FID and GPT‑4o scores to assess the contribution of each training component and the effect of high-level：

Table 9: More Ablation Study with FID and GPT4o Score

As shown in the updated table, after incorporating both FID and GPT‑4o scores, the model's performance continues to improve progressively across different stages, exhibiting a clear stepwise enhancement. Each component contributes meaningfully to the generation quality, validating the overall design of our pipeline.

R4Q6. What noise levels are used in DPO Stage II, and how do different levels affect performance?

Response:

For this experiment, we tried three noise levels. Our perturbations covered several types—objects placed outside the room, overlapping, and random placements. Before settling on the current medium setting, we evaluated a heavy setting (severe overlaps, large out‑of‑room shifts, or fully random layouts) that made the preference gap between positive and negative samples too wide. We also tried a light setting, where objects were moved only slightly; this made the positive‑negative gap very small. We trained with each noise level and evaluated 100 scenes per setting with OOR and UR:

Table 10: Impact of Noise Levels on OOR and UR in Stage-II DPO

While heavy noise leads to a sharp drop in OOR, it also results in the lowest usability. We believe this is because when negative samples are extremely poor, the model quickly learns to avoid only the most obvious errors. This causes it to converge prematurely, without learning to handle more nuanced spatial relationships—ultimately leading to layouts that still contain moderate overlaps or awkward placements. On the other hand, light noise provides minimal supervision beyond Stage I, offering little signal for further refinement. As a result, performance shows limited improvement and may even slightly degrade. The medium noise setting strikes the best balance: it introduces enough discrepancy to guide learning while avoiding extremes that cause saturation or instability. This produces the most effective training signal, yielding the best overall performance in Stage-II DPO.

R4Q7. Does the model support layout generation based on object spatial relationships?

Response:

Thank you very much for this question. It also serves as a test of the model’s instruction-following capability. When processing user instructions through our wrapper, users may wish to specify object-to-object relationships as priors for the model input prompt. We have already discussed this issue in our responses to R3Q4, which we kindly refer you to for more details.

R4Q8: Some notations and equations are confusing.

Response:

Thanks for the helpful suggestion. We acknowledge that some notations may be ambiguous or inconsistent. We will revise and standardize the notation in the final version to ensure clarity and avoid confusion with related concepts.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

We sincerely thank you for the encouraging feedback and thoughtful summary. We are pleased that you found our dataset, two-stage DPO framework, and overall method design valuable. Below, we address your comments and respond to the specific concerns you raised.

R3Q1. Clarify key missing details in the SFT stage: supervision methods, instruction-to-JSON conversion, and the object retrieval process?

Response:

We agree these aspects were not sufficiently clear in our manuscript. We clarify each sub-question explicitly below, and will incorporate these detailed explanations into our revised version:

1. Supervision during SFT:

During SFT, the model jointly generates (A) a semantic summary $S^{\text{sem}}$ and (B) a structured JSON layout with 3D coordinates $c_{1\ldots N}$ and orientations $r_{1\ldots N}$. We serialize both outputs into token sequences and apply token-level cross-entropy loss against human-annotated ground truth. This unified supervision encourages the model to capture both global semantic intent and precise spatial configurations in a single forward pass.

2. Instruction-to-JSON Wrapper:

We use a Qwen2.5-7B-Instruct LLM as a wrapper to convert natural language user instructions $\mathcal{U}$ into structured JSON $\mathcal{X}$. The process works as follows:

• The wrapper extracts room size, room type, object types, and object counts from the instruction.
• If any field is missing, the wrapper heuristically infers it based on existing information (e.g., number and type of objects).
• It then retrieves suitable 3D assets for each object (details in part 3).
• The extracted values are inserted into a predefined JSON layout template, which is then used as part of the prompt input to OptiScene for inference. This wrapper provides robustness by normalizing diverse instructions, modularity for easier debugging, and flexibility through noise filtering. It is also easily extensible—indeed, we further extended it in R3Q4 to extract spatial relationships between objects.

3. Object Retrieval Process:

We follow a purely text-based retrieval approach, inspired by HOLODECK (Yang et al., 2024c), to match the objects mentioned in the user instruction to assets in our 3D object database. Due to the character limitation, please refer to the R4Q1 for more details, where we discuss the object retrieval process and show the performance test of the text-based approach.

R3Q2. How does the method handle naturally valid bounding-box overlaps? Are these explicitly modelled?

Response:

Certain overlaps (e.g., chairs under desks) naturally occur in realistic scenes, and explicitly handle these cases as follows:

• Dataset Preservation: We intentionally retain the valid overlapping layouts (e.g., chairs tucked beneath desks) in our synthetic dataset—474 scenes (2.8% of total), predominantly in living rooms—allowing the model to recognize natural overlap distributions without skewing its collision avoidance capability.
• Prompt Engineering (Appendix Table 7): Our prompts explicitly instruct minimal distances for specific object pairs (e.g., chair–desk), guiding the model to correctly interpret these overlaps as intended and acceptable rather than as collisions.
• Stage 2 DPO Noise Injection: During DPO Stage 2, we preserve chair–desk overlaps as positive examples, avoiding random perturbations on these pairs. Conversely, overlaps involving inappropriate object pairs (e.g., sofa–sofa intersections) remain labelled as negative examples. Thus, our approach robustly differentiates between valid overlaps and unintended collisions, ensuring realistic, high-quality layout generation.

R3Q3. How does the method handle biases inherited from Holodeck and limited negative sample diversity during DPO training?

Response:

We clarify that our training dataset is not directly derived from raw Holodeck outputs. Instead, we carefully filtered and applied handcrafted refinements to ensure data quality and alignment with human preferences. Nonetheless, we acknowledge the possibility of residual biases from the Holodeck remaining in our data, which we addressed systematically:

• Manual Selection of Positive Samples: During the DPO stage 1, we manually selected layouts that closely align with human preferences, ensuring positive samples specifically counteract residual biases inherited from Holodeck and original 3D-FRONT datasets.
• Two-stage DPO Training: To enhance the diversity and effectiveness of negative samples, we introduced a DPO Stage 2. We deliberately applied controlled perturbations—including slight random displacements, overlaps, and out-of-bound placements—to relatively high-quality layouts, increasing the model’s ability to robustly differentiate valid and invalid configurations.
• Temperature-based Diversity for Negative Samples: Additionally, during DPO Stage 1, we generated negative samples by varying the sampling temperature from the SFT model. Consistently similar negative outputs at different temperatures indicate persistent errors that require repeated exposure, whereas varying outputs suggest multiple error patterns beneficial for robust training. Moreover, using SFT-generated negative samples for DPO training is a validated approach, as demonstrated in prior works [1,2].

[1] Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation

[2] Iterative reasoning preference optimization

R3Q4. How does instruction complexity affect generation performance?

Response:

To support consistent layout generation across varying instruction types, we first enhance the wrapper, which parses natural language instructions into structured JSON representations by extracting room attributes, object descriptions, and (when available) object-to-object spatial relationships. We design the wrapper to handle two main cases:

• For highly specific and directional prompts, we use the wrapper to extract spatial relations and then encode them as [objects relationship] tokens, which are injected at inference time to guide generation without requiring retraining.
• For vague or underspecified prompts, where explicit object information is lacking, we rely on the wrapper to creatively infer plausible object categories and quantities from contextual cues.

Instruction Complexity Evaluation

We first assessed the model’s robustness to instruction complexity by categorizing prompts into three levels: Simple (≤5 objects, precise object description and number), Medium (6–15 objects, moderately vague description with some object relationships), and Complex (15-20 objects, highly ambiguous). Each group contains 50 prompts evaluated on living room scenes. The wrapper and retrieval process achieves over 95% parsing success across all levels. However, under complex or vague instructions such as “some chairs” or “a few shelves,” the number of extracted objects tends to vary (typically between 2–4), leading to decreased layout reliability.

Table 5. Generation performance under different instruction complexity levels

Despite the increased difficulty, our model maintains reasonable performance even under complex instructions, achieving an average GPT-4o score of 7.1 and a usability rate of 30%. This demonstrates the robustness and stability of both our wrapper and the generation model when handling ambiguous or underspecified inputs.

Instruction-Following with Extreme Prompts

We further evaluate how the model handles two extreme instruction types: (1) Highly specific and directional instructions, (2) Extremely vague and underspecified instructions. Each of these settings was evaluated over 20 test cases across four room categories.

Directional Instructions

To evaluate the model's ability to follow directional prompts, we measure both scene-level and instruction-level relation success rates, along with OOR and scene usability, as summarized in Table 6. A scene is considered successful only if all specified spatial relations are satisfied. Under this criterion:

• 12/20 scenes met all spatial constraints (60% scene-level success)
• 30/46 individual relations were followed (65.2% instruction-level success)

Table 6: Performance on directional prompts

We also evaluated single-instruction examples. For instance:

• “Place the desk parallel to the bed” → correct placement in 66.7% of 12 cases
• “Place the chair next to the desk” → correct placement in 80% of 20 cases

These results suggest that, despite not being explicitly trained on spatial language, the model exhibits a limited but notable understanding of spatial semantics. However, the model often fails with instructions involving corner placements, likely due to the absence of explicit geometric encoding of room corners.

Vague Prompts

To assess robustness under ambiguous inputs, we tested prompts containing vague quantifiers (e.g., “some chairs”) or high-level intents (e.g., “a cozy living space”). We report JSON parse success and scene usability:

Table 7:Performance on vague instructions

In 65% of cases, the wrapper generated valid object layouts, but many still suffered from usability issues (e.g., missing object types or implausible combinations). The remaining failed parsing due to overly abstract input or absence of essential categories (e.g., no sofas in a living room). While the wrapper handles moderately vague prompts reasonably well, it struggles with highly underspecified instructions. We plan to explore this direction in future work.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

We thank Reviewer 2 for the clear summary and positive evaluation. We appreciate your recognition of our motivation, modular design. Below, we address your concerns and provide clarifications on the specific points you raised.

R2Q1. Clarification on Object-Centric Navigation (L302)

Response:

Here is our answer about navigation with LLM:

We first provide clarification on the navigation setting as follows:

For navigation experiments, we adopt the settings in Benchmark 1: Object Loco-Navigation in InfiniteWorld [1]. Specifically, the process by which a robot/agent explores a scene to searches for a target object is as follows: First, a 2D grid representation of the environment is constructed through an occupancy map, which divides the space into free zones, obstacle zones, and unknown zones. This grid serves as the foundational spatial reference for subsequent path planning, enabling the agent to identify navigable areas and avoid obstacles. Then, natural language instructions such as "Find an [object] in [room]" are parsed: LLM extracts semantic information from the instruction to determine the target object (e.g., "black bedside table") and its approximate location clue (e.g., "in the bedroom"), and further queries the scene's built-in object coordinate data or real-time scene graph to obtain the specific coordinates of the target object. Subsequently, the path follower, based on the 2D grid, uses an algorithm to plan the optimal path from the agent's current position to the target coordinates. If the target coordinates fall within an obstacle zone, the nearest non-colliding point is selected as an alternative target. During movement, the agent proceeds along the planned path, with the final requirement that the target object is within its 60-degree horizontal field of view and within 2 meters. If these field-of-view conditions are not met due to occlusion or other reasons, the task fails.

Then we discuss the connection from navigation to OptiScene:

Although the navigation algorithm itself is not modified, we evaluate it on scenes generated by OptiScene. Since indoor datasets suitable for such embodied tasks are limited, our method helps expand the data space for training and testing. The fact that existing navigation algorithms can be directly applied to scenes generated by OptiScene, and achieve comparable behavior, validates that our synthetic scenes are well-aligned with real ones and usable for downstream embodied tasks such as navigation. And also, we think navigation is one kind of way to evaluate the usability of the generated indoor scenes. As in R1Q3, we introduce navigability to test models' performance. Although navigability is different from the object loco-navigation results, they both show the overall spatial rationality of the scene.

Ref:

[1] Ren, P., Li, M., Luo, Z., Song, X., Chen, Z., Liufu, W., ... & Liang, X. (2024). Infiniteworld: A unified scalable simulation framework for general visual-language robot interaction

R2Q2. Is the semantic reasoning step part of the input prompt or the model’s internal reasoning process? Would alternative high-level inputs (e.g., fixed positions, centroids, BEV images) also improve performance?

Response:

Thanks for your question! For the first part of the question, we apply the semantic reasoning because we believe that if we let the model try to understand the relationship between each other can help to generate more reliable 3D scenes. So we construct this data in the training dataset and make the semantic reasoning as a step in the chain of thought before outputting the final 3D scene result. So this is not a part of the prompt for the model; we just try to let the model learn object-object relationships of the scenes.

And for the second part of the question. As we understand, the reviewer would like us to test fixing the positions of certain objects as the input. For example, when generating a bedroom, first specifying the bed’s position and then predict the positions of the remaining objects. We believe this experiment is interesting and clearly demonstrates our model’s robustness, so we performed inference on four room categories while fixing the positions of some selected objects (e.g., Bed, sofa, bathtub...). This setup also aligns closely with the downstream task of scene editing we discussed in the paper. In addition to directly performing inference with fixed object positions, we further included a fine-tuned version of our model specifically trained for editing as a baseline comparison. And we provide the quantitative results based on key metrics, Object Overlap Rate (OOR), GPT4o, and Usability:

Table 3: Compare the Fixed Position OptiScene with other settings

As shown in Table 3, even when fixing the positions of certain objects during inference, the model maintains good performance with only a slight drop, which demonstrates our model's robustness.

Additionally, we conducted further experiments using highly specific user instructions that include object relationship descriptions (see R3Q4 for details). The results show that, without any re-training, our model can leverage the generalization ability of the LLM to interpret such structured instructions and still generate reliable 3D scenes. But our model is designed to use an LLM to generate spatial placements of indoor objects and does not currently integrate other modalities (like 2D image or 3D point cloud) or change the overall architecture to help the scene generation. We agree that introducing other modalities is an exciting direction, and we plan to explore it in future work.

R2Q3. How do the proposed solutions perform with more complex scene layouts? For instance, 5, 10, 20, 30 objects. Is there any limitation in this matter?

Response:

Thank you for your insightful question. Evaluating the robustness of our model under varying scene complexity is indeed crucial. To address this, we conducted comprehensive experiments on scenes containing 5, 10, 15, 20, and 30 objects, sampling 20 living-room scenes for each scenario and evaluating performance using Object Overlap Rate (OOR), FID scores, GPT-4o ratings, and usability metrics. We compared our results against several baselines, including DiffuScene, LLPlace, and Holodeck. While Holodeck's OOR always be 0 due to its processing step that enforces, we won't compare Holodeck's OOR in the table. As summarized in the table below, our method consistently outperforms baselines (DiffuScene, LLPlace, Holodeck) across most complexity levels, maintaining strong performance up to 15 objects. However, at higher complexities (20+ objects), especially at 30 objects, the model exhibits significant performance drops (e.g., OOR rises to 0.289, usability drops to 0%), clearly indicating a limitation. We attribute this decline primarily to the scarcity of densely-packed scenes in our training data, resulting in an out-of-domain scenario at very high object counts. This limitation is prevalent across other state-of-the-art methods as well as Holodeck, for example, completely fails to generate coherent layouts at 30 objects, even after we change the code to enlarge room sizes to accommodate them.

Table4. Performance Across Object Counts (avg. OOR ↓ / avg. GPT‑4o ↑ / UR ↑)

As for the unseen objects question. We discuss the concern about how our model handles previously unseen object descriptions as follows:

• (1) Retrieval-based Generalization: As discussed in R3Q4 and R4Q1, our model relies on a retrieval module that maps natural language object descriptions to corresponding 3D assets. This module achieves a 98% retrieval accuracy for matching object classes, ensuring reliable selection even for unseen or varied descriptions.
• (2) LLM Generalization + Test-time Coverage: The LLM demonstrates strong generalization ability in interpreting semantically novel or rare descriptions (e.g., mapping “seats” to “chair”). Moreover, our test set already contains many object descriptions not seen during training, and the model successfully handles them, confirming its robustness to unseen inputs.
• (3) Focus on Large Floor-based Furniture: Our task targets large furniture items that are placed directly on the floor (e.g., beds, sofas, wardrobes), rather than small or decorative objects. This reduces ambiguity and improves consistency in object grounding.
• (4) High-quality 3D Asset Pool: All retrieved assets come from our annotated 3D database based on Objaverse, ensuring sufficient diversity and realism to support generalization.

In summary, our model demonstrates robustness and quality in moderately complex scenes, yet encounters significant challenges at extremely dense object arrangements with an extremely large number of objects (>20). To improve performance in such scenarios, expanding the training set to include richer and denser configurations, combined with targeted data augmentation strategies, will be essential future directions. Moreover, we believe our model is capable of handling unseen objects during inference in the context of indoor scene layout generation.

I partially agree with the authors about OptiScene’s role in the navigation task as presented in the current manuscript. From my understanding, using OptiScene increases the amount of training data available for the navigation tasks (as the authors mentioned in their rebuttal, ' it helps expand the data space for training and testing'). However, the experiments showed in the manuscript and in the rebuttal do not convey this idea. OptiScene was mainly used for testing predefined solutions for navigation. The impact of OptiScene for training was not presented. Additionally, it remains unclear to me how OptiScene may helps for navigation task in larger environments, with multi-rooms, corridors, and multiple areas, which is the main setting for the embodied navigation tasks.

Nevertheless, based on the authors' response R2Q3, I want to emphasize the clear contribution of this paper. Although, the proposed solution’s performance declines as the number of objects increases (i.e., which is totally expected), it still significantly outperforms the baselines. The selected baselines are both appropriate and fair, and the current results provide strong evidence for the manuscript’s contribution.

Even though, there are some minor flaws, I keep my rating as borderline accept.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

We sincerely thank you for the positive feedback. We appreciate your recognition of evaluation rigor, and the effectiveness of our RL-based fine-tuning. Below we address your questions.

R1Q1: Omitted small elements (doors, windows, wall‑mounted items) not addressed despite their inclusion in Holodeck/FlairGPT.

Response:

First, our model is designed for indoor scene layout generation with a focus on placing large furniture items. Accordingly, we extended our dataset based on the 3D‑Front format. The original 3D‑Front does not include doors, windows, or wall‑mounted objects, so—for a fair comparison with prior methods—we omitted those elements during training. As in previous methods, DiffuScene, LLplace, etc., do not predict the wall-mounted objects, we follow this setting.

However, when we generated synthetic scenes with Holodeck and applied post‑processing, we did retain the door/window and wall‑object annotations. In this setting, we then ran two new experiments: one trained solely on 3D‑SynthPlace data augmented with doors, windows, and other wall objects; and another trained on a mix of the original 3D‑Front data (without wall objects) and the 3D‑SynthPlace-with‑wall‑objects data. We then inference 100 times on four room categories. The results are demonstrated in Table 1：

Table 1: Effect of including small wall‑mounted elements on placement recall and overall layout quality.

While our method focuses on generating layouts with large furniture, we intentionally did not include small objects in our training data. Our primary focus is indoor scene layout generation, consistent with prior works such as ATISS, DiffuScene, InstructScene, I-design, LLplace, LayoutVLM, and LayoutGPT—all of which emphasize the placement of large furniture. The arrangement of small objects (e.g., books, cups, desk clutter) is a fine-grained subtask that we consider out of scope for this work and plan to explore in future extensions. Our goal in constructing the dataset is to improve large-object layout modeling, which is foundational to scene generation. As shown in Table 1 of our main paper, our dataset significantly improves the performance of strong baselines such as LLplace and DiffuScene. This aligns with our core motivation: to augment and refine the 3D-FRONT dataset to support high-quality, human-aligned layout generation. While Holodeck includes many small items, their placements are often random and unrealistic, such as scattered desktop objects or items placed directly on the floor. Including such noisy placements would distract the model from learning coherent spatial structure. Therefore, we intentionally excluded unfiltered small-object data from Holodeck to maintain semantic focus and layout consistency. But we think adding the windows/doors/wall-mounted objects can enhance scenes realistic, we will update the dataset with these and include Table 1 in the next version paper.

R1Q2. The observation that "SFT memorizes while RL generalizes" is not novel—does the contribution lie in applying it to scene layout generation for the first time?

Response:

While it is well known that SFT tends to memorize its training data whereas reinforcement‑style methods generalize (Chu et al.), to our knowledge, OptiScene is the first work to integrate Direct Preference Optimization (DPO) on top of SFT specifically for 3D scene layout generation. We begin by fine‑tuning the model on the annotated dataset and then perform extensive human‑in‑the‑loop curation to extract only those layouts that reflect natural, human‑preferred living environments.

In Stage 1 DPO, these curated positive examples teach the model to correct SFT’s memorization biases and prioritize realistic furniture arrangements—effectively aligning it with genuine human preferences. In Stage 2 DPO, we introduce controlled hard negatives— random translations, overlaps (e.g., a floor lamp inserted into a sofa; excluding overlaps common in normal scenes such as a chair under a desk), and minor out‑of‑bounds placements—to force the model to distinguish plausible layouts from artifacts. As reported in Table 6 of main paper, our two‑stage DPO pipeline delivers significant improvements in scene usability and substantially reduces collision and overlap errors compared to an SFT‑only baseline, demonstrating that DPO not only mitigates memorization but also produces layouts that more faithfully mirror real‑world living spaces.

R1Q3. Using LLMs (e.g., GPT-4o) as layout evaluators via 1–10 rating scales is unreliable—why do we use this, and what can we test more?

Response:

Assessing the usability of generated scenes, which is closely aligned with human preference, is indeed a challenging task. In this field, leveraging GPT-based models for evaluation has become a widely adopted practice, nearly all recent works—including LLplace, I-Design, and LayoutVLM—have relied on GPT-4o for scoring. In our case, we do not simply feed the generated JSON layouts into GPT-4o. Instead, we render the scenes into images and then provide these images as input to GPT-4o for evaluation.

We believe that thanks to GPT-4o’s extensive prior knowledge and alignment with human preferences, it provides a reasonable proxy for assessing output quality. However, we also acknowledge its limitations. The final score can be influenced by rendering quality, lighting, and camera perspectives, which may not directly reflect the underlying model’s generation capability.

To address this, we complement GPT-4o scoring with additional metrics such as FID, OOR, and a human usability score to provide a more comprehensive evaluation of model performance. We agree on the need for a unified quality metric and are building a comprehensive benchmark, inspired by SceneEval, that uniformly renders each scene in Blender from a top‑down view and evaluates against the original JSON and user instruction. In addition to our existing OOR (object overlap rate), FID, GPT‑4o scores, and usability rate metrics, we will include:

• CNT (Object Count): compare expected vs. detected object quantities by rendering each object’s front view and using Qwen2.5‑VL‑7B-Instruct to classify and count instances.

• NAV (Navigability): compute the ratio of the largest connected free‑space region in a 2D occupancy grid.

• OOB (Out‑of‑Bounds): analytic check of bounding boxes crossing room boundaries.

• ACC (Accessibility): identify each object’s main functional face via Qwen2.5‑VL‑7B-Instruct and compute the unobstructed area in front of it.

This streamlined suite combines rendering, JSON validation, and automated VLM/analytic checks to robustly assess scene completeness, correctness, and real‑world usability. The model we tested is the version we retrained with the 3D-SynthPlace dataset:

Table 2:Comparison of DiffuScene, LLplace vs. our method across all benchmark metrics.

As shown in Table 2, our method outperforms DiffuScene and LLplace across all key metrics, achieving lower overlap and FID, higher GPT‑4o scores and usability, as well as stronger CNT, NAV, OOB, and ACC performance. This demonstrates the effectiveness and robustness of our proposed approach. We will include all of these metrics in the appendix of the future version. However, scene generation research still lacks a clear, unified benchmark, so we are developing a new evaluation suite that builds on our current metrics by adding curated test sets with standardized rendering protocols, rigorous FID comparisons under consistent conditions, and a CLIP‑based evaluator fine‑tuned on paired scene images and text instructions—together providing a more robust and reproducible standard for future work.

R1Q4: Does the structured JSON proposed here bring any advantage over other very similar representations proposed in related works?

Response:

Thank you for the question. We chose a structured JSON format for representing 3D scene layouts because it offers an effective balance between human readability and machine interpretability. The format uses clear key-value pairs such as "x": 1.2 and "rotation": 90, which are easy to read and understand. At the same time, JSON supports schema validation, type checking, and straightforward extensibility, allowing new fields to be added without disrupting existing workflows.

Unlike formats such as GLB or USD, which tightly integrate geometry, textures, and scene graphs, JSON separates spatial properties from rendering details. This separation makes it particularly suitable for large-scale layout generation and editing, where assets can be dynamically loaded and replaced in engines like Isaac Sim or Blender. In contrast to CSS-style in LayoutGPT, 2D layout representations that rely on properties such as left, top, width, and height and often require complex transformation logic to describe spatial arrangements, JSON provides a more precise and semantically rich encoding of full 3D information, including position, rotation, and size. It is also inherently more extensible, allowing new spatial attributes or metadata to be added without altering the existing structure.

Most importantly, JSON serves as a practical intermediate format between natural language and executable code. It enables LLM to convert user instructions into structured scene layouts that can be directly used in downstream tasks such as 3D modelling and rendering in Blender or Isaac Sim.

Dear authors,

Thanks for the rebuttal! I am satisfied with the resonse and shall be increasing my score. I would recommend including these discussions and evalaution in the paper to strengthen it. In the revision, please cite past works which have studied "SFT memorizes while RL generalizes" and explictly state the JSON format is used in other works and have inspired the method. I do not think including them would diminish the novely aspects, as the authors correcly state that it has not been studied in this particular context for RL. I would also advise adding some results with small objects.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.