Q5jV-Q1. Why not directly use GPT to generate scene graphs as part of the chain-of-thought process (similar to [1]), instead of building a new SGClip model to learn the results perceived by GPT?

We appreciate the reviewer's valuable feedback and evaluate against CoT-SG [1], a direct baseline where scene graphs are generated through a Chain-of-Thought MLLM pipeline. This approach prompts the MLLM to produce structured scene graph representations directly, without grounding. Further studies are detailed in kXSe-Q1.

EB-Navigation Results: ESCA consistently outperforms CoT-SG across all tested models. The Chain-of-Thought strategy actually degrades performance, particularly on long-horizon tasks where target objects are not initially visible. CoT-SG often fails to detect target objects after performing a few rotation actions to scan its surroundings, and it exhibits more hallucinations than the raw MLLM.

EB-Manipulation Results: While CoT-SG shows modest improvements over raw models, ESCA maintains superior performance across all tested MLLMs, demonstrating the effectiveness of structured grounding over direct generation approaches.

Q5jV-Q2. … raises concerns about whether the inclusion of too many relation keywords and actions could exceed the model's context length, potentially impacting both the effectiveness and efficiency of the MLLM.

Training vs. Inference Distinction: While our training dataset (ESCA-Video-87K) contains a large vocabulary (35,415 relation keywords, 57,930 actions, 220,905 names) reflecting natural language diversity, this does not translate to excessive token usage during inference. SGCLIP's core mechanism relies on cosine similarity between visual and textual embeddings rather than exhaustive binary classification over the entire vocabulary.

Efficiency Design: ESCA's four-phase architecture inherently prevents vocabulary explosion through progressive refinement. The concept extraction phase generates only task-relevant keywords specific to the given instruction and visual scene, while both object identification and scene graph prediction work within this constrained vocabulary.

Empirical Analysis: Our quantitative analysis on EB-Navigation shows that ESCA generates an average of 5.57 keywords per instruction, producing 84 unique target objects, 199 related objects, 296 relationships, and 117 attributes across the entire dataset. Even in the most complex scenarios, the total token count for scene graph representations remains manageable.

Q5jV-Q3. The system relies on 2D visual inputs instead of video or 3D representations, which limits depth-aware reasoning and spatial precision.

We believe 3D representation generation is definitely important for embodied agents application. However, this direction is beyond the current scope of our work, which focuses on visual grounding using the available 2D egocentric inputs as defined by the benchmark setup. We will clarify this design choice in the paper and include 3D integration as a promising direction in our future work section.

Q5jV-Q4. The entire system requires multiple calls to models such as MLLM and Grounding DINO, which may result in low operational efficiency.

ESCA achieves significant perceptual grounding improvements while maintaining computational efficiency. Our method introduces only 1.09 seconds of additional processing per planning step, less than 5% overhead compared to baseline systems. To put this in perspective: on the EB-Navigation benchmark with GPT-4o, a complete planning step takes 22.25 seconds with ESCA versus 21.16 seconds using the baseline MLLM + Grounding DINO approach. This minimal 5% increase in runtime delivers substantial benefits through structured, interpretable scene graphs that significantly enhance visual-semantic alignment. We consider this an excellent tradeoff, the marginal computational cost enables ESCA to generate rich scene representations that drive superior task performance and generalization across diverse embodied planning scenarios. The interpretability and accuracy gains far outweigh the modest time investment.

kXSe-Q1. Necessity of the SGClip: Could the authors provide a direct comparative analysis of SGClip's accuracy, robustness, and inference efficiency in relationship prediction against directly prompting an advanced VLM (e.g., GPT-4o) to describe inter-object relationships, assuming the same underlying Concept Extraction and Object Identification?

To clarify SGCLIP's critical contribution within the ESCA framework, we conduct a systematic ablation study comparing SGCLIP against (a) direct scene graph generation using MLLMs and (b) alternative grounding methods such as Grounding DINO (GD). We also provide qualitative case studies & failure analysis on the above models, illustrating with examples how ESCA boosts perception and embodied planning.

(a) SGCLIP vs. Direct MLLM Scene Graph Generation

We evaluate against CoT-SG, a direct baseline where scene graphs are generated through a Chain-of-Thought MLLM pipeline. This approach prompts the MLLM to produce structured scene graph representations directly, without grounding.

EB-Navigation Results: ESCA consistently outperforms CoT-SG across all tested models. The Chain-of-Thought strategy actually degrades performance, particularly on long-horizon tasks where target objects are not initially visible. CoT-SG often fails to detect target objects after performing a few rotation actions to scan its surroundings, and it exhibits more hallucinations than the raw MLLM.

EB-Manipulation Results: While CoT-SG shows modest improvements over raw models, ESCA maintains superior performance across all tested MLLMs, demonstrating the effectiveness of structured grounding over direct generation approaches.

(b) SGCLIP vs. Grounding DINO

Results in Figure 5 demonstrate that the full ESCA pipeline significantly outperforms the MLLM + Grounding Dino (GD) baseline across all tasks. In that experiment, we ablate the object identification component by comparing two variants: one using Grounding DINO (GD) alone, and the other using the full ESCA pipeline, which integrates GD for object identification and SGCLIP for scene graph generation. This performance gain indicates that SGCLIP's structured, probabilistic scene graph predictions provide more effective grounding for embodied planning compared to GD’s object detection alone.

As detailed in Appendix C.1 and C.2, both configurations share the same concept extraction interface and prompting strategy; the only difference lies in the transfer protocol and grounding mechanism.

(c) Qualitative Analysis of ESCA v/s raw MLLM v/s MLLM + GD

We demonstrate the superior scene graph representation of ESCA via case studies and error decomposition analysis, which illustrate that ESCA achieves SOTA performance on the benchmarks by generating higher-quality scene graphs & reducing perception error, particularly in identifying hard-to-see objects and reducing hallucinations.

Case Studies on EB-Navigation & EB-Manipulation: We kindly refer reviewers to Figures 14 and 16 in the supplementary material, Appendix C, for a detailed qualitative analysis on EB-Navigation & EB-Manipulation. For instance, Figure 14 shows scene graphs generated using raw MLLM (gpt4o), MLLM+GD, and ESCA. Figure 14’s task is “navigating to a pillow”, where the raw MLLM doesn’t recognize any pillow, MLLM+GD misclassifies a shadow as the pillow, while ESCA correctly recognizes the pillow, and navigates to it.

ESCA Reduces Perception Error: We highlight the error decomposition analysis of GPT-4o with and without ESCA in Figure 17, which shows that ESCA significantly reduces perception errors among all errors, from 40% to 24%, highlighting its effectiveness in improving grounded visual understanding.

Error Analysis of Scene Graphs Generated via CoT-SG. In addition to figure 17, we conducted an error analysis by sampling 10 failure cases from EB-Navigation tasks, where scene graphs were generated using GPT-4o with a Chain-of-Thought (CoT-SG) approach. The failure modes were categorized as follows:

• Failure Type 1: Failure to detect the target object (6/10 cases)
• Failure Type 2: Hallucination of non-existent objects (3/10 cases)
• Failure Type 3: Incorrect object identification (e.g., misclassifying a red object as lettuce) (1/10 case)

All of these failure types stem from perception errors, whereas in our ESCA framework, only 24% of failures are attributed to perception issues, as shown in Figure 17. The discrepancy highlights a key limitation: despite leveraging a powerful MLLM, the CoT-SG method lacks sufficient grounding to reliably detect objects present in the visual scene. Further, as more entities are described without corresponding visual grounding, the planning context becomes diluted. This often leads to suboptimal action sequences and ultimately, task failure.

We sincerely thank the reviewer for their insightful feedback and take this opportunity to further clarify the discussion on compositional limitations.

Addressing Limitations of CLIP-based Models
We agree that vanilla CLIP and its variants often underperform on fine-grained compositional and spatial reasoning tasks (e.g., distinguishing “A is to the left of B” vs. “B is to the left of A”). This is precisely why our training approach for SGCLIP was designed to mitigate such weaknesses. Following LASER [1], we incorporate positional encoding during training via distinct colored masks marking subject and object entities. This positional encoding guides the model toward precise subject–object localization, mitigating spatial ambiguity.

As shown in appendix table 2, and our two extra evaluations, ESCA shows consistent improvements in downstream embodied-agent tasks that demand spatial reasoning:

These results collectively show that while CLIP alone struggles with complex spatial and compositional relationships, our targeted training strategy, combined with positional encoding and dataset design, substantially enhances its performance in embodied-agent settings requiring such type of reasoning. Thus, SGCLIP is not a vanilla CLIP, but designed specifically for relational grounding.

[1] Huang, Jiani, et al. “Laser: A neuro-symbolic framework for learning spatial-temporal scene graphs with weak supervision,” in The Thirteenth International Conference on Learning Representations (ICLR), 2025.

szsY-Q1. "Open-Domain" Capability in Practice: The candidate sets for attributes and relations seem to be pre-defined, and concept extraction relies on environment-specific textual feedback.

We respectfully clarify that ESCA's attributes and relations are generated dynamically on-the-fly, not pre-defined. EmbodiedBench itself is an open-domain SGG benchmark, and our evaluation demonstrates this capability through comparison with other open-domain methods (Grounding DINO and CoT-SG). For a comprehensive analysis of “open domain” capability please refer to our detailed response in SqXB-Q2.

szsY-Q2. Deeper Result Analysis and Case Studies: Can you provide a deeper analysis to explain when and why ESCA performs better?

The impact of ESCA varies depending on the visual demands and action structure of the task. In general, tasks that involve more detailed spatial reasoning or complex object interactions tend to benefit more.

Task characteristics

We illustrate this by further conducting experiments on two high-level embodied planning benchmarks: EB-Habitat and EB-Alfred. ESCA showed the smallest performance gain on EB-Alfred, which we attribute to its abstract action space and lower reliance on fine-grained visual grounding.

EB-Habitat performance (NEW)

EB-Alfred performance (NEW)

szsY-Q3. Can you include qualitative examples or case studies from the manipulation task to better illustrate how ESCA operates in the manipulation task?

For a detailed qualitative analysis, we kindly refer reviewers to the full paper in the supplementary material, Appendix C, which provides fine-grained examples of intermediate scene graphs generated using different methods, including raw MLLM, Grounding DINO, and ESCA (see Figures 14 and 16). These examples demonstrate that ESCA improves scene graph accuracy in both EB-Navigation and EB-Manipulation environments.

Additionally, in Figure 17, we highlight the error decomposition analysis of GPT-4o with and without ESCA, on a manual inspection of 50 EB tasks. ESCA significantly reduces perception errors, from 40% to 24%, highlighting its effectiveness in improving grounded visual understanding.

szsY-Q4. Unclear Contribution of SGClip… A critical baseline is absent: using a powerful MLLM to directly generate the scene graph zero-shot... Comparing (ESCA) against this simpler, distillation-free approach is essential to justify the complexity and necessity.

We appreciate the reviewer's valuable feedback and we evaluate ESCA against CoT-SG, a direct baseline where scene graphs are generated through a Chain-of-Thought MLLM pipeline. This approach prompts the MLLM to produce structured scene graph representations directly, without grounding. Further studies are detailed in kXSe-Q1.

EB-Navigation Results: ESCA consistently outperforms CoT-SG across all tested models. The Chain-of-Thought strategy actually degrades performance, particularly on long-horizon tasks where target objects are not initially visible. CoT-SG often fails to detect target objects after performing a few rotation actions to scan its surroundings, and it exhibits more hallucinations than the raw MLLM.

EB-Manipulation Results: While CoT-SG shows modest improvements over raw models, ESCA maintains superior performance across all tested MLLMs, demonstrating the effectiveness of structured grounding over direct generation approaches.

szsY-Q5. Error Decomposition: Can you clarify how error types are defined and disambiguated in your analysis, especially given that failures may be compounded and minor errors may occur even in successful trajectories?

We appreciate the reviewer’s concern and would like to clarify that formal definitions of the error categories are provided in Footnote 1 on Page 8 of the paper, which is also consistent with the paper:

“The three top-level error types are Perception, Reasoning, and Planning. The second-level categories include Hallucination, Wrong Recognition, Spatial Understanding, Spatial Reasoning, Reflection Error, Inaccurate Action, and Collision. For clarity, acronyms are used in the figure to label these error types.”

While eventual failures may involve compounded errors, we classify each failure case based on the first identifiable error in the execution trace to avoid ambiguity. Specifically, errors typically follow a cascade, where early perception or reasoning mistakes lead to downstream planning failures. For instance, incorrect spatial understanding may lead to collisions, and hallucinations can result in inaccurate actions.

As for errors that occur during successful trajectories, they are often corrected by subsequent planning steps. For example, the model may initially fail to find the target object due to distance, but successfully locates it after exploring further. Given that these recoverable deviations do not lead to task failure, we focus our error analysis on unrecoverable failures only.

szsY-Q6. Inadequate Engagement with Relevant Literature: LLM-Planner, MART, and P-RAG.

We thank the reviewer for providing these highly relevant literature and we will include the following discussion in the related works section. While these methods and ESCA both improve embodied agents, they target distinct pipeline components and are complementary: LLM-Planner focuses on hierarchical plan generation with ALFRED-specific grounding, while ESCA provides general-purpose, open-vocabulary grounding via scene graphs across diverse tasks. MART enhances planning through retrieval of past successful trajectories, P-RAG introduces progressive retrieval-augmented planning, while ESCA enhances the perception module with structured visual grounding.

SqXB-Q1. Novelty: the only difference is introducing scene graphs, which is straightforward and cannot capture the outstanding part … the proposed SGClip is only a fine-tuned CLIP with the proposed dataset.

We clarify that the introduction of scene graph for the embodied agents is non-trivial, leading to our novel contributions of (a) selective grounding, (b) design of transfer protocol, and (c) the fine-tuned model SGClip and its training dataset ESCA-Video-87K which can realize the prior goals.

Selective Grounding. Rather than injecting full scene graphs indiscriminately, which we show in the COT-SG experiment, leading to degraded performance, we focus on presenting only the most relevant information to guide the MLLM in planning. Specifically, we use the MLLM to identify the objects, attributes, and relationships that are most pertinent to the instruction, and then determine which key objects are essential for task completion with symbolic execution. For navigation, this is the object the agent must locate; for manipulation, it is the precise (x, y, z) position mapping to the desired object feature.

Design of Transfer Protocol. We define the transfer protocol as the process of leveraging concepts extracted by the MLLM to construct prompts enriched with scene-specific information. The protocol filters and aggregates relevant scene elements tailored to the agent’s goal. For this purpose, it performs probabilistic reasoning over object names, spatial relationships, and attributes to identify the top-k most likely target objects. This fine-grained inference is enabled by our probabilistic scene graph design, which encodes SGCLIP predicted likelihood. This mechanism, detailed in Appendix C, is a technical contribution that goes beyond simply plugging in a scene graph module.

SGClip and ESCA-Video-87K. Regarding the novelty of SGCLIP, we emphasize that the contribution lies not in architectural innovation but in the data collection and training strategy. SGCLIP is trained on a large-scale, weakly-supervised video dataset generated entirely via multimodal LLM prompting, described in Appendix A. Our approach overcomes the scarcity of manually labeled scene graph annotations in video by enabling scalable, diverse, and compositional weak supervision at scale, requiring only raw video as input. SGCLIP thus represents a practical and generalizable solution for scene graph grounding in open-world tasks.

To the best of our knowledge, ESCA-Video-87K is the largest automatically labeled video scene graph dataset featuring rich object semantics, relations, and actions. Most other annotated SG datasets are much smaller. For comparison:

• Ego4D-EASG includes 221 long videos annotated with 407 object classes, 219 verbs (action types), and 16 spatial/prepositional relation types.
• OpenPVSG contains 400 videos, covering 126 object classes and 57 relation types.
• BDD-100K, the largest densely labeled video dataset, focuses solely on driving scenarios. While it offers 100K videos with object trajectories, it includes only 10 object classes and lacks relation and action annotations.

In contrast, ESCA-Video-87K features automatic annotations of diverse object names, inter-object relations, and object actions across 87K videos. Human-labeled fine-grained datasets like OpenPVSG are limited in scale due to the complexity of dense annotation, highlighting the scalability advantage of our weak-supervision pipeline.

SqXB-Q2. How does ESCA compare against open-domain SGG methods? and on open-set SGG datasets?

Embodied Bench is an open-domain SGG benchmark

A key strength of the ESCA framework is its ability to handle open-domain natural language instructions, which often contain free-form and compositional descriptions not present in the simulated environments. For example, a banana might be described as a "curved yellow fruit" and a loaf of bread as a "freshly baked baguette."

This open-domain capability is clearly demonstrated in our quantitative analysis of the EB-Navigation dataset. Using GPT-4o as the concept extractor, our pipeline generates an average of 5.57 free-form keywords per instruction, resulting in 84 unique target object descriptions across the dataset. This diversity significantly exceeds the 26 canonical object names defined in the ground-truth vocabulary. Beyond target objects, the process also produces 199 related objects, 296 relationships, and 117 attributes as keywords. This expanded vocabulary demonstrates how SGCLIP captures rich, unconstrained concept expressions that arise from both natural language variability in instructions (e.g., "curved yellow fruit" instead of "banana") and observations derived from the visual scene.

To further demonstrate generalizability across open-domain SGG datasets, we report performance results on EB-Alfred and EB-Habitat as follows:

EB-Habitat performance (NEW)

EB-Alfred performance (NEW)

Grounding-dino and CoT-SG are two open-domain SGG methods

To empirically validate ESCA's open-domain capability, we compare against two baseline approaches. First, as reported in our main paper, we evaluate against a baseline that omits SGCLIP and relies solely on Grounding DINO for open-world object grounding (MLLM + GD). We further evaluate against CoT-SG, a Chain-of-Thought approach that prompts MLLMs to generate structured scene graph representations directly without any visual grounding. This method represents a direct generation baseline where the model produces scene graphs purely through reasoning rather than structured visual-textual alignment.

EB-Navigation Results: ESCA consistently outperforms CoT-SG across all tested models. The Chain-of-Thought strategy actually degrades performance, particularly on long-horizon tasks where target objects are not initially visible. CoT-SG often fails to detect target objects after performing a few rotation actions to scan its surroundings, and it exhibits more hallucinations than the raw MLLM.

EB-Manipulation Results: While CoT-SG shows modest improvements over raw models, ESCA maintains superior performance across all tested MLLMs, demonstrating the effectiveness of structured grounding over direct generation approaches.

SqXB-Q3. how to integrate SGG with the agent and how spatial and temporal relationships are integrated into the agent.

We illustrate how the ESCA framework integrates scene graphs through the transfer protocol for EB-navigation, as described in Appendix C. We will provide more information in a revised version.

Concrete Example - EB-Navigation Task: This protocol combines the object name likelihood, attribute likelihood, and spatial relation likelihood to compute the probability of each object being the target. Consider the instruction "navigate to the loaf of bread that is on the counter”, and three objects A, B & C. Our protocol processes this as follows:

• Object Recognition: A and B recognized as "loaf of bread" (0.8 likelihood) and "counter" (0.2). C recognized as "counter" (0.7) and "bread" (0.3).
• Spatial Relations: SGCLIP predicts A is "above" C (0.9), B is "above" C (0.1)
• Probabilistic Aggregation: We compute the aggregated likelihood using our three-factor formula:Target Likelihood = Object Confidence × Spatial Relation Confidence × Landmark Confidence. The target likelihood for A: 0.8 (bread confidence) × 0.9 (spatial relation) × 0.7 (counter confidence) = 0.504, likewise for B = 0.8 × 0.1 × 0.7 = 0.056. This calculation enables the agent to prioritize objects that not only match in name but also satisfy the instructed spatial context.

From Scene Graph to Action: We pass the top-k targets downstream. Assume k = 1, Object A becomes the primary navigation target, with its bounding box coordinates and confidence score (0.504) passed to the motion planner. This probabilistic grounding enables robust decision-making under perceptual uncertainty. Though evaluated only on EB in static environments, our framework could generalize to dynamic environments with temporal relations. For "watch the person cook pasta", scene graphs would track action sequences with temporal edges, aggregating probabilities across time.

Dear Reviewer SqXB,

We appreciate the time and effort you dedicated to your initial review and hope you have had a chance to read our rebuttal and follow-up clarifications. We hope these have addressed the concerns you raised.

With less than 12 hours remaining in the discussion period, we would be glad to clarify any remaining questions you may have. Your feedback at this stage would be greatly valued.

Thank you again for your thoughtful review.

Best regards, The Authors