S3E: Semantic Symbolic State Estimation With Vision-Language Foundation Models

We thank the reviewer for their feedback. We respectfully disagree with the characterization of our work as a straightforward application of VLMs and would like to clarify several points:

Our contribution extends beyond simply applying VLMs to robotics. S3E presents the first general-purpose framework for translating arbitrary symbolic predicates into natural language queries that can be evaluated by VLMs. While the demonstration scenarios may appear simple, the method itself is not limited to these scenarios - it can work with any domain that can be represented using symbolic predicates. This includes complex predicates describing non-rigid object states, temporal relationships, or abstract properties, provided they can be expressed in the domain description language.

We acknowledge that we should better situate our work within the context of existing VLM applications in robotics, including the papers mentioned by the reviewer (please see the revised version). However, our approach differs from these works in an important way: rather than using VLMs for specific robotic tasks or failure detection, we provide a general framework for bridging symbolic task planning with visual perception through systematic predicate translation and evaluation. This enables S3E to work with any planning system that uses symbolic state representations.

Our offline evaluation focused on establishing the reliability of the state estimation component in isolation. While integration with complete robotic systems is important future work, understanding the capabilities and limitations of the core state estimation mechanism provides valuable insights for such integration.

Regarding the specific concerns:

Question: how to extend the state estimation to more general scenarios, e.g., non-rigid object, where the object state is beyond the location and spatial relationship? Response: The apparent simplicity of our test scenarios was chosen for clear demonstration and validation of the core method. Nothing in our approach inherently limits it to rigid objects or spatial relationships. Any predicate that can be expressed symbolically (e.g., "is_folded(towel)", "contains_liquid(cup)", "is_locked(door)") can be translated and evaluated using our framework, as long as it is visually discernible.

Question: What are some existing works in robotics using VLM as state estimation? Summary their approach and limitations and see how the proposed work can address them. Response: We agree that expanding the related work discussion and demonstrating S3E in more complex scenarios would strengthen the paper. However, we believe our contribution provides a valuable foundation for developing more sophisticated symbolic state estimators that can leverage the continuing advances in vision-language models.

We hope that our responses have shifted the reviewers perspective on our paper and have encouraged a reevaluation through of the revised submission.

Thank you for your feedback and for recognizing the importance of our work in semantic state estimation. We address your main concerns below:

Concern: The proposed approach of determining the validity of predicates with VQA addresses only a part of the problem of estimating the symbolic state. The proposed approach is restricted to simple predicates in small clutter free settings. Scalability to scenarios with complex object arrangements and to higher-order predicates is limited.
Response: We agree that our current evaluation focuses on relatively simple predicates in clutter-free environments. Though one might argue that photorealistic blocksworld with 10 objects can become cluttered at times, using a clean and clear environment was a deliberate choice for the initial demonstration of S3E's capabilities. Our results show that, there are several issues to overcome before we can tackle more complex environments. Specifically, the presented uncertainties and limitations that come bundled with using VLMs directly as state estimators must be addressed. Some of these issues will improve as general VLM quality improves, and we have suggested the rest as future work.

Concern: The problem of eliciting uncertainty from LLMs has received attention in the past. The authors are requested to highlight the technical gap they are filling in this regard. Please refer to works such as Robots That Ask For Help and Can LLMs Express Their Uncertainties.
Response: We appreciate you pointing out these relevant works on uncertainty elicitation in LLMs. While these works advance the field, they primarily focus on different problem settings. They aim to calibrate the confidence scores produced by LLMs to better reflect their actual accuracy while other approaches focus on detecting and mitigating overconfidence in LLM responses, particularly in conversational settings. The technical gap we address is leveraging these advancements for symbolic state estimation within a task planning context. Our work bridges the domains of symbolic planning and VLM-based uncertainty quantification, exploring how to utilize uncertainty information to improve state estimation and ultimately task success. We will clarify this distinction in the revised paper and include a more comprehensive discussion of related work on LLM uncertainty.

Concern: The problem of inferring semantic properties of the robot's environment has also been addressed in works that extract a scene graph from raw sensor data. Please see Maggio et al. for an example. I request authors to situate their work along side such efforts.
Response: We agree that scene graph extraction techniques offer a valuable approach for understanding the robot's environment. However, our work differs in its focus and objective. Scene graphs aim to represent the relationships between objects in a scene comprehensively. They are often used for tasks like image captioning or visual question answering. S3E, on the other hand, focuses specifically on estimating the truth values of symbolic predicates relevant to a given task. We use VLMs as a tool to efficiently assess these predicates without needing to construct a full scene graph, which can be computationally expensive. In the revised paper, we clarify this distinction and discuss the complementary nature of S3E and scene graph extraction methods. We will also suggest ways of integrating these approaches to further improve the robustness and accuracy of symbolic state estimation.

We appreciate the reviewer's thorough analysis and constructive feedback. We address each weakness and the questions below:

Concern: While S3E is presented as a zero-shot solution, it still requires significant assumptions about the environment, including full observability, visually distinguishable objects, and unambiguous domain descriptions - constraints that may not hold in many real-world applications.
Resonse: We agree that the assumptions of full observability, visual distinguishability, and unambiguous domain descriptions represent significant constraints. These challenges are fundamental to state estimation in general, not just our approach. In our paper, we explicitly analyze how these assumptions affect S3E's performance (as shown in Tables 5-6 and discussed in Section 6.2), demonstrating both the capabilities and current limitations of our method. This transparency about performance limitations under different conditions helps establish a baseline for future work in this area.

Concern: he experiments, while showing promising results, are limited to relatively controlled environments and simple manipulation tasks. The evaluation could be strengthened by exploring more challenging scenarios such as partial observability conditions, dynamic environments, or multi-agent interactions. This would help better understand the method's full potential.
Response: We acknowledge that exploring more challenging scenarios would provide valuable insights. Our work represents a first step in demonstrating the viability of using VLMs for state estimation. Note, however, that while results show promise, these controlled environments have already shed light on the limitations of VLMs as state estimators, e.g., out-of-distribution observations, setting the foundation for important future work.

Concern: The paper's handling of uncertainty, while acknowledged, could be more rigorous - the proposed mitigation strategies of environment design and natural language instruction essentially sidestep rather than solve the underlying uncertainty challenges. Additionally, the method's heavy dependence on the quality of the vision-language model's training data suggests potential brittleness when encountering out-of-distribution scenarios, as evidenced by the poor performance with certain objects in the simulated environment.
Response: The reviewer makes a valid point about our handling of uncertainty. Our goal was to first identify and analyze these uncertainties rather than solve them completely. While our mitigation strategies may appear to sidestep the challenges, they provide practical solutions for immediate deployment through an iterative process between the user and the VLM while highlighting areas needing further research. Regarding the dependence on VLM training data quality, we view this as both a limitation and an opportunity - as VLM technology improves, our method automatically benefits from these advances without requiring modifications to the core approach.

Regarding the specific questions:

Question: Have you explored any cases where the domain descriptions contain partial or ambiguous information? What were the results?
Response: Yes, we did test the case of ambiguous information in the domain description. Specifically, the ambiguity presented in Figure 4 arises from ambiguity in the meaning of "is gripping". This is addressed using the "Pose" modification. Furthermore, the photorealistic blocksworld domain presented in Appendix C uses a variation of the "on" predicate, leaving room for interpretation of what "block A is on block B" means.

Question: Were there any failure cases in the controlled environments that could provide insights into the system's limitations?
Response: Yes, we explicitly documented failure cases in our paper. For example, in Section 6.2 and Figure 5, we discuss how certain objects (like the bread) were consistently misrecognized even with natural language instruction, highlighting limitations in the VLM's ability to handle visually unrealistic objects. Additionally, Figure 4 shows an example where object gripping state was ambiguous, demonstrating how uncertainties can arise even in controlled environments.

We hope that our responses satisfy the reviewers concerns and curiosities about our approach.

We thank the reviewer for their thoughtful feedback. Below we address each concern:

Concern: It's unclear how the system automatically determines that there is uncertainty in the prediction, which then needs to be addressed by asking for clarifying instructions or better camera views.
Response: We appreciate this observation about uncertainty detection. To clarify, our paper's scope was to identify and analyze two types of uncertainties that emerge when using VLMs for state estimation, rather than to propose an automated system for detecting and resolving these uncertainties. We quantified these uncertainties through our experimental results and demonstrated their impact on state estimation performance. The development of automated uncertainty detection and resolution methods represents an important direction for future work. Currently, the calibration of the system is an iterative process between the user and the VLM. This is to be expected anyway because specific usage often leads to out-of-distribution examples. Should we replace S3E with a human, we would still need to explain these examples and make sure they understand before we develop trust.

Concern: The predicates this paper tests are limited to determining whether an object is on a specific table region or whether the object is in hand or in the air. These predicates probably can be determined using very simple heuristics based on gripper state or detected object bounding boxes. It's hard to assess whether the method can reliably classify more diverse predicates for real-world robotic tasks. I suggest the authors to look at other photo-realistic simulation environments that provide annotated object states.
Response: While the real-robot demonstration used relatively simple predicates for clarity, our evaluation included more complex scenarios through our simulated environments. In particular, our evaluation of the photorealistic blocksworld domain (Appendix C) tested the system with diverse object attributes including size, color, material, and shape, along with spatial relationships between objects. The success of S3E in handling these more complex predicates suggests that the method can scale to more sophisticated real-world scenarios, limited primarily by the underlying VLM's generalization capabilities.

Concern: Even though the authors motivate the paper for task planning, there is no evaluation of the method integrated with a task planning framework. It's unclear how the improvement in classification performance translates to the improvement in overall task success.
Response: We acknowledge that evaluating S3E's impact on overall task success would provide valuable insights. However, such evaluations insert bias through the choice of planner, the strategy for sampling states, and the stochasticity of irreversible action failure. Our work focused specifically on improving the state estimation module, which is one critical component of a complete robotic system. While we demonstrate good state estimation accuracy, evaluating how this accuracy translates to overall task success when integrated with planning and control modules represents important future work.

Concern: In line 176, the authors mention that the assumption is unrealistic. However, I think this assumption is valid if the action sequences are correct (e.g., if the robots are being teleoperated by a human user)
Response: We respectfully disagree about the assumption in line 176. While action sequences may be reliable under human teleoperation, our work targets autonomous operation in unstructured real-world environments where action outcomes can be uncertain due to various factors (e.g., environmental dynamics, sensor noise, control errors). In such scenarios, reliable state estimation and replanning capabilities are crucial for robust task execution.

Concern: In general, the writing of the paper can also be improved. I found the discussion on uncertainty and experiment section a little confusing.
Response: We appreciate the feedback about the clarity of our uncertainty discussion and experimental section. In the revised version, we improved the organization and presentation of these sections to make our methodology and results more accessible. Specifically, we restructured Sections 5 to more clearly delineate between the two types of uncertainties and their implications.

We believe these clarifications address the reviewer's main concerns while acknowledging limitations and future work directions.