ActiveVOO: Value of Observation Guided Active Knowledge Acquisition for Open-World Embodied Lifted Regression Planning

Rebuttal

[Weakness 1, Question 3]: LLM/VLM Consistency and Sensitivity

Reviewer comment:

“The VOI calculation highly relies on LLM and VLMs, which we know face the problems of inconsistency and prompt-dependency, especially when the scene is complicated, or the instruction is unclear.”

“Is there any measure taken to overcome LLM’s problems of inconsistency and prompt-dependency, and/or help correct possible bias?”

Response:

We appreciate the reviewer’s concern about the reliability of LLM and VLM outputs in complex scenes and under prompt variability. Below, we describe the specific design choices we implemented to mitigate inconsistency, reduce prompt sensitivity, and ensure robustness in ActiveVOI’s commonsense reasoning pipeline.

Regarding LLM/VLM Prompt Sensitivity and Inconsistency: To mitigate prompt sensitivity and reduce reliance on domain-specific prompt engineering, we adopt three key design choices.

• We kept the prompts minimal and generic without additional task-specific prompt tuning or templates (shown in Appendix D.1).
• We leverage test-time scaling techniques of self-consistency to improve reliability of probability estimates by aggregating LLM outputs 10 times, resulting in a more stable estimation of commonsense priors.
• ActiveVOI operates entirely in a zero-shot setting, avoiding prompt-induced bias and ensuring better generalization across unseen tasks.

On Handling Complicated Scene: We handle complex scenes by reducing the object search space through compact lifted object descriptions and by removing duplication of semantically equivalent objects. These strategies can reduce LLM/VLM inconsistencies by limiting queries to a small set of goal-relevant objects, rather than reasoning over all observed entities.

On Handling Unclear Instructions or Goals: Unclear instructions or ill-defined goals pose fundamentally different challenges in open-world planning that require separate methodologies and benchmarks. Prior work frames these problems through contingent planning [1] or preference elicitation [2]. While important, these issues are orthogonal to our focus on object-centric knowledge acquisition in open-world planning.

---

[Weakness 2, Questions 1, 2]: Motivation for ALFWorld Compared to Other Benchmarks

Reviewer comment:

“The experiment is conducted on ALFWorld exclusively, which lacks validation on generalizability.”

Reviewer questions:

“Will there be experiments with other embodied environments like Habitat and iGibson?”

“Will there be experiments under real-world settings?”

Response:

We thank the reviewer for raising the important question of generalizability. Below, we clarify our motivation for choosing ALFWorld over other embodied benchmarks and explain the current limitations of evaluating in real-world settings.

Habitat, iGibson Feasibility: While Habitat and iGibson are popular embodied AI benchmarks, they are primarily designed for object navigation and rearrangement tasks. In such settings, object relevance is trivial, as all necessary object knowledge is directly provided. As a result, these benchmarks are not suitable for evaluating open-world planning and active sensing methods, where agents must infer which objects are relevant under partial knowledge and semantic uncertainty.

Motivation for Using ALFWorld: We selected ALFWorld tasks because they have several key characteristics essential for our open-world active sensing evaluation requirements: (1) task-relevant objects are not directly given and require semantic inference; (2) the environment features high diversity in objects, scenes, and tasks, along with rich visual observations; (3) it is partially observable and can be configured without access to ground-truth object and relational information; and (4) the benchmark poses significant challenges for SOTA models (VLMs), even with extensive domain-specific fine-tuning (shown in Table 1).

On Real-World Evaluation: To the best of our knowledge, there is currently no real-world setup that can support the level of object diversity, action complexity, with object state changes required to test open-world planning and active knowledge acquisition. Building such a testbed would require significant engineering effort, which we view as outside the core contributions of our work which is our ActiveVOI framework based on formal decision-theoretic reasoning.

---

[Weakness 3]: Regarding Novelty of ActiveVOI

Reviewer comment:

“Some might have concerns about novelty, as notions of active knowledge acquisition and VOI are already established, and this paper just makes a combination in a new context.”

Response:

We would like to clarify that ActiveVOI is not merely a combination of existing approaches, but addresses an overlooked and fundamental challenge in open-world planning: (1) determining which objects are relevant for sensing, and (2) estimating the expected utility of sensing those objects. This problem is particularly difficult in open-world settings, where the agent operates under partial observability and lacks prior knowledge about object types, configurations, and affordances, while the set of potentially observable objects is unbounded. In contrast, most existing work on active perception assumes that the set of relevant objects is given. Thus, ActiveVOI fills this important but often overlooked gap in existing active sensing approaches [3].

---

References

[1] Dearden, Richard, et al. "Incremental contingency planning." 13th International Conference on Automated Planning and Scheduling. 2003.

[2] Ren, Allen Z., et al. "Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners." Conference on Robot Learning. PMLR, 2023.

[3] Crespo, Jonathan, et al. "Semantic information for robot navigation: A survey." Applied Sciences 10.2 (2020): 497

Dear reviewer:

We sincerely appreciate your insightful review. With the reviewer-author discussion period ending soon, please let us know if we can further clarify our responses to any of your concerns.

best,

The Authors

We sincerely thank the reviewer for the detailed and thoughtful feedback. Below, we address the concerns and questions raised.

[Weakness 1] Regarding Presentation and Clarity

Reviewer comment:

"The writing and the figures of the manuscript requires further improvement. For example, Fig. 1 shows complex pipelines, while the differences of the two pipelines only mainly lines in the Active Knowledge Acquisition step."

Response: We thank the reviewer for their feedback on Figure 1. Our intention was to illustrate the difference between 1) passive and active knowledge acquisition and 2) grounded progression and lifted regression. We can certainly revise the figure to improve clarity; to help us improve it, please let us know which of the following distinctions depicted in the figure seems unclear:

• Difference between planning strategy (Steps 4,6 Left, Steps 8,9 Right): Grounded progression replans if actions fail or when knowledge is insufficient for plan generation. In contrast, lifted regression generates a compact set of feasible lifted plans up front.
• Difference in Knowledge Base Size (KB blocks): Passive sensing requires the agent to store all object and relational information, resulting in a much larger KB than active sensing, which stores only task-relevant information.
• Active Perception Pipeline (Step 5 Left): The active method uses lifted subgoals and the agent’s knowledge base to selectively acquire task-relevant information from observations.

[Weakness 2] "Open-World" Clarification

Reviewer comment:

"ALFWorld is not an Open-World Embodied setting ... similar works tackling the active reasoning problems in environments like Minecraft or Crafter ... The author should consider testing their algrithims in more scenarios."

Response:

Open-World Clarification: Our use of the term open-world follows the Open World definition from a model-theoretic perspective, where the agent operates with “incomplete knowledge of the model” [1], i.e., incomplete information about object types, configurations, and relational properties.

Minecraft-based benchmark: Most Minecraft-based benchmarks are considered open-world due to their expansive environments and open-ended task structures. However, they are highly domain-specific and offer limited generalizability, as they focus on high-level actions like “mining” and “crafting” over a fixed set of predefined objects and materials unique to the Minecraft domain [2]. Minecraft recipes are also widely available in LLM training data, removing the need to infer new object types, relations, or affordances, which are core challenges in open-world planning and active sensing.

ALFWorld benchmark: We configure ALFWorld to align with our definition of open-world settings, where agents operate under partial observability using only egocentric RGB input, without access to ground-truth object types, attributes, or relational information. This makes it well-suited for evaluating open-world active sensing. A comparison with other embodied AI domains is provided in the next section (see [Weakness 3, Question 1]).

[Weakness 3, Question 1] Novelty with Comparison of Existing Approaches

Reviewer comment:

"Active perception and information-driven exploration are long-standing topics in AI and robotics ... LLMs (and even VLMs) for goal decomposition, subgoal extraction ... not sure whether the integration of VOI with LLMs for active object-centric search significantly novel..."

"Are there concrete differences between using LLMs for task-relevant object prediction/information gain and previous LLM- or information-theoretic approaches to active perception?"

Response:

ActiveVOI introduces a novel formulation of active sensing by explicitly reasoning over goal-driven object relevance and utility in the open-world setting. Here, we highlight the novelty of ActiveVOI with a comparison with existing active sensing work.

Novelty of ActiveVOI: Most existing work on active sensing focuses on object or semantic navigation, where the goal is to (1) locate a known object, and (2) to maximize information gain [6]. In contrast, ActiveVOI tackles important but underexplored problems in active sensing: (1) determining what type of object the agent should sense, and (2) assessing how much utility (not just information) is gained from sensing an object. These two components constitute novel contributions to the field.

Comparison with LLM/VLM subgoal decomposition: LLM/VLM subgoal decomposition suffers from several known issues: (1) lacking any correctness guarantee of generated plans, (2) requiring few short examples, (3) poor alignment with the environment and agent’s actions, (4) performance degradation over long-horizon. In contrast, ActiveVOI is based on a formal decision-theoretic framework of lifted regression that: (1) guarantees correctness of the plan, (2) can plan in a zero-shot setting, (3) is based on the agent’s action model, and (4) is capable of tracking long-horizon reasoning.

Comparison with Information-theoretical approaches: Traditional information-theoretical approaches often define utility as generic information gain (such as entropy reduction) without directly connecting this gain to the agent’s goal [7]. In contrast, ActiveVOI is task-centric, where the utility is defined with respect to goal achievement. This distinction is crucial in goal-oriented tasks, where not all information is equally valuable, and task relevance must guide sensing.

[Question 2] Real-World Feasibility and ALFWorld Choice

Reviewer comment:

"Do the authors anticipate that this approach would transfer smoothly from simulated benchmarks (ALFWorld) to real-world robot settings? If not, why they choosing the ALFWorld simulator rather than other Open-World Embodied tasks?"

Response:

We want to clarify how the method can support real-world deployment as a high-level reasoning module, and justify our choice of ALFWorld over other embodied AI benchmarks in the context of open-world active sensing.

Real-world transfer: The primary obstacles for the transfer of ActiveVOI lie in integrating low-level control and hardware-specific tuning. ActiveVOI, as a reasoning and planning module, should be readily adaptable to real-world applications due to (1) being zero-shot without the need for fine-tuning, (2) avoiding the combinatorial explosion of objects by using a compact lifted-representation of relevant objects, which increase scalability.

Suitable Embodied AI benchmarks: We considered several other benchmarks based on Habitat, iGibson, and VirtualHome [3, 4, 5], but found them insufficient for evaluating open-world planning and active sensing. Tasks designed on these benchmarks are focused on 1) object navigation, 2) object rearrangement, 3) instruction following where all relevant objects are assumed known (given in the instructions), making semantic reasoning and object-centric active sensing irrelevant.

ALFWorld Rationale: We selected ALFWorld tasks because they have several key characteristics essential for our open-world active sensing evaluation requirements: (1) task-relevant objects are not directly given and require semantic inference; (2) the environment features high diversity in objects, scenes, and tasks, along with rich visual observations; (3) it is partially observable and can be configured without access to ground-truth object and relational information; and (4) the benchmark poses significant challenges for SOTA models (VLMs), even with extensive domain-specific fine-tuning (show in Table 1).

[Question 3] Could a Classical Planner Succeed with Full KB? Is LLM Commonsense Really Needed?

Reviewer comment:

"One of the claims ... If a traditional symbolic/planning agent ... could it not also generate an optimal plan ... is the reliance on LLM commonsense truly necessary ... or is it ... for incomplete or noisy object knowledge?"

Response:

LLMs and VLMs are used not only for knowledge completion of the agent’s KB, but also to estimate the non-deterministic outcomes of sensing actions. As a result, a plan generated by a standard PDDL model, even with a fully specified and correct KB, may not be optimal, as it typically assumes deterministic outcomes. ActiveVOI uses LLM-derived commonsense priors to approximate the transition probabilities of sensing actions (the likelihood of encountering a relevant object (e.g., a heat source) in a given environment (e.g., a kitchen). This allows the agent to compute the expected utility of sensing different objects under our Value of Information framework. In summary, LLMs and VLMs are essential in our framework, both for completing missing knowledge and for modelling the non-determinism of sensing actions.

References

[1] Finzi, Alberto, Fiora Pirri, and Raymond Reiter. "Open world planning in the situation calculus." AAAI/IAAI. 2000.

[2] Wang, Zihao, et al. "Describe, explain, plan and select: interactive planning with llms enables open-world multi-task agents." Advances in Neural Information Processing Systems (2023)

[3] Puig, Xavier, et al. "Virtualhome: Simulating household activities via programs." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[4] Savva, Manolis, et al. "Habitat: A platform for embodied ai research." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[5] Shen, Bokui, et al. "igibson 1.0: A simulation environment for interactive tasks in large realistic scenes." International Conference on Intelligent Robots and Systems (IROS) 2021.

[6] Crespo, Jonathan, et al. "Semantic information for robot navigation: A survey." Applied Sciences 2020

[7] Stachniss, Cyrill, Giorgio Grisetti, and Wolfram Burgard. "Information gain-based exploration using rao-blackwellized particle filters." Robotics: Science and systems 2005.

Rebuttal

[Weakness 1] Benchmark Choice

Reviewer comment:

"The experiments are primarily conducted on the ALFWorld benchmark, which is a well-regarded testing environment, but additional evaluations across other benchmark tasks, such as Alfred or VirtualHome would help to confirm whether the observed performance improvements are consistent across different types of environments and tasks. Expanding the evaluation scope would provide more robust evidence of the framework's general applicability."

Response:

We thank the reviewer for highlighting the importance of evaluating generalizability across diverse embodied benchmarks. However, we find most benchmarks inadequate for our evaluation purposes. Below, we explain our rationale for focusing on ALFWorld and clarify the limitations of alternative environments for evaluating open-world planning and object-centric knowledge acquisition.

ALFRED and VirtualHome: We considered both ALFRED and VirtualHome, but found them unsuitable for evaluating semantic open-world planning and object-centric active knowledge acquisition. Tasks in both environments assumes subgoals or step-by-step instructions are available to the agent which already contain all necessary objects required to achieve the goal. This makes object-centric active sensing unnecessary.

ALFWorld: We selected ALFWorld because it exhibits key characteristics essential for our evaluation:

1. Task-relevant objects are not directly specified and require semantic inference.
2. The environment features high diversity in objects, scenes, and tasks, with rich visual observations.
3. It is partially observable and can be configured without access to ground-truth object types or relations.
4. The benchmark remains challenging for state-of-the-art models (e.g., VLMs), even with extensive domain-specific fine-tuning (as shown in Table 1).

---

[Weakness 2, Question 2] Assumptions on Object Knowledge Independence

Reviewer comment:

"The framework makes several assumptions regarding the independence of object attributes and relational properties. In complex real-world scenarios with intricate object interactions, these assumptions may lead to inaccuracies in the VOI calculations, affecting the agent's decision-making process."

"How does the proposed VOI calculation handle dependencies between object attributes and relations in more complex environments, where the assumptions of independence may not hold? Would relaxing these assumptions improve performance?"

Response:

We agree that assumptions of independence must be handled carefully. While we discussed the motivation in Section 2.1.3, we provide further elaboration below that we will include in the revision.

1. Justification for the Independence Assumption

• Reduce sensing targets: By assuming independence between object attributes and relations, we enable object-level sensing, drastically reducing the number of sensing candidates. Without this assumption, the agent would have to reason over a joint space of all object-relation configurations, which is computationally intractable at scale.
• Enable reliable LLM reasoning: LLMs are more reliable at generating commonsense priors for concise object descriptions (e.g., "a knife is likely in the kitchen") than for complex relational queries (e.g., "is it likely to find a dirty and metal knife near a sponge with a dirty plate?"). As shown in the "w/o VOI and Object Partition" row in Table 2, reasoning over complex subgoals leads to a significant performance drop (−18%), emphasizing the value of object-level prioritization.

2. Robustness to Biased VOI Calculation While initial VOI estimates may be biased due to the independence assumption and model limitations, the agent's belief state is updated incrementally with new observations. VOI is re-computed over this updated KB, allowing recalibration and alignment with the actual environment over time. This dynamic adjustment mitigates the impact of initial bias.

---

[Weakness 3, Questions 1 & 3] Scalability and Cost

Reviewer comment:

"Could the authors provide further details on how the method might scale in real-world environments? Specifically, how might the approach be adapted to handle environments with significantly more objects or dynamic interactions between objects?"

"While the method is effective in small-scale environments, it remains unclear how well the system scales to much larger, more dynamic real-world environments with a high number of objects and complex interrelationships."

"Can the authors elaborate on potential strategies for reducing the computational cost of VOI calculations, especially when scaling the method to larger environments with many objects?"

Response:

Handling Environments with Many Objects: We thank the reviewer for raising the important issue of scalability, which is one of the core motivations behind ActiveVOI. Real-world environments may contain thousands of objects, yet only a small fraction is typically relevant for a given goal. ActiveVOI is expressly designed to exploit this observation:

• Lifted relational abstraction: Our planner uses lifted regression to perform goal-directed backward search, deriving relevant subgoal descriptions that filter out irrelevant objects.
• Object description consolidation: When multiple subgoals share object descriptions, we consolidate them to eliminate redundant sensing queries, further improving efficiency.

These strategies ensure that the agent’s sensing and reasoning complexity scales with the number of goal-relevant objects, not the total number of visible objects.

Reducing VOI Computation Cost: The above design choices greatly reduce VOI computational cost by narrowing the search space. Another major contributor to cost is LLM query overhead. ActiveVOI minimizes this cost through:

• Symbolic planner offloading: Planning is delegated to a symbolic planner, whose computation is negligible compared to LLMs.
• LLM token efficiency: ActiveVOI uses ~2,000 LLM tokens per task, compared to >50,000 in step-by-step LLM planning methods like ReAct.

Handling Dynamic Environments: We acknowledge that adapting to dynamic or probabilistic settings is an important future direction. However, doing so would require different methodologies and baselines (e.g., conformant or FOND planning [2,3]). These settings are orthogonal to our current focus, but we will include discussion of this in the final version.

---

References:

[1] Valmeekam, Karthik, et al. "On the planning abilities of large language models—a critical investigation." NeurIPS (2023).

[2] Hoffmann, Jörg, and Ronen I. Brafman. "Conformant planning via heuristic forward search: A new approach." Artificial Intelligence 170.6-7 (2006): 507-541.

[3] Muise, Christian, Sheila McIlraith, and Christopher Beck. "Improved non-deterministic planning by exploiting state relevance." ICAPS (2012).

Dear reviewer:

We sincerely appreciate your insightful review. With the reviewer-author discussion period ending soon, please let us know if we can further clarify our responses to any of your concerns.

best,

The Authors

We thank the reviewer for the detailed and insightful feedback. Below, we address the reviewer’s concerns and questions regarding LLM/VLM accuracy, failure recovery, and the potential for knowledge persistence and reuse.

[Weakness 1, Question 1] Reliance on LLM/VLM Accuracy and Failure Handling

Reviewer comments:

“The method assumes that GPT-4 (or similar models) provides reliable estimates for object relevance. As shown in ablations, performance drops with weaker LLMs (e.g., GPT-3.5). This dependence may limit robustness, especially when commonsense priors are uncertain or incorrect—an issue not deeply explored in the failure analysis.”

“LLM/VLM failure handling: In cases where the LLM or VLM provides incorrect or misleading priors (e.g., low belief for a relevant object), does ActiveVOI have any fallback or exploration mechanism? Could belief calibration, ensembling, or conservative VOI strategies improve robustness to noisy commonsense reasoning?”

Response:

Reliance on LLM/VLM accuracy: We appreciate the reviewer’s concern regarding the reliance on LLM/VLM estimates for object relevance. In fact, ActiveVOI is not statically bound to these initial priors. The agent maintains a knowledge base (KB) that is incrementally updated with each new observation, and VOI is re-computed over this updated belief state. As the agent gathers more information, its KB progressively aligns with the actual environment. This incremental recalibration of VOI helps mitigate the impact of initial commonsense errors.

Fallback mechanisms and potential for improving robustness: While the current version of ActiveVOI does not implement a fallback mechanism, this was a deliberate design choice to isolate and evaluate the contribution of VOI-driven acquisition. That said, we included ablations against three alternative acquisition strategies that may be seen as surrogates for a fallback mechanism: Exhaustive, Goal-directed, and LLM-Subgoal-guided acquisition (Table 2), all of which show over 40% performance degradation. The ablations suggest that ActiveVOI is already comparatively more robust than other strategies. We agree that robustness could be further improved by integrating fallback strategies, especially in safety-critical or real-world applications. One potential direction is to incorporate belief calibration or LLM/VLM uncertainty thresholds to trigger a conservative default strategy when model confidence is low.

We appreciate the reviewer’s comments on these important issues and will highlight this discussion in the revised manuscript.

---

[Weakness 2, Question 2] Scalability and Memory Persistence

Reviewer comment:

“Scalability and memory handling remain open questions: The planning horizon is capped, and the system operates episodically without a persistent knowledge base. In lifelong or multi-task settings, the agent would likely re-discover the same information repeatedly, as no explicit mechanism exists for long-term memory, knowledge abstraction, or reuse.”

“Knowledge persistence and reuse: Have you considered augmenting ActiveVOI with a memory system for storing verified object properties or relations across episodes? This could support lifelong learning and reduce redundant sensing in related future tasks. Would VOI still be usable in such a setting?”

Response:

We thank the reviewer for raising the important points of planning horizon selection, scalability, and long-term knowledge persistence. Below, we clarify how the current ActiveVOI framework addresses these issues and highlight future extensions.

Planning horizon: The lifted-regression planner used in ActiveVOI requires a finite planning horizon $t$ to enumerate all lifted plans up to depth $t$. In practice, $t$ is chosen based on the upper limit of the planning horizon reflecting time and cost constraints. For all benchmarks in our paper, we set $t = 15$, which corresponds to the upper bound of the expected episode length.

Per-episode KB and VOI caching: ActiveVOI maintains a structured KB during each episode. The KB compactly represents all verified object properties and relations as grounded predicates and is continuously updated through the agent’s observations and interactions. Additionally, we cache computed VOI scores to prevent redundant reasoning, which improves both cost and efficiency.

Persistent memory and VOI for lifelong/multitask settings: Supporting persistent memory across episodes could be an interesting future direction. Unlike LLM-based approaches that store full interaction trajectories, our framework enables structured transfer via compact symbolic KBs (and optionally their VOI caches). These representations are lightweight, abstract away irrelevant task-specific details, and can be easily reused across tasks in a lifelong learning setting.

The primary adaptation required to support VOI in lifelong settings is to provide a mechanism for Bayesian updating (recalibration) of object sensing probabilities in light of previous sensing experience. Existing comparison benchmarks do not evaluate this setting, but we will discuss this important future work direction in the revision.

Dear reviewer:

We sincerely appreciate your insightful review. With the reviewer-author discussion period ending soon, please let us know if we can further clarify our responses to any of your concerns.

best,

The Authors