Rejecting Hallucinated State Targets during Planning

HOW EVALUATOR PERFORMS IN PARTIALLY OBSERVABLE DOMAINS? WOULD THE APPROACH NATURALLY EXTEND IF THE “SOURCE STATE” IS UNCERTAIN?

We agree with your intuition that it naturally extends.

The evaluator takes on paired inputs of the source state representation and the target representation, both are outputs from the TAP agent to which the evaluator is attached. NO change is needed when the evaluator is fed representations (that can encode uncertainty) from a POMDP-compatible encoder.

We are confident about this, yet we know it may be more convincing to conduct experiments to provide empirical evidence. Unfortunately, ICML prohibits revisions this time.

IF THE GENERATOR PRODUCES MORE ABSTRACT OR HD LATENT GOALS (E.G., LANGUAGE INSTRUCTIONS OR SUBTASK DESCRIPTORS), WHAT CHANGES ARE REQUIRED FOR THE EVALUATOR?

Since the evaluator takes in target representations, our design needs NO CHANGE. We validated this empirically in Sec. 5.3. We tried to make clear that our approach currently only applies to TAP agents with a function $h$ verifying if a target is fulfilled. TAP agents’ $h$ gives us the compatibility to general targets including language instructions.

Instead of changes to the evaluator, the focus should be on a proper $h$ (e.g., ask the LM if a state matches the target). We discussed this at the end of Sec. 7.

THE GRID-WORLD-BASED TASKS REMAIN SIMPLE COMPARED TO MORE COMPLEX, HD ENVIRONMENTS. UNCLEAR IF THE APPROACH WILL SCALE WITHOUT ADDITIONAL ENGINEERING. … THIS … MAY BE LESS STRAIGHTFORWARD TO REPLICATE OR VALIDATE IN CONTINUOUS OR PARTIALLY OBSERVABLE DOMAINS.

Our reasons for the selected experiments are as follows:

1. These environments provide much more convincing quantitative validations of our claims on hallucinated targets and delusional estimates. Common benchmarks such as Atari, due to the lack of access to ground truth, cannot be properly diagnosed. As a result, we cannot prove that our method can indeed reduce delusional planning behaviors in those environments due to their nature.
2. We aimed to show our approach’s generality by applying it on many categories of TAP methods. The compute demanded in these experiments already exceeds what our limited academic environment provides.
3. Visual simplicity does not mean task simplicity. Due to the multi-task, generalization focused setting, agents are met with difficult combinatorial challenges that even state-of-the-art hierarchical planning methods cannot solve well, see [Zhao et al., 2024]. As an example, despite the visual simplicity, the hallucination rates remain high even with the used SOTA methods. For the points above, we focused on depth rather than breath when considering the environments for our experiments. Our approach does not assume the input space to be discrete or continuous. We chose discrete input spaces because it is nearly impossible to solve the ground truths, used to provide convincing, rigorous analytics, in continuous spaces.

For POMDP, please check our prior responses.

DISCUSSIONS ON THE COMPUTATIONAL OVERHEAD WOULD BE VALUABLE.

Technically, it is useful to view our solution as a special form of rejection sampling, where the proposal distribution and the support are provided by the generator, the target distribution is the re-normalized distribution over only feasible targets and feasibility estimates by the evaluator is used to determine the rejection. Thus, the more accurate the evaluator, the more efficient the sampling process. It is impractical to assume full access to either the target or proposal distributions because they change with the used 1) environment and 2) generator (whose output is not only different per method but also changing with learning). This means that we cannot give a blanket statement about the computational overhead, prompting us to have used only brief discussions about overhead in Sec. 4 (L189, left col.).

Practically, each generated target needs only 1 evaluation. For background TAP agents that generate batches of targets, the improper ones can be rejected and the whole batch can be all rejected without problem (no "Dyna" update this time). For decision-time TAP agents, targets act as subgoals and when they are rejected, the agent can retry or commit to more random explorations.

With DRL, the overhead also depends on evaluator’s networks, complexity of the state / target representations. However, since the evaluator is a rather lightweight secondary end-to-end network, we can expect evaluations to be fast. We added a condensed version of these comments to the revised manuscript.

---

We appreciate your review and your concerns about the applicability of our approach to more generic cases. We explained to you our choices and wish you could perceive the difficulties we overcame in acquiring the convincing results with limited resources.

We hope our responses addressed your concerns well and please consider increasing your rating.

We reordered the questions to streamline our response.

Discuss the relation … to [Z et al.] & [A et al.]. Are the targets for “generate” generated during sampling, or …?

[Z et al.]

Our approach of rejecting infeasible targets is indeed similar to that in [Z et al.]. Yet, the approaches differ significantly. [Z et al.]’s verifier trains on a collected dataset that informs about the generated subgoals (not relabeled) & requires separate training for the verifier. Unlike the method-specific approach in [Z et al.], we propose a generally applicable secondary system running alongside TAP agents, using relabeling, and thus it can figure out the infeasible targets (sets of states) that were not followed, without extra interactions or supervised training.

We wanted to try our best to acknowledge the developments leading to ours. This is why we discussed MHER (Yang et al., 2021) in Sec 6 for the 1st work of model-based to relabeling transitions (even closer to “generate”) and (Jaferjee et al., 2020) for their early idea of rejecting delusional generations. It was an honest mistake that we missed [Z et al.] and we now added proper acknowledgements.

"generate"& JIT relabeling

"generate" is novel not only for its flexibility to be used just-in-time (JIT, only relabeling after batch is sampled to adhere to the generators’ changing output distributions), but also for its effectiveness to address feasibility delusions. Compared to approaches like MHER, "generate" lowers needed storage and provides timely coverage of the generators’ outputs, especially helpful in continual learning settings.

"pertask" & [A et al.]

"pertask" is ONLY equivalent to “random” in [A et al.] when agents are trained on a single task. In settings where agents are trained on a few tasks and are expected to generalize during evaluation (where our exps were based, to force the evaluators to understand infeasible targets, instead of memorize), "pertask" enables relabeling beyond trajectory-level against delusions, per Sec 5.1.

the exps focus on strategies of training rather than the impact of evaluator on performance. … only indirectly argues that identifying invalid subgoals is useful. the main claims do not state any performance advantage … the impact of the evaluator on the performance of hierarchical methods?

This is a crucial misunderstanding that we wish to clarify, so we can resolve your other concerns effectively.

Our aim was not to show methods can do better with an evaluator, but a non-delusional one. Exps on the two HP agents focused on the fact that their basic relabeling strategies produce delusional evaluators (they have built-in evaluators). Only for methods without estimators to begin with, e.g. Dyna, we need to completely inject an evaluator. We discussed these explicitly in Sec 2, Sec 5.1.2 & Sec 6.

Due to the reply limit, plz find MORE details on this, in the 1st reply to Reviewer GvFL.

We added clarifications in the revision.

The naive approach to invalid targets is to ignore, as they aren’t very common, and even invalid subgoal guidance lead somewhere, from where another subgoal can be generated.

In Sec 2, we formulated your described scenario for the naïve approach, which motivated our contributions. The naïve approach only works when 1) infeasible targets are indeed rare and 2) the state space is simple. Also, the approach is boosted by 3) survivorship bias.

1. There are generally no measures of how frequent infeasible targets can appear, because most existing methods are not tested on proper environments, that can be solved to produce the true frequency of infeasible targets.
2. it is not hopeful that invalid guidance could lead agents to a recoverable region of a complex state space
3. Empirical observations of how naïve approaches could work are influenced by survivorship bias, as methods that are more affected by delusional plans have worse performance and are thus less likely to be recognized.

computational overhead?

Plz see our reply to Reviewer bSxw (last point).

the main claims should be reformulated, as they are too optimistic. The idea … to identify infeasible subgoals was already proposed …

We hope our previous explanations addressed your concerns on this point. We summarized our work’s claims and tried our best to clarify in the revision:

Building on the ideas of rejecting invalid goals such as in [Z et al.], our extensive study reveals the types of infeasible targets in TAP agents. Accordingly, the generality of our proposed solution leads to non-delusional feasibility estimates beyond HP methods.

---

We appreciate your detailed review! The intention of this work is to inform the research community about the associated risks and improper assumptions and save everyone’s time and effort. We take your comments very seriously and tried our best to address your concerns. We hope you can increase your rating to recognize our attitude and the positive impact this work could have.

WHAT IS THE PERFORMANCE OF SKIPPER & LEAP WHEN THE EVALUATOR IS NOT USED TO REMOVE PREDICTED INFEASIBLE STATES?

As discussed in Sec. 2 (Line 57 right column), Sec 5.1.2 (L383 left col.) , Sec. 6 (L388 r. col.), these methods already use their own evaluators to remove the targets they think that are infeasible. Yet, they suffer from delusions due to the lack of exposure to truly infeasible targets absent from collected experience.

Thus, our exps with both methods "reused" their built-in distance / discount estimators, which were originally trained with basic relabeling strategies. We showed that by fixing their relabeling, these methods become robust against delusional planning behaviors. Only for methods without built-in estimators at all, e.g., Dyna, we have baselines with the evaluator removed.

In other words, exps with Skipper & LEAP focuses on the importance of training data for non-delusional estimates. While exps 5/8 - 8/8 validate the combination of architecture, learning rules and training data at the same time.

Hope that you can now see that your wanted baselines are already there: they are the variants with "future" and "episode" (basic relabeling that causes delusions).

We realized that this could be somewhat confusing and tried to clarify in the revised manuscript.

THE METHOD IS EVALUATED ON ONLY 2 SIMPLE ENVIRONMENTS (SSM AND RDS). WHILE THE PAPER SHOWS THAT PERFORMANCE CAN BE IMPROVED BY REMOVING DELUSIONAL PREDICTIONS. IT WOULD BE INTERESTING TO EXPERIMENT ON COMMONLY USED BENCHMARKS …

Our reasons for the selected experiments are as follows:

1. These environments provide much more convincing quantitative validations of our claims on hallucinated targets and delusional estimates. Common benchmarks such as Atari, due to the lack of access to ground truth, cannot be properly diagnosed. As a result, we cannot prove that our method can indeed reduce delusional planning behaviors in those environments due to their nature.
2. We aimed to show our approach’s generality by applying it on many categories of TAP methods. The compute demanded in these experiments already exceeds what our limited academic environment provides.
3. Visual simplicity does not mean task simplicity. Due to the multi-task, generalization focused setting, agents are met with difficult combinatorial challenges that even state-of-the-art hierarchical planning methods cannot solve well, see [Zhao et al., 2024]. As an example, despite the visual simplicity, the hallucination rates remain high even with the used SOTA methods. For the points above, we focused on depth rather than breath when considering the environments for our experiments.

FROM MY UNDERSTANDING, THE LABELING OF TARGETS AS FEASIBLE OR UNFEASIBLE REQUIRES HAVING ACCESS TO GAME INNER LOGICS (HAVE SWORD, HAVE SHIELD). SUCH THAT WE CAN CORRECTLY LABEL SAMPLED TARGETS AS FEASIBLE OR UNFEASIBLE GIVEN THE START STATE. I DO NOT SEE HOW IT COULD BE APPLIED TO ENVIRONMENTS WHERE GAME LOGIC IS NOT ACCESSIBLE.

We respectfully note that this is a misunderstanding; our approach distinguishes itself from methods that require access to more environmental mechanics. The proposed evaluator figures out the feasibility of all targets through Eq. (2) without the need for labels: Eq. (2) exploits the fact that infeasible targets will never be reached.

Eq. (2) enables the evaluator to learn as a secondary system alongside the TAP agent to which it is attached, and figure out from data, the feasibility of the sampled targets it is trained on.

Ground truths are only used for experimental analyses, our solution operates without ANY.

THE PAPER TALK ABOUT POSSIBLE EXTENSIONS TO OTHER TAP METHODS SUCH AS MUZERO, SIMPLE OR DREAMER BUT DO NOT PERFORM EXPERIMENTS ON THESE KEY METHODS.

In Table 2, we categorized the compatible methods into 3 categories of similar behaviors, and in experiments, we implemented one representative method for each of the categories.

We made a conscious effort to provide you with the most convincing results out of the limited resources at our disposal. We hope that you understand our difficulties in an academic setting.

---

Thank you for your comments. Your biggest concern was about the applicability of the method without access to feasibility labeling. We hope that our explanations will address our miscommunication on this point. For your comments on experiments, we clarified that the certain baselines in the mentioned figures are the original performance of the agents, as both Skipper and LEAP are equipped with built-in feasibility-like estimators, which we dual-purposed as evaluators.

We are thankful that you acknowledged that “this direction deserves to be explored” and hope that our responses have addressed your concerns, and you could increase your rating of this work to recognize our contributions.