Thank you for your positive feedback and thoughtful analyses of DynaGuide. We appreciate your highlighting of Dynaguide’s robustness to lower quality goal conditions that would otherwise cause goal-conditioned baselines to fail. Below, we answer the questions and address the limitations.

Answers to Questions

Sensitivity to negative goals

Empirically, we didn’t have to cherry-pick the negative goals to get DynaGuide working on the multi-objective setup. Sampling the negative states as the final states in a randomly selected set of validation trajectories was sufficient.

The performance of DynaGuide depends on the ability of the latent space to separate out desired and undesired outcomes. We use the DinoV2 latent space, as past work [1] had yielded successful results in the planning domain. If a particular task has similar success and failure states, then a special latent space should be constructed that separates them appropriately. This latent space will probably need to be trained in a task-specific way. The DinoV2 space was sufficient for our experiments, but improving the latent space is a meaningful direction for future work.

Alternative guidance modalities

As long as you can embed the guidance conditions into the latent space, any format of guidance condition should be valid. For example, if a CLIP embedding is used, then text can be used as guidance conditions as well as images. The latent space can also encode history and kinematic properties of the trajectory, if trained with the correct objectives. We leave the exploration of latent spaces to future work. The DynaGuide method is general and should work on these specialized latent spaces.

Discussion of Limitations and Weaknesses

Guidance is Outcome-Based / Lack of Trajectory Control

In the simplest formulation DynaGuide, we steer the policy with guidance conditions that represent outcomes. However, because Eq. 2 is a soft minimum, it is also possible to add intermediate guidance conditions that represent how a task should be done. We actually did this for the real robot setup due to the wrist camera’s end observation being not a good representation of the task. We found that these intermediate guidance conditions were critical in getting the desired performance improvement. DynaGuide as a framework supports guidance at any point(s) in the trajectory, as long as they can be represented as anchor points in a latent distribution.

Reliance on Dynamics model

We agree with this concern. By construction, DynaGuide relies on the dynamic model’s predictions for good guidance. However, the dynamics model can be trained on additional lower-quality and/or less-relevant data compared to an expert policy, making it easier to improve the robustness of the dynamics model compared to doing the same for the expert policy. In our Calvin experiments, we train the dynamics model on interactions in other domains (ABC split). In our real robot experiments, we add data from other object interactions. In both cases, the dynamics model benefited from this additional data.

Limited Baselines

There is indeed a line of work that integrates world models into the planning and diffusion process. Because we were focused on the ability to guide existing policies, we were less thorough in the planning-based baselines like DINO-WM and Diffuser. On the bigger picture, these approaches are still relevant and we will try to implement & test some of these approaches for the final publication. In the meantime we added the following baselines in response to other reviews

• Adding DynaGuide-Sampling (GPC) to real world results (see: RYQt)
• Reducing the dynamics model training dataset to D from ABCD (see: yJvQ and n1oB)
• Increasing the data of the goal-conditioned policy to ABCD (see: yJvQ and n1oB)
• Adding HER-like training to goal-conditioned policy (see: n1oB)
• Testing classifier-free guidance on the multi-objective task (see: n1oB)

Computational Cost

The MCMC sampling is indeed adding to the computational overhead, although unlike sampling, DynaGuide’s computational cost is not dependent on the rarity of the target behavior or the difficulty of the task. When run on an RTX 3080 with a real robot, this inference process still runs in real-time due to the model predicting entire action chunks at once. For a profile of run times, see our response to Reviewer n1oB.

No online learning

It is true that the guidance is forgotten afterwards. However, the guidance is repeatable because the guidance conditions are stored. Guidance conditions can be added or removed dynamically based on past performance–and this is a notion of adaptation / persistence. This online learning may not be as expressive as finetuning, but it is more explainable, robust, and easily reversible if needed. It adds to the modular capabilities of DynaGuide.

Summary

Thank you for your thoughtful observations and questions. We have done our best to provide answers to questions about DynaGuide’s limitations and possibilities for other modalities and applications. We welcome any follow-up questions or comments.

[1] Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning.

Thank you for the thorough comments, critiques, and questions. We appreciate your highlighting of the real robot experiments present in DynaGuide and the quality of experimental design. Below, we include additional experiments and discussion based on the points raised.

Addressing Main Limitation Remarks: Additional Experiments

Lack of GC baselines like HER

In the table below (additional results on exp. 3), we add GC with HER. To implement a HER-like algorithm, we train the GC policy on intermediate goals selected uniformly from a future observation in a trajectory. The results show some improvement on select tasks (notably the door task) but drops in other tasks. Similar results happen when the GC baseline is enriched with the ABCD data split (GC with ABCD). Overall, training on intermediate goals / enriched data had mixed results on performance. DynaGuide still consistently outperforms it. We will correct citations in the final publication.

Wall Clock comparisons of DynaGuide

We have included some wall time analyses below on a 3090 GPU with no additional optimizations.

• DynaGuide. Setup: 4 stochastic sampling steps, 10 inference denoising steps (DDIM). Time for inference (for chunk of 16 actions): 1.2±0.08 seconds
• DynaGuide-Sampling. Setup: 5 samples, batched. Time for inference (for chunk of 16 actions): 1.05±0.02 seconds
• Base policy / Goal Conditioned. Time for inference (for chunk of 16 actions): 0.09±0.01 seconds

When batched, DynaGuide-Sampling runs marginally faster than DynaGuide due to fewer runs through the dynamics model. However, for rare behaviors, the sampling baseline must run many samples to get a desired behavior. Scaling this through batching would require larger memory. In contrast, DynaGuide does not require any additional computation or memory scaling to search for underrepresented behaviors successfully (Exp. 5).

Dynamics Model Gets More Data

This is a concern shared by reviewer RYQt. In response, we have run DynaGuide using just the D split. Reducing the dynamics model dataset does reduce performance, as expected. However, as seen in the GC table above, adding the ABCD split to the goal-conditioned baseline does not improve performance much. Therefore, the dynamics model makes better use of the additional data compared to the GC baseline.

Classifier-free Guidance

This is a reasonable baseline for the multi-objective setup. We implemented CFG and provide some basic results below for multi-objective, focusing on two tasks in Exp. 4. We tuned CFG’s guidance strength using a grid search to obtain these values.

As you suggested, CFG can follow multiple objectives. However, its average per-task success (0.3) is lower than DynaGuide (0.38). One possible reason for this is the averaging of signals from the conditional models in the CFG implementation. In contrast, DynaGuide’s distance-based log-sum-exp prioritizes objectives that are the most reachable, leading to better performance with conflicting objectives.

Other Main Remarks and Questions

Novelty of Contribution

DynaGuide uses classifier guidance like ITPS, but the guidance signal is propagated through a dynamics model instead of euclidian distance in the action space. DynaGuide uses a latent space like DINO-WM, but instead of running model-based planning, it modifies the behaviors of a base policy. DynaGuide also consolidates the latent distances of multiple objectives (eq. 2), which is not possible in DINO-WM.

In general, DynaGuide builds upon DINO-WM and ITPS, but combining these approaches required distinct innovations. The final result yielded a novel approach to guiding diffusion robot policies. We also provide real robot results, which were not provided by DINO-WM.

Improper comparison of future state to desired end state

This is a valid concern about the method. In the CALVIN environment, early task disambiguation allowed us to let $t+H$ be the last state of the trajectory (mentioned in Lines 149-150; this will be clarified in the final publication). Because the dynamics model predicted the final state, it was directly comparable to the desired final states. This trick was also used in DINO-WM [1].

On the real robot experiments, the longer horizon and the wrist-mounted camera meant that $t+H$ could no longer be the final state. To account for the exact issue raised by your comment, we sampled states throughout desired behavior trajectories as the guidance conditions. The log-sum-exp acts like a soft minimum for the latent distances, so it automatically downweights states that are too far away / unreachable.

These details are somewhat hidden in the current paper version so we will add more details in the final publication.

Expanding DynaGuide to stochastic environments

The current dynamics model is best for deterministic environments, but Eq. 1 can also be interpreted as a stochastic log-probability loss with unit variance. To properly adapt the dynamics model to a stochastic environment, we would need to make the variance learnable, i.e. getting the model to provide a measurement of uncertainty through specialized architecture like a VAE. This is an interesting direction for future work.

Other Remarks

Lack of position guidance baseline in moving object experiment

Position Guidance requires knowing the exact 3d location of an object to define a euclidian distance objective for use during steering. In ITPS, a human provided this 3d location in every environment rollout, which enabled its use on movable objects. In all our baselines, we don’t assume privileged state knowledge. For fixed objects, we get their locations by using the final gripper positions in the training data. This heuristic is not available for movable objects.

Choice of action horizon value

This action prediction horizon value is 16, and while we never conducted extensive experiments for this value, we found that the guidance is weaker for shorter values like 8 and the diffusion policy performs worse for very long values like 24 or 32, so we picked 16 as a balancing point.

DynaGuide requires base policy to have ability to solve target task

For steering approaches, it’s generally assumed that the base policy has the skills to achieve a behavior. Otherwise, the base policy will not be of much use. The referenced claim in lines 34-42 focuses on assuming knowledge of the steering objective, which is necessary for goal-conditioning because it is trained on the steering signal. In contrast, DynaGuide can handle novel steering objectives that leverage existing skills.

An elucidating example: without using CFG, our goal conditioning won’t be able to avoid a behavior unless we train it specifically. However, we can avoid a behavior with DynaGuide by negating the guidance signal. This is an example of an unanticipated steering objective being applied to known skills. This should be clarified in the paper and we will do this in the final publication.

Goal Assumptions

Assuming access to goal observations for guidance is a much stronger assumption than relying on goal specification via natural language, for example.

Yes, this is absolutely true. Although, the data needed to train a language-conditioned model requires more assumptions (task segmentation, good task labels, etc). We were having this interpretation in mind for line 103, although we see the ambiguity and will correct accordingly.

Typing Errors in Algorithm 1

Thanks for finding these typing errors. We will fix them in the final paper.

Formatting concern

Based on our observation of past NeurIPS papers, our interpretation of the CFP remarks was that they referred to main paper margins / main text size. We are happy to change the caption formatting if it was the incorrect interpretation.

Summary

Thank you for the very detailed review and thoughtful questions. We have done our best to answer them by providing additional results including wall clock analysis, results with a different data split, stronger goal-conditioned baselines, and others. We also answer other conceptual questions and provide some discussion. We can provide further clarification if anything was not clear.

[1] Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning.

Thank you for the response. The GC baselines using HER and the experiments with CALVIN-D are appreciated.

Wall Clock comparisons of DynaGuide

The appendix specifies that the denoiser used 100 steps for all experiments (line 984). Therefore, the wall clock times in the rebuttal do not consider the same setting as the experiments in the paper. I request the authors to perform this analysis using the same parameters as the experiments to ensure fairness.

Classifier-free Guidance

My comment meant that CFG is a natural baseline for all settings, and can also handle multi-objective guidance like DynaGuide. In general, CFG is the preferred method for conditional sampling using diffusion models over classifier-based guidance (like DynaGuide), so I believe this is an essential baseline, which should have been part of the paper.

Improper comparison of future state to desired end state

Thank you for the clarification. If I understand correctly, if some tasks do not show early task disambiguation, then a similar procedure of sampling intermediate states from desired behavior trajectories must be used? This will be a much stronger signal for guidance (since now guidance is provided every H steps rather than only at the goal state). This raises questions about general applicability of DynaGuide beyond tasks like CALVIN, and how it compares against other methods if it uses the same guidance signal as other methods (only the goal state).

Lack of position guidance baseline in moving object experiment

Perhaps I am missing something, but why can't the end gripper position from the dataset be used to get the position of movable objects?

Thank you for your comments and additional points.

Wall Clock comparisons

The lines referenced in the appendix (B1-Dynamics model) references the scheduler used to train the Dynamics model, not the denoiser of the diffusion policy. As is standard for diffusion policies, we train on 100 denoising steps but run inference on 10 denoising steps through the DDIM scheduler. All the results obtained in the paper have been done according to this DDIM 10-step scheduler, so we maintain that the wall clock results are correct and fair.

Classifier-free Guidance

We agree that CFG is a meaningful general baseline and we will add a full sweep of these numbers in the final publication. We also agree that CFG is more standard than classifier-based guidance in generative modeling outside of robotics. However, in robot-learning, an external guidance signal provides distinct advantages, including additional expertise and an explicit understanding of dynamics. In contrast, with CFG this understanding is implicit. The vanilla CFG implementation also lacks the ability to explicitly shape guidance signals based on their reachability, which is enabled with the latent distances of DynaGuide. These challenges are less present in non-robot settings, and quality data is also more abundant for training CFG as compared to the robot setting. These factors might be the cause behind the lower performance of the CFG baseline in the robot setup, although we will have stronger conclusions once we get more CFG results.

Improper comparison of future state to desired end state

Thanks for continuing this discussion. Yes, for the real robot experiment we used a uniform sampling of intermediate states in the trajectory as guidance conditions due to the wrist camera giving partial observability. In general, DynaGuide works for both endpoint guidance and intermediate point guidance due to the log-sum-exp of the latent distances that provide a soft reachability selection for guidance signals. Therefore, DynaGuide is not only limited to tasks with early disambiguation like CALVIN. As an immediate example, the toy task (A.1) addresses these questions about early and late disambiguation. There, the base policy is trained on spline curve data with up to 3 turn points, meaning that the guidance must happen continually to reach the correct color. The Hidden Cup experiment in the real robot also shows a late decision example. DynaGuide is still able to meaningfully guide the agent towards a desired color (Figure 7) and cup (Figure 6). In our CALVIN experiments, we intentionally only use the goal state to keep it fair with goal conditioning and other methods that only used end state conditioning.

Lack of position guidance baseline in moving object experiment

For stationary objects, the end gripper position in the dataset will be on average the location of that object. However, the movable objects are randomly shuffled between environment instances (i.e. between resets). The end gripper position on those tasks will always be on those movable objects, but the locations will be randomized. Therefore, the average gripper position is no longer meaningful. I think the movable objects may have been a bit of a misnomer / confusing term to use on our part. What we explicitly mean is this: stationary objects--things like cabinets, buttons. Does not move between environment resets. movable objects--things like the colored blocks. Placed randomly between environment resets. We investigate these objects as a means of showing conceptual guidance as opposed to place memorization. These results are supported with our real robot results and toy agent experiment.

Thanks again for the response, and we welcome additional discussion!

Thank you for your positive response, detailed feedback about the CALVIN environment, and clarification questions about our experiments. We appreciate your recognition of DynaGuide’s versatility and our results in both simulation and real setups. Below we address your questions and add additional experiments inspired by your comments.

Dataset Splits: ABCD vs. D on Calvin (additional experiment)

Yes, you are correct that the base model is trained on D and the dynamics model is trained on ABCD, which contains more data. We used the ABCD split to showcase the performance of DynaGuide with a dynamics model trained as well as possible. This is how we ensured fair comparisons in our paper results:

• Comparing DynaGuide to DynaGuide-Sampling, we used the exact same base policy and dynamics model, varying only the steering method (sampling vs. guidance)
• Comparing DynaGuide to Position Guidance, we used the exact same base policy and steering method, varying only the steering signal (dynamics model vs position euclidian distance).
• Comparing DynaGuide to Goal Conditioning, we trained the base policy and goal-conditioned model on the same dataset and used the same guidance conditions.
• Comparing DynaGuide to base policy, we used the same base policy inside of DynaGuide, meaning that performance improvements are explained solely by the guidance.

We agree that training the dynamics model on the D split is a meaningful additional experiment because it completely isolates performance improvements of DynaGuide from data quality. Below are the results on Exp. 1 with ABCD (paper results) and D splits for the dynamics model (requested)

As expected, the dynamics model performs worse when trained on a smaller dataset. However, in practical implementation, the dynamics model may be trained on a more diverse dataset than the base policy because it requires fewer quality assumptions about data than the base policy. Therefore, we can ask another question: does DynaGuide make better use of a larger data split than a goal-conditioned baseline? To answer this question, we train the goal-conditioned baseline on the ABCD instead of just the D split shown in the paper. Below are the results on Exp. 3 (which originally shows a large impact in goal-conditioned performance.)

Notice how the goal-conditioned model benefits slightly from additional data but still is far away from DynaGuide’s performance on Exp. 3. This showcases that DynaGuide can make better use of this additional data split compared to the goal-conditioned baseline.

Conclusion from these two additional experiments: although the dynamics model degrades performance when trained only on D (DynaGuide w/ D exp), it leverages this additional data better than the goal-conditioned baseline (GC w/ ABCD exp).

Calvin Underspecified Objective Question

I am confused by the CALVIN underspecified objective. I assume any highlighting in Figure 3 is just for illustrative purposes? So what does it mean to "randomize robot states (and other states)"? Because I am quite surprised by goal-conditioning's performance drop.

Yes you are correct that the highlighting / greyscaling in Fig. 3 is purely for illustration. In reality, in Exp. 3 we supply the methods with guidance conditions (goal images) that are out of distribution by showing the desired environment state but not the correct robot state.

Concretely, for all other experiments, the guidance condition shows the robot interacting with the target object because the guidance condition image is taken after the robot accomplishes the task. For exp. 3, we still show the target object in the desired configuration but we show the robot in a random position (randomly selected state in a validation dataset). As an example: normally for the drawer-open task, the guidance condition shows the robot holding onto the open drawer by the handle. In Exp. 3, the guidance condition shows the same open drawer, but the robot is not grabbing the handle.

Because all the models (dynamics, goal-conditioned) were trained with the robot on the target object, these guidance conditions in exp. 3 are generally out of distribution. This is why the goal-conditioned model's performance drops significantly. DynaGuide does not drop as much because these guidance conditions are still in-distribution to all individual components of the method; the guidance signal just becomes noisier. The guidance signals can still pull the policy in the direction of future states with open drawers, for example.

We conducted Exp. 3 to showcase how the DynaGuide process can handle these out-of-distribution guidance conditions better than a model trained end-to-end on the goal distribution. Especially when we are trying to steer an off-the-shelf policy in a new environment with our own guidance conditions, this robustness is especially important. We are happy to clarify further if this response does not fully answer your confusion.

Counteracting BC Performance drop

Could DynaGuide guidance counteract BC's performance drop? Such an experiment would also more nicely highlight DynaGuide capabilities to work with different, "off-the-shelf" policies.

Assuming that the dynamics model is still in distribution, it would be reasonable to hypothesize that DynaGuide can improve BC performance when the base policy is out of distribution. We base this hypothesis on the results of Exp. 5, where DynaGuide can pull a very underrepresented behavior from a base policy. Executing this behavior is out of distribution for the base policy due to intentionally removing that part of its training distribution. DynaGuide is still able to coax the base policy to execute that behavior.

We also see an example of this on the real robot setup, where DynaGuide is able to get the robot to deliberately search for a red-colored cup when the base policy defaults to going towards the closest cup (Exp 6, Fig. 6 middle plot).

In general, if the dynamics model is robust, then DynaGuide could be a tool for counteracting BC performance drop. To say this with more certainty, however, more experiments are needed.

Multi-objective guidance

On multi-objective guidance: Am I understanding correctly, that each objective is used to produce a guidance signal independently, which is then combined across all objectives, for example by addition? Or does the Dynaguide model actually receive a set of objectives at once?

So DynaGuide actually takes a collection of guidance conditions and aggregates the distances into a single metric (Eq. 2). This seems like the latter approach in your question. This means that we only need to backpropagate through the dynamics model once per diffusion step. It technically is possible to compute guidance signals (action gradients) independently and aggregate them after backpropagation, although it is less computationally efficient.

Summary

Thanks for your comments and feedback. We add additional results using a dynamics model trained on the D CALVIN split as well as a goal-conditioned model trained on the ABCD CALVIN split to further isolate the impacts of individual DynaGuide components. We also address various questions about the method. If any of your concerns were not addressed by this response, we welcome further discussion.

Thank you for your detailed and thoughtful review of our paper. We appreciate your recognition that DynaGuide presents a novel approach for steering diffusion policies, and we appreciate your highlighting the performance boosts of DynaGuide in situations with under-represented behaviors and multi-guidance conditions.

Two main points are raised in the weaknesses / questions section, which we will address here, along with other minor points.

Real-World Experiments: Lack of other Baselines

Lack of goal-conditioned baseline

The primary concern mentioned in the weaknesses section is the lack of a goal-conditioned baseline in the real robot experiments. As mentioned in lines 325-326 and lines 71-74, one of the primary purposes of the real robot experiments is to test the hypothesis that DynaGuide can work on any pretrained policy. To do this, we use a publicly-accessible off-the-shelf policy [1], which unfortunately does not have a goal-conditioned version. Therefore, in that set of experiments, it was not possible to conduct a fair comparison against a goal-conditioned model, which we would have needed to train ourselves.

On the longer time scale, we can conduct an additional robot experiment set where we train our own policies and allow for goal conditioned comparisons. If we train our own policies with our environment-specific data, we anticipate the results to be stronger across the board.

Lack of other baselines (additional experiment)

The simulation environments provided a stabler and repeatable environment for head-to-head comparisons, which is why we chose the simulation setup to conduct extensive baseline comparisons (exp 1-5). DynaGuide does not make any assumptions that would cause a significant difference between real and simulated experiments. However, we acknowledge the reviewer’s concerns about the need for more baselines in a real-world setup.

We added the DynaGuide-Sampling (GPC) baseline on the real robot. Due to the short rebuttal turnaround time, we were only able to evaluate it on the first real robot experiment (pick red cup and place on black saucer). We got the following results (20 trials per experiment)

As noted in the paper, the pretrained base policy is biased towards one side of the environment (i.e., it will typically grab cups on the left side). We shuffle the cups equally for fairness, but to get above ~50%, the method must overcome this bias. Sampling was not effective in removing the bias, while DynaGuide was able to accomplish this.

Hyperparameter Tuning

Selection of this scaling parameter might require additional tuning to achieve a reasonable policy versus other more direct optimization methods.

This is a valid concern, as additional parameters mean additional hyperparameters to tune. However, as seen in Figure 8, the guidance strength parameter (the scaling parameter) is robust to large changes in scale (leftmost plot). DynaGuide is also relatively robust to the sigma parameter (second plot from right) and even the number of guidance conditions (rightmost plot). The stochastic sampling steps (second plot from left) generally increase performance with more steps, so they can be chosen based on resource constraints.

To get the results of DynaGuide shown in the paper, we only needed minimal tuning. Notably, these hyperparameters remain relatively stable across individual tasks within an environment. As an example, here are our optimal hyperparameters for the individual tasks:

The two most commonly changed parameters are Scale and Sigma. Generally, to deploy DynaGuide, we advise starting with guidance scale 1-2, and increasing if the policy does unwanted behavior. Choose a smaller Sigma if the guidance conditions have more variance and you suspect that only a few are relevant per environment, and vice versa.

Summary

Thank you for the comments and feedback. In our response, we detail why our current real-world experiment setup was unable to utilize a goal-conditioned baseline fairly. We conduct an experiment with a GPC baseline on the real robot setup. Finally, we discuss the sensitivity of DynaGuide to its hyperparameters and provide simple rules for tuning them. If any of your concerns were not addressed by this response, we welcome further discussion.

[1] Chi et al. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. RSS.