Latent Weight Diffusion: Generating policies from trajectories

• Could the authors compare the performance of the two models under disturbances to show how the learned closed-loop policy improves stability? Can it improve stability with disturbances?

This is a great suggestion. We are running evaluations now to compare the performance of the two models under disturbances. We will update you with the results when we have them.

• Warmstarts for diffusion policy: While we do not currently have warmstarts for the diffusion policy, it would be a great direction for future work. We will add this to the future work section.

• MLP in table 2

Does it predict a sequence of future actions or just a single- step action?

We want to clarify that LWD does not predict a sequence of future actions. It generates a neural network that acts as a feedback policy, which takes in an observation and outputs a single-step action.

The MLP in table 2 is predicting a single-step action. Moreover, the policy generated by LWD is also predicting a single-step action from a single observation. Thus, it is an apples-to-apples comparison.

• The authors didn't show the success rates of diffusion policy and LWD.

Yes, we showed the relative performance change with respect to sucess rate -- this is to be consistent with Fig. 6 of the Diffusion Policy for the same ablation.

• Why is LWD close-loop and why does the author say diffusion policy is open-loop?

The LWD policy is closed-loop because the generated policy takes in the current observation and outputs a single-step action. The action is then executed, and the next observation is fed back into the policy. So, while LWD generates a new polcy every $H_a$ steps, the generated poicy itself is closed-loop.

By contrast, the diffusion policy takes in the current observation and outputs a sequence of actions. The sequence of actions is then executed open-loop, without feedback from the environment.

• didn't compare the success rates between LWD and diffusion policy on any task

As mentioned above, the metric explained in Fig. 8 is the success rate over 50 seeds, for the pushT task. We will make it clear in the manuscript.

Furthermore, we have now added comparisons of the success rates of LWD and Diffusion policy on the MetaWorld MT10 suite of 10 tasks. We have updated Table 2 in the manuscript with the success rates for both policies. Surprisingly, diffusion policy seems to struggle in multi task settings, even with 27 times more parameters than the LWD generated policy.

We would like to thank the reviewer for their detailed comments.

• cite [2, 3] for the statement "...to learn smooth cost functions for the joint optimization of grasp and motion plans."

We have added these to the manuscript.

• comparing foot contact distributions between models for ablation study is not meaningful.

We appreciate the reviewer's suggestion to show behavior diversity outside of locomotion tasks. The field of QD works a lot on locomotion tasks, and this is where behavior diversity is most commonly shown, in the form of foot-contact distributions (e.g. PPGA, PGA-ME).

Having said that, we provide an example manipulation task that does not involve foot contacts. We have applied LWD to reconstruct behaviors on the Adroit hammer task, where an agent needs to pick up a hammer and hit a nail into a wall. The dataset for this task consists of 5000 trajectories each for two behaviors, named 'cloned' and 'expert'. The results are added in Figure 5 and discussed in Section 4.1 of the main manuscript.

• "Fig. 3 would be much clearer if merged into one figure"

We have added a new figure that combines the plots into one. The combined figure is attached at the end of the supplementary pdf. We feel this image is slightly cluttered but if the reviewer prefers this to the original, we can update the manuscript accordingly.

• In all experiments, how many diffusion steps are typically needed to decode the policy network?

1000, we have added this detail in the appendix.

We would like to thank the reviewer for their detailed comments.

• The discussion in Sec 4.2 is fairly lacking, with regards to decoded policies representing the original trajectories.

You mentioned two quotes from Sec 4.2: One states that the policies decoded from trajectory snippets follow the behaviour of the original input snippets. Another states that the policies decoded from the trajectory snippets did not perform as well as the original policies. I can see why this can seem as a contradiction. However, the difference between these two is the task on which it is trained.

For the first quote, the task trained was a half-cheetah locomotion task, with three different policies being test. This is a cyclic task; our hypothesis is that a trajectory snippet has enough information to indicate the source policy.

To test this hypothesis, we tested LWD with non-cyclic tasks next. These are the suite of 10 manipulation tasks from the MT10 task suite. We noticed that for these non-cyclic tasks, even though we had the same size decoder as the half-cheetah tasks, the policies decoded from trajectory snippets were not faithful to the original policies that created these trajectories. This was the second quote you pointed to in section 4.2.

Thus, while the two quotes seem inherently contradictory, it is a feature, not a bug. Our goal was to show this contrast in policy reproduction through an ablation of task choice.

We have changed the text a bit to reflect this in section 4.2.

• I am fairly curious about the hypernetwork decoder setup

We have added all details regarding our model in the appendix

• \tau is not defined when it is first introduced in Sec. 3.1, as well as \varepsilon in Sec. 3.3

$\tau$ is actually defined on line 156, in section 3.1. Unfortunately $\mathcal{E}$ was not defined in sec 3.3. We have fixed that.

• In Fig. 8, how is Relative Performance Change defined?

Is it defined relative to the policy itself, or on an absolute scale? I find it quite misleading.

Relative performance is an absolute scale. So if you have a performance drop from 50% to 45%, the drop is 5%, not 10%. This helps keep the same benchmark (absolute 100%) to compare relative performance drops for all methods.

• What is the action parameterization that LWD generates? I assume it is not diffusion?

LWD's output is a policy. Thus, it does not have a action parameterization directly. However, the policies that LWD generate compute an action for a single step, given the current observation, i.e. $a ~\sim \pi(s,a)$.

• Why is the added action noise needed?

a_t= \pi(s_t, \theta) + e, where e is normally distributed around 0.

This is to account for system noise in the collected data. Every time we collect data from a robot, the robot’s actions will always have some noise. The policy we're collecting under is actually $\pi_{data}(s_t, \theta)$. But the true action executed can be slightly different from this policy. In the theory, we typically add the noise term to account for process and measurement noise (e.g. uncertainties in the physics model, observations).

We assume this noise is gaussian, and is centered around the robot's desired action. This assumption is common for regression as it simplifies to MSE loss, which we show in sec 3.2.

• Other dimensions of the PCA analysis in Figure 5

The first few dimensions of the PCA will explain most of the variance in the data. We have added the information for the subsequent dimensions as well, in appendix D of the updated manuscript.

• Three lines over 10 page limit

We are now under the page limit in the updated manuscript.

I thank the authors for answering my questions and concerns.

Re: discussions in Sec 4.2. I agree with your discussion, but it again confuses me: are you saying that your method does not work on the common MT10 tasks then? Or you mean that in Sec. 4.3, using a bigger decoder size addresses this issue? I appreciate the experiments and ablations, but I just find the presentation and the overall message quite confusing (If you would like to clarify some changes you've made, please highlight it here)

Re: Relative Performance Change. I see. Do LWD and the original diffusion policy have similar success rates on an absolute scale? i.e., does using LWD affect the performance negatively compared to just using the original diffusion policy?

Re: action parameterization. "LWD's output is a policy." This is a bit confusing to me again. I consider a policy consisting of (1) model architecture and parameters (e.g., parameters of a MLP, or transformer, etc) and (2) action parameterization (e.g., Gaussian, diffusion, GMM, etc). I assume the last layer of the generated policy is used as the mean and std of a Gaussian parameterization?

Re: added action noise. Thanks for the clarification. I missed the sampling from the Gaussian part in the text.

Thank you for the clarifications.

I would like to maintain the original scores for now since my main concern about the paper, which is the lack of novelty and technical contribution, has not been addressed. Also, although I better understand the results with the halfcheetah and MT10 tasks now, I think the overall presentation can be improved to convey the affirmative conclusions drawn from the experiments.