Residual-MPPI: Online Policy Customization for Continuous Control

Thank you for your thoughtful feedback and valuable suggestions. We are glad you found the Residual-MPPI novel and that our experiments are solid in complex environments. Please find our responses to address your concerns regarding our work below.

1: Writing and Orgnization

W1: The quality of the work should ideally be improved before the work is published. The full 10 pages are used, but there are many aspects which are unclear/undefined (see below), while there are large sections of text for which the key point is not clear.

W2.2: 'Few-shot' and 'zero-shot' are mentioned throughout the paper, but not defined until Section 5.1 and it is not clear what these terms refer to until this point. These should be defined in this context before use.

W3.1: The add-on rewards used for the MuJoCo tasks are not stated in the main paper, only the Appendix. Some examples or at least the kind of add-on tasks considered should be included in the main paper to help the reader intuitively understand what kind of customization is being considered, and to define the last but one column of Table 1, which is meaningless otherwise.

Q5: Why are two decimal places used in Table 1? This would be clearer to the nearest integer (except the undefined last but one column).

Thank you for the valuable suggestions on paper organization and presentation clarity. We provide a more concise version of the discussion, abstract, and related work. The citations are now in brackets, and the appendix comes after the references in the main paper. We add a brief introduction to the exact add-on tasks before presenting the results in MuJoCo. Please find more detailed updates in the revised paper.

2: Novelty of Theory

W2.1: The relevance and novelty of the theory is unclear to me. The approach seems to be a direct application of MPPI to Resdiaul-SAC. What is the intended contribution of Theorem 1 to the paper? It is derived in the Appendix, but the resulting Equation 5 just appears to be a direct result from [1] and it is mentioned that reference [2] already combines MPPI with SAC. Calling it a Theorem seems unnecessary if it is not a new theoretical result. The purpose of this theorem and the logic behind the derivation of Residual-MPPI could be made clearer.

Thank you for asking the questions about the Residual-MPPI theory. Please find a thorough discussion in Comment 4 in the general response.

Theorem 1 is required to validate the max-entropy property of MPPI, which is a prerequisite for the theoretical foundation established in RQL. The augmented MDP in our formulation is derived under the assumption of max-entropy optimization. Only when optimizing a max-entropy policy on the proposed augmented MDP $\mathcal{M}^\mathrm{aug} = (\mathcal{X}, \mathcal{U}, \omega^\prime \log \pi(\boldsymbol{u}|\boldsymbol{x}) + r_R, p)$ does the resulting policy become equivalent to optimizing a max-entropy policy on the full MDP $\hat{\mathcal{M}} = (\mathcal{X}, \mathcal{U}, \omega r+r_R, p)$.

Additionally, we would like to clarify that the primary theoretical contribution of our work lies in deriving Residual-MPPI by applying max-entropy MPPI to the formulated augmented MDP. Specifically, the reference [2] is not targeted on solving the policy customization task and can be viewed as the Valued-MPPI baseline. While the max-entropy property of MPPI has been proved in the literature [2], it is derived with a different problem formulation. Thus, we provide a self-contained and concise notation of this property in Theorem 1 and Appendix A to better serve our purpose.

Since it is not our main theoretical contribution, we have renamed it to Proposition 1 in the revised version to more accurately denote the scope of our theoretical contribution.

3: Additional Baselines

W3.2: There are no external baselines for the MuJoCo experiements. Could residual-SAC be used as a baseline here?

Thank you for noting the additional learning-based baselines. We have included the results of suggested baselines in MuJoCo. Please find a complete discussion and additional results in Comment 1 in the general response and the revised Table 1 in Section 4.

Thank you for your thoughtful feedback and valuable suggestions. We are glad you found the Residual-MPPI novel and that our experiments are solid in complex environments. Please find our responses to address your concerns regarding our work below.

1: Reward-Level Evaluation in GTS

W1: In Section 5.1, the paper argues that the metrics to evaluate on GTS are lap time and off-course steps since the rewards are complex. Nonetheless, I believe the work should provide both the prior task reward and the add-on reward for each method, as in the MuJoCo case, to understand how the method influences the returns that are optimized. Perhaps the considered small changes in lap time are associated with big changes in the returns, which would mean that Residual-MPPI is actually considerably deviating from the prior task designed by the environment.

Thank you for noting an additional way for evaluation in GTS. In GTS, we seek to solve a complex, realistic racing task with the proposed Residual-MPPI, where the most important metrics are lap time and off-course steps. GT Sophy 1.0 was trained to minimize the lap time. Since lap time is sparse and hard to optimize, a dense reward with multiple regularization terms was designed through reward shaping. This shaped and dense reward makes reward-level evaluation less informative and indirect. Therefore, we choose lap time and off-course steps as the evaluation metrics to quantify the performance of the policy customization in the basic and add-on tasks, respectively.

Specifically, the nominal term in the basic reward is the speed along the centerline, which is correlated with the lap time. This relationship implies that a small change in the lap time indicates a minor variation in the basic reward and, consequently, a small deviation from the prior policy.

2: Additional Learning-based Baselines

W2: Similarly, the work could provide results for Residual SAC in the MuJoCo environments, as the most direct comparison baseline in the literature. Currently, the only results are in GTS. Providing MuJoCo results should also give a better understanding about this direct comparison.

W3: Another baseline that is important is training the “prior policy” on the full task (prior reward + add-on reward). Since MPPI is just a local search around the prior policy, perhaps it is too regularized towards the prior task. By providing the prior policy on the full task, the work provides a potential upper bound on joint performance, which contextualizes the performance of Residual-MPPI (as a policy customization method) to standard full RL training. This also helps clarify the trade-off of retraining a policy from scratch (expensive, but potentially better asymptotic performance) OR performing policy optimization (cheap, but potentially suboptimal).

Q1: As mentioned in my previous points, how does Residual-MPPI performs in comparison to training a prior policy on the full task (initial task + add-on reward)?

Thank you for noting the additional learning-based baselines. We have included the results of the suggested baselines in MuJoCo in Table 1 in Section 4. Please find a complete discussion in Comment 1 in the general response.

3: Highlight the Sample-Efficiency

W4: Lastly, the work should probably highlight more the results in terms of sample efficiency. I understand that sample efficiency is key to policy customization, particularly in the physical world. The work just mentions that MPPI leverages 2100 laps while Residual-SAC leverage 80000 laps, which is a big difference. From my understanding, this is a big contribution of the method and should be highlighted. Some suggestions are either bringing a temple of sample analysis to the main paper or at least highlighting these results in the text.

Thank you for noting the key advantage of our proposed methods. To highlight the sample efficiency of Residual-MPPI, we have added the results of Residual-SAC trained on the same amount of rollouts in both MuJoCo and GTS. Please find the corresponding discussion and the additional experiments in Comment 2 in the general response and revised Table 1 and Table 2 in Section 4 and Section 5.

Dear Reviewer,

Thank you for your feedback! We were wondering if you have gotten a chance to go over our responses --- we have now provided clarifications and new experiment results on your concerns. We are happy to discuss further.

Thank you for your thoughtful feedback and valuable suggestions. We are glad you found the Residual-MPPI novel and that our experiments are solid in complex environments. Please find our responses to address your concerns regarding our work below.

1: Comparison between Residual-MPPI and Greedy-MPPI

W1.1: The empirical performance of the proposed method, Residual-MPPI, is quite similar to the heuristic modification of MPPI, i.e., Greedy-MPPI. On the main metrics, Full-task and Add-on Task, the reported performance of Residual-MPPI is approximately the same as that of Greedy-MPPI. The paper pointed out one difference on Ant Full Task. But it is unclear how the total reward was computed here and why this total reward should matter more than the add-on reward, given that Greedy-MPPI yields higher rewards than Residual-MPPI on add-on task. So please provide a more detailed analysis of the differences between Residual-MPPI and Greedy-MPPI, particularly in cases where their performance is similar.

W2.2: Also, comparing Greedy-MPPI against Residual-MPPI, it can be hard to say it is necessary to include $\log \pi$ as a reward in Residual-MPPI since the two perform very similarly on HalfCheetah, Swimmer, and Hopper.

Thank you for noting the important comparison between Residual-MPPI and Greedy-MPPI. Please find a thorough comparison and discussion in Comment 3 in the general response.

The biggest difference is that Greedy-MPPI optimizes the objective of a biased MDP $\mathcal{M}^\mathrm{add} = (\mathcal{X}, \mathcal{U}, r_R, p)$. As we discussed in Section 4.2, the Greedy-MPPI baseline relies solely on sampling from the prior to strike a balance between the basic and add-on tasks. In the first three tasks, where the add-on tasks are more "aligned" with the basic goal, like moving forward with less use of one joint (HalfCheetah, Swimmer) or jumping higher while jumping forward (Hopper), this biased characteristic does not lead to severe consequences and may even achieve better performance due to a simpler optimization objective and reduced reliance on biased dynamics.

Nevertheless, in tasks where the add-on reward is sparse or orthogonal to the basic reward, this approach would bring apparent suboptimality. In Ant, the agent is rewarded to make progress along the x-axis in the basic task. The add-on reward further encourages the agent to make progress along the y-axis. These two reward terms are orthogonal. Solely maximizing the progress along the y-axis would sacrifice the progress along the x-axis, which explains the performance gap of Greedy-MPPI. This issue is further exaggerated in the challenging GTS task. In GTS, the Greedy-MPPI only optimizes the trajectories by not allowing the policy to be off-course, where a straightforward local optimal but undesirable solution would be just staying still. Such a trivial policy would behave better on the add-on tasks (stay on course) but not on the full task (stay on course while maintaining racing superiority). As shown in the Greedy-MPPI results in Figure 3 in Section 5, this bias leads to severe failure in GTS, which directly demonstrates the superiority of Residual-MPPI over Greedy-MPPI.

The main advantage of Residual-MPPI lies in its objective, which is defined as a principled trade-off between the basic and add-on tasks. The other heuristical modifications of MPPI, including Greedy-MPPI, may work fairly well in some cases, but their performance cannot be assured. Instead, Residual-MPPI always performs equally well or even better, thanks to its theoretical ground.

I understand that the goal of policy customization is to jointly maximizing an add-on reward and the underlying reward of the prior policy. But my question was why the customised the policy need to maintain the same level of performance on the basic tasks?

Thank you for your thoughtful feedback and valuable suggestions. We are glad you found the Residual-MPPI novel and that our experiments are solid in complex environments. Please find our responses to address your concerns regarding our work below.

1: Additional Baselines

W1: The paper lacks SOTA baselines for policy customization. For instance, methods like those in [1, 2] demonstrate few-shot adaptability to new environments without an additional parameter training phase. A comparison with such method would strengthen the evaluation.

Thank you for noting these two related works. In our experiments, we have conducted comparisons under the planning scope with various SOTA approaches with different level of assmuptions. Please find a complete discussion in Comment 1 in the general response.

Regarding the two mentioned works, despite their few-shot adaptability to new environments, we believe they are not appropriate baselines for the policy customization task. Prompt-DT [1] designs a Decision-Transformer conditioned on sequences of states, actions, and rewards to capture information about the new task, achieving "few-shot" adaptability. Ceil [2] introduces an additional contextual variable $z$ and the corresponding contextual policy $\pi(a|s, z)$ to recover the expert data information via the learned $z^*$. Their goal is to construct a policy that can adapt to a new task with few demonstrations. In contrast, policy customization aims to customize a given prior policy towards jointly maximizing an add-on reward and the underlying reward of the prior policy, which is unknown in a priori. This task is not solely about adapting the prior policy to a new task but also ensuring that the customized policy inherits the characteristics of the prior policy. Moreover, both Prompt-DT and Ceil rely on expert demonstrations of the new task to enable adaptation, which is unavailable in our online policy customization setting. We have added these two works and the discussion above in the revised Section 6.

2: Comparison between Residual-MPPI and Greedy-MPPI

W2: The performance difference between Residual-MPPI and the comparison baseline, Greedy-MPPI, is not sufficiently analyzed. In Table 1, Residual-MPPI and Greedy-MPPI show very similar performance, yet the paper only briefly mentions on Line 430 that “Greedy-MPPI, as mentioned above, also leads to severe failure in complex GTS tasks” without further explanation. A more detailed analysis of Off-course Steps across different baselines would add clarity to the comparison.

Q1: I am curious about the performance difference caused by the inclusion of $\log \pi$. How does the trajectory change when the hyperparameter is increased or decreased?

Thank you for noting the important comparison between Residual-MPPI and Greedy-MPPI. The major difference between the Residual-MPPI and Greedy-MPPI is the inclusion of $\log \pi$, where the Greedy-MPPI can be seen as Residual-MPPI with $\omega^\prime = 0$.

Theoretically, $\omega^\prime$ represents the trade-off between the basic and add-on rewards. In GTS, larger $\omega^\prime$ should lead to a behavior closer to the prior policy with shorter lap time and more off-course steps. Conversely, an excessively small $\omega^\prime$ leads to a reward term that is overly biased towards the off-course penalty. As a result, the customized policy tends to deviate too much from the optimal racing line taken by GT Sophy, leading to suboptimal performance in both lap time and off-course steps. We have conducted the corresponding ablation studies in GTS, which follow our analysis here. Please find a thorough comparison and discussion in Comment 3 of the general response and the ablation results in Appendix F.2 and F.3 in the revised paper.

3: Approach for Expert-Data-Only Customization

Q2: When performing policy customization without a given reward, how can the proposed Residual-MPPI be applied if only expert data or instructions are provided instead?

Thank you for noting this question. In this work, we focus on applications where an add-on reward function is defined and study how to solve the policy customization problem in a sampling-efficient manner online for continuous control problems. It would be an interesting future direction to further extend Residual-MPPI to cases where only expert data or instructions are provided. If expert demonstration samples are provided, inverse RL can be adopted to infer the add-on reward from the demonstration. For instance, MEReQ [3] has explored the synergy of inverse RL and RQL. If instructions are provided, the add-on reward model can be learned from human or AI feedback [4]. We plan to explore these extensions of Residual-MPPI in our future work.

Reference

[1] Xu, Mengdi, et al. "Prompting decision transformer for few-shot policy generalization." international conference on machine learning. PMLR, 2022.

[2] Liu, Jinxin, et al. "Ceil: Generalized contextual imitation learning." Advances in Neural Information Processing Systems 36 (2023): 75491-75516.

[3] Chen Y, Tang C, Li C, et al. MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention[J]. arXiv preprint arXiv:2406.16258, 2024.

[4] Ziegler D M, Stiennon N, Wu J, et al. Fine-tuning language models from human preferences[J]. arXiv preprint arXiv:1909.08593, 2019.

Thank you for your thoughtful response.

Additional Baselines
I think that adding additional external baselines in the experiments is necessary to justify the performance improvements of the Residual-MPPI. Regarding Prompt-DT [1], I agree that additional datasets are required. However, as demonstrated in Figure 3 of [1], the dataset used for PromptDT does not necessarily need to be of expert-level quality.
In this study, the experiments involve a setup where add-on tasks are appended to a basic task. Therefore, even using few-shot data collected with a policy trained on the basic task, the dataset quality is expected to be comparable to the medium dataset used in [1]. Thus, it seems feasible to apply Prompt-DT in this context.

[1] Xu, Mengdi, et al. "Prompting decision transformer for few-shot policy generalization." international conference on machine learning. PMLR, 2022.

Thank you for the detailed response. I have adjusted the score based on the author's feedback.

2: Data-Efficiency

Some reviewers (R3, R4) mentioned the fair comparison by using the same amount of data in the learning-based baselines, especially in GTS. It is more solid to demonstrate the data-efficiency of the proposed method by showing what would happen if Residual-SAC was trained on the equivalent smaller amount of data, which is from ~80k laps to ~2k laps.

According to our experiments in GTS, the Residual-SAC policy does not intuitively transition gradually from being aggressive to conservative during training. Instead, Residual-SAC undergoes a significant performance drop, leading to failure, and slowly recovers from it during the training process. Even though we have applied various techniques (see Appendix D.4) to enhance the training stability and final performance, we still observed a large performance drop in the initial stage of the training.

In the revised manuscript, to highlight the outstanding data-efficiency of Residual-MPPI, we reported the performance of the Residual-SAC with the same amount of training samples used to train the dynamics model in Residual-MPPI. In GTS, Residual-SAC could not finish a lap at the checkpoint with 2k laps of training data (see Table 2, Section 5.2). The in-game evaluation video of the corresponding checkpoint can be found in the supplementary material. The MuJoCo results can be found in Table 1, Section 4.

3: Comparison between Greedy-MPPI and Residual-MPPI

Some reviewers (R1, R2, R4) raise their questions on the performance and analysis between the Greedy-MPPI and Residual-MPPI.

The biggest difference is that Greedy-MPPI optimizes the objective of a biased MDP $\mathcal{M}^\mathrm{add} = (\mathcal{X}, \mathcal{U}, r_R, p)$. As we discussed in Section 4.2, the Greedy-MPPI baseline relies solely on sampling from the prior to strike a balance between the basic and add-on tasks. In the first three tasks, where the add-on tasks are more "aligned" with the basic goal, like moving forward with less use of one joint (HalfCheetah, Swimmer) or jumping higher while jumping forward (Hopper), this biased characteristic does not lead to severe consequences and may even achieve better performance due to a simpler optimization objective and reduced reliance on biased dynamics.

Nevertheless, in tasks where the add-on reward is sparse or orthogonal to the basic reward, this approach would bring apparent suboptimality. In Ant, the agent is rewarded to make progress along the x-axis in the basic task. The add-on reward further encourages the agent to make progress along the y-axis. These two reward terms are orthogonal. Solely maximizing the progress along the y-axis would sacrifice the progress along the x-axis, which explains the performance gap of Greedy-MPPI. This issue is further exaggerated in the challenging GTS task. In GTS, the Greedy-MPPI only optimizes the trajectories by not allowing the policy to be off-course, where a straightforward local optimal but undesirable solution would be just staying still. Such a trivial policy would behave better on the add-on tasks (stay on course) but not on the full task (stay on course while maintaining racing superiority). As shown in the Greedy-MPPI results in Figure 3 in Section 5, this bias leads to severe failure in GTS, which directly demonstrates the superiority of Residual-MPPI over Greedy-MPPI.

The main advantage of Residual-MPPI lies in its objective, which is defined as a principled trade-off between the basic and add-on tasks. The other heuristical modifications of MPPI, including Greedy-MPPI, may work fairly well in some cases, but their performance cannot be assured. Instead, Residual-MPPI always performs equally well or even better, thanks to its theoretical ground.

Following R1's suggestions, we include an ablation study on $\omega^\prime$ to more clearly show the comparison against Greedy-MPPI. Greedy-MPPI can be seen as Residual-MPPI with $\omega^\prime = 0$, and $\omega^\prime$ represents the trade-off between the basic task and the add-on task, where larger $\omega^\prime$ should lead to a behavior closer to the prior policy. We have conducted the corresponding ablation studies in MuJoCo and GTS, which follow our analysis. Please find a more detailed discussion and the ablation results in Appendix F.2 and F.3 in the revised paper.