Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvement

Thank you very much for your positive, constructive feedback. We particularly appreciate your points regarding the discussion of guarantees and hyperparameter study. All specific responses are added below. Please do not hesitate to reach out for further open questions or comments.

Improvement Guarantees

“there is no discussion about why (or when) maximizing this is sufficient as an imitation learning objective.”

We worked on different paths towards providing a guarantee for improvement but did not obtain any strict ones. One challenge occurs around the use of a value function representing a different policy than the one that is improved. While some improvement guarantees exist for special cases [1], none matched the present case. Nevertheless, we put rigor into obtaining empirical evidence, a practice that is not uncommon in the field.

[1] Cheng et al., Fast policy learning through imitation and reinforcement.", 2018.

Stationarity during Self-Improvement

“During iterative self improvement, could it not happen that the agent learns to stay stationary in a region of the state space which was visited by the expert even though the expert was not stationary and covered a bigger region of the state-action space?”

This is an interesting comment. We don’t believe that stationarity during iterative self-improvement should be able to persist: If a policy would be stationary in a certain region this would increase the frequency these states are visited when rolling out the policy. In the next value learning cycle this would then decrease the value of these states which in turn would incentivise the policy to find other states. However, we believe this is a very valuable comment as this could explain the oscillations observed in figure 6 for Hopper and Walker: a policy decides to stay stationary in a region and then manages to resolve it again.

Hyperparameter Study for Temperature

“I would appreciate a hyperparameter study for lambda.”

We conducted a sensitivity analysis for both the mixing parameter and the temperature and added them to the appendix. These confirm that a mixing parameter of 0.5 (in line with the original SQIL) and a temperature of 1.0 are sensible parameters. However, the analysis also shows that there is a broad range of values for which positive improvement can be achieved.

Improved Plots

“The labels in figures are too small.“, “I would furthermore encourage the authors to provide additional plots with the absolute performance”, “It is not clear to me what exactly the subplots with the confidence intervals show (in figures 2, 3, 4, 5).”, “The order of the data points (to which iteration they belong) is a bit hard to make out in figure 6. “

Quality of plots is important and we are thankful for the reviewer’s comments. We increased font size for all plots. Provided further information regarding the confidence intervals and encoded iteration progress in Figures 6 and 7. The confidence intervals are taken across the spectrum of background data and summarize the performance of the algorithm.

We believe that the difference based plots make it easier to see whether actual improvement is achieved and improve comparability of methods via better resolution. However, we acknowledge that these types of plot are novel and thus add plots with absolute returns in the appendix.

Bootstrapping

“In line 334, the bootstrap value is introduced a bit ad hoc.”

The employed bootstrapping generates better results than no bootstrapping. Particularly for terminating tasks (all of the robomimic tasks) this was important as it increases the value on the last states. For cyclic tasks this had no large effect.

Qualitative Differences

“It would be interesting to discuss if there are qualitative differences between the behaviors learned by the VfO and baseline agents.“

It would indeed be interesting to see if there are qualitative differences between VfO which draws on demonstrations and AWR which draws on rewards, as these are two different sources of information. However during experimentation we did not observe any qualitative differences for the D4RL tasks, maybe because these tasks tend to be solved using very fast motions.

Typos

Thank you for mentioning mistakes and inaccuracies, we fixed them.

Thank you for the additional experiments, improvements to the presentation, and clarifications!

Stationarity during Self-improvement

I understand your point about the agent spending more time around a state leading to a lower value estimate. I agree that this should automatically penalize stationary behavior that does not correspond to the expert behavior. You raised an interesting point about the oscillations in Figure 6. Perhaps reducing the amount of new data being collected over iterations could mitigate this.

Hyperparameter Study for Temperature

Thank you for adding the sensitivity analysis for lambda and alpha! It indeed looks like the choice 1.0 and 0.5 should provide good results (except for Hopper perhaps).

Improved Plots

Thank you for improving the readability of the plots, and providing additional information on the intervals of the box plots. I agree that the plots showing the difference in performance are more interesting. It is nevertheless a good addition to have the absolute values in the appendix.

My other questions and concerns were sufficiently addressed. Thank you!

Thank you for your response. We are happy to hear that your concerns were sufficiently addressed and we hope this could improve your confidence in your assessment.

Stationarity during Self-Improvement

Your further comment regarding the amount of generated data is interesting and raises the question of how to compose the background data during iterations: Currently 1000 simulated rollouts are produced from the weights of the last self-improvement iteration and used as the sole source of background data for the next iteration (all previous data is discarded). However, experimenting with self-improvement cycles which leverage older data would be very interesting and could also help attenuating oscillations. This may however require an offline RL backbone that better supports off-policy learning (maybe importance weighting could be enough).

Thank you very much for your knowledgeable feedback. We particularly appreciate the mentioned connection to optimal transport and your point regarding the use of further reward functions and increasingly complex tasks. All specific responses are added below. Please do not hesitate to reach out for further open questions or comments.

Clarifying Novelty

“I am concerned about the novelty of the approach:”

With respect to the method, the novelty lies in how to use imitation learning-based rewards (such as SQIL or ORIL) in the context of imitation learning from observations (IfO) where no actions are available. We show how to do such an adaptation and how it performs on action-free demonstration, something that has not been done for the mentioned rewards. Further, we sketch a path towards iterative self-improvement, and successfully show how VfO can be applied in such a setting where other algorithms fail. Importantly, we also investigate the lack of representativeness of previous offline benchmarks and propose a novel offline evaluation recipe that is more predictive of iterative self-improvement performance. Our findings are very relevant for the research community as they put previous results under a new perspective and provide improved evaluation and baseline for future research.

Further Reward Function

“Consider experimenting with a wider variety of reward functions (as baselines) “

We appreciate the reviewers input and the additional references, we will include them in our revision. We also very much agree that further imitation learning-based rewards could be employed. However, we would want to emphasize generality-requirements of the employed reward, e.g. rewards that rely on domain knowledge or rewards that need action annotations would not fit into our paradigm. However, after looking at the paper, the optimal transport based reward would seem to fulfill this requirement and we would be keen to try it out as part of future work.

More Challenging Problem

“it would be valuable to see evaluations on more challenging problems, such as those with longer time horizons or real-world applications.”

Imitation learning from observations is a young field with many open questions regarding its application and evaluation. We believe that we have made an important contribution by looking into its applicability to self-improvement and providing a more targeted evaluation recipe. By resorting to well-established simulation based tasks we made this paradigm shift tractable for us. While reporting simulation only results is common practice in the field (of all baselines only DILO shows a real-world experiment), we acknowledge the importance of real world experiments and application on more complicated tasks such as the mentioned D3IL benchmark. We will thus focus next on collecting a real world based offline proxy and will also move towards multi-task learning in order to scale data and model sizes.

Thank you very much for your thoughtful feedback. We particularly appreciate your points regarding improved analysis regarding different data configurations and bimodal performance. All specific responses are added below. Please do not hesitate to reach out for further open questions or comments.

Possibly Good Background Actions

“The assumption that background actions have zero reward may overlook potentially valuable information.” “Does the assumption that the background contains actions that should receive a reward of 0 risk losing potentially informative action sequences?”

Good background data does not represent a problem. On the contrary, with higher returns on the background data the performance of the policy improves. Usually beyond the performance of the background data itself. Also, in the extreme case where the background data exhibits expert performance we do not usually observe degradation (except maybe for Hopper which in our past RL experience can get stuck with suboptimal behaviors). One explanation for this is that our policy only learns from background data. But the value is learned on both expert and background and by assigning a higher reward to the expert data we automatically also increase the value of background states that are similar to expert states. These observations are also in line with the original SQIL paper.

Sensitivity Analysis

“I’m curious about the results of a sensitivity analysis of algorithm parameters to better guide practitioners on how they should utilize VfO.” “How does the value of mixing parameter alpha affect the training of the policies?”

This is a good comment and we provide a sensitivity analysis for both the mixing parameter and the temperature. These confirm that a mixing parameter of 0.5 (in line with the original SQIL) and a temperature of 1.0 are sensible parameters. However, the analysis also shows that there is a broad range of values for which positive improvement can be achieved.

Clarification About Expert vs Background Data

“I’m unsure about how compelling the paradigm of VfO is in practice.“

The notion of Expert vs Background data and how we envision these to scale might not have been clearly communicated. We have further clarified both terms in the paper.

Expert data provides the “what”, i.e. indicates what should be done. This is the data we envision to scale to web-scale data. In the present work we use single tasks, but extending this to multiple tasks should be possible without significant adaptation.

Background data: provides the “how”, i.e. an understanding of dynamics (relationship between action and state). This is embodiment specific and we envision it to be collected in a self-improvement cycle to enable an agent to learn any task contained in the expert data.

Similarity of Expert and Background Data

“How similar does the background data need to be to the desired task?”

The data collected with the different BC policies represents an ablation over background data quality and provides insights into the question of similarity between expert and background data. The background data with low returns has little overlap, the background data with high returns are likely to cover expert data. We can show that across this wide spectrum of background data we are able to consistently attain performance improvements.

Bad Performance of VfO on Bimodal Data

“In the bimodal task, why did VfO fail to achieve improvement beyond simple BC?”

This is something we were wondering about as well. One possible explanation is the importance of picking the right action in a bifurcation. In the bimodal setting expert and background data are likely to be mostly distinct. It would thus be important to pick the right action once data bifurcates (after which simple BC would be enough). However, VfO seems to struggle doing so. This could be because the weighted policy regression is not strong enough in those bifurcation states which is likely caused by non-discriminative values and high temperature parameters. However, temperature can not be decreased sufficiently without incurring instabilities.

Furthermore, note that experiments are in a low data regime, which is more sensitive to architectural decisions and where inductive biases play an important role. So it is definitely possible that another policy architecture would improve results. But also, these effects may just disappear when increasing the amount of data.

Real World Application

“ or learning robotics policies in the real world”

Real world experiments can be extremely time consuming and given that we are moving in a young field subject to high uncertainty we think that short experimental cycles are of high importance. Thus we mainly focussed on simulated environments, which is also in line with most prior work (from the implemented baselines only DILO shows a real world experiment). However, we agree that frequently testing algorithms in the real world is very important. Given that we have found a good offline proxy for self-improvement, we believe the corresponding data should effectively be collected as a next step.

Thank you very much for your in-depth feedback. We particularly appreciate your points regarding lack of theoretical guarantees, additional context regarding data, further experiments, and explanations regarding scaling. All specific responses are added below (split into 3 comments). Please do not hesitate to reach out for further open questions or comments.

Improved Writing Clarity

“There are a few typos which can be addressed in a revised print.”

We thoroughly proofread the paper, fixed typos, resolved vague statements, reduced redundancy, and improved motivation.

Lack of Theoretical Guarantees

“It would be nice to see rigorous study grounded in theory regarding policy improvement ?”

We worked on various ideas to provide theoretical guarantees for improvement, however this turned out to be challenging. A main challenge is that we use the value of the mixture policy to improve the target policy. While there are special cases where guarantees can be obtained [1] no such guarantee can directly be applied here. Nevertheless, we put rigor into obtaining empirical evidence, a practice that is not uncommon in the field.

[1] Cheng et al., Fast policy learning through imitation and reinforcement, 2018.

Surpassing Expert

“Can the policy do better than the expert demonstrations, if yes, in what settings ?”

The primary goal is to imitate the expert as there is no other source of information on what to do. The only effect we intend to leverage in the future is generalization by training on a large dataset of expert demonstrations covering a broad spectrum of tasks. One could consider combinations of imitation and reinforcement learning with external reward signals to explore this direction further but it is out of the scope of the current submission. Further, external reward may not be that amenable to scaling.

Comparing against AWR

“it would be nice to see some equations and proofs on cases where this method meets the performance of Advantage Weighted Regression”

A theoretical comparison against AWR is difficult. The reason being that both employ a different driving source of information: AWR uses reward annotations (which VfO has no access to) while VfO employs expert demonstrations (which AWR has no access to). So in a sense this also compares said sources of information and scenarios can be constructed where each source is better than the other. For instance in a scenario with dense rewards but scarce expert demonstration AWR is likely to be better. Inversely, a scenario where expert demonstrations are plenty but reward annotations are sparse VfO could be better. This can partly be observed for the robomimic dataset where reward annotations are sparse. It would indeed be interesting to formalize these situations mathematically, but this is a difficult challenge which we leave for future work.

Data Ablations

“Rigorous study and Ablations for the mixture of datasets is missing”

Studying the property of the data is indeed very relevant. In comparison to other work, we already provide quite a lot of information by clearly taking the quality of the background data into account and by providing the return distribution across the spectrum of background data for an example task. Regarding the maximum performance that could be achieved with the background data, we believe that the return of the data itself, the performance of BC, and the performance of AWR cover this question. Maybe higher performance could be achieved by using privileged information or domain knowledge (e.g. the task being solved in a final state). However, we believe that such information may not be applicable at larger scales.

Regarding the expert being out-of-distribution, examples are provided for this via the lower quality background data, where we can observe that only little improvement is achieved. This however strongly depends on how improvement is measured.

In the extreme case where not even the initial state distributions overlap improvement may not be possible. In such cases exploration would play an important role. However, we envision a scenario where sufficient unlabeled expert demonstrations are available. Partly because we believe that exploration without prior knowledge is ill-posed. Such prior knowledge could originate from data, which brings us back to the scenario at hand.

Impacted Performance of DILO/SMODICE

“Explanations on why SMODICE[4], DILO[5] perform worse”

This question has intrigued us for a while. Our current hypothesis is that the Bellman error employed in both algorithms has a weak signal. This is similar to what has been observed in Bellman Residual Gradient algorithms [1]. E.g. for AWR, if we remove target network and stop gradient on the subsequent value, the value learning remains perfectly stable and even reaches a lower TD-error, however the policy derived thereof will have a significantly worse performance. Since SMODICE and our version of DILO also employ this form of free optimization (no alteration of gradients, no target network) we believe that this could be a reason for the reduced performance. Also, note that the published version of DILO employs orthogonal gradient descent which could behave differently, however, the code was not publicly available at submission and our version with unaltered gradients worked better for us.

[1] Baird L., Residual algorithms: Reinforcement learning with function approximation, 1995.

Single Iteration

“In the algorithm, is it necessary to learn value function and policy in a single iteration ? “

Not sure we understand the comment regarding single iteration? Could you please clarify.

Expert Policy in Equation 2

“In equation 2, it is not clear how do you have access to the expert policy, pi_E(a|s).”

Equation 2 refers to the virtual policy that the learned value function corresponds to. The algorithm does not need it explicitly, but rather has access to samples thereof. We restructured this paragraph to make this clearer.

Value Function on Expert Only

“Have any experiments been conducted where the value function is learnt just on the expert data and then it is used to learn a policy on background datasets?”

The value could be trained on the expert data only and this would be equivalent to setting the mixing parameter alpha to 0. However, we found that alpha = 0.5 works well for most cases. We added an ablation of the alpha parameter in the appendix and can observe that training the value on expert data only strongly decreases performance. This is likely due to the fact that we have little expert data and we might observe different results otherwise.

Expert Subset of Background

“It could also be possible that if a similar set of states are encountered by the background datasets, it could confuse the value function since all the background states are given a reward of zero ? “

A form of this problem is also encountered in SQIL: as the samples (background data) get closer to the expert the effect of the expert decays. However, according to the authors this did not represent a significant problem. Intuitively, if the background data is close to the expert less change to the policy is required. A similar effect is also encountered with GAIL at convergence.

Comparison Against (Adapted) SQIL

“Although the Value function is appropriate for a setting without demonstrations, it would also be nice to see how well the algorithm performs when action annotations are available with SQIL. “

If we understand right, you are asking for a comparison where privileged expert actions are used. We thus apply the AWR style policy improvement across background and expert action in order to obtain an offline version of the original SQIL paper. We ran these experiments and added them to the appendix. However, given the little amount of expert demonstration we had to increase the mixing parameter alpha to get good performance. Overall performance is similar to VfO, likely because there is not that much expert data.

About Scaling and Autonomous Driving

“With reference to the title and the mentions regarding scaling in Section I” “Studies on the number of episodes versus the policy performance and proposing any scaling laws ?” “(Performance on Carla Leaderboard[1], or using Offline Autonomous Driving Datasets”

Scaling is an important theme and the present work attempts to lay the foundation for scaling in the future. We envision scaling to take place in the expert data which could incorporate more and more tasks and domains and finally attain web-scale dimensions. Autonomous driving would indeed be an interesting domain: While this usually has action annotations, learning from observations only may be an effective way to use data across different embodiments. However, this is out of scope of the current work and it would require immense additional computational resources. Also, we believe that a natural next step would be to investigate the multi-task setup where indeed scaling laws could be explored since we would move beyond the low data regime.

For each gradient step in the algorithm, is it necessary to update both the value function and the policy simultaneously? One extreme approach would involve fully learning the value function first before optimizing the policy. My question on this topic has been addressed through the discussion on value functions trained exclusively on expert data, which provided some clarity.

I appreciate the authors’ responses to my questions and their effort in incorporating the suggested updates into the manuscript. While I recognize the challenges of conducting real-world experiments and addressing scaling concerns, incorporating such aspects in future work would significantly enhance the practical contributions of this research. Furthermore, deeper exploration of how the concepts are grounded in theory could broaden the work's applicability and impact.

Overall, the revisions and responses have improved the quality of the paper from the initial version. I commend the authors for their efforts, and based on the updates, I am raising my initial score.

Thank you for your response. We are happy to hear you think the paper has considerably improved. Nevertheless, we would like to better understand why your updated rating remains under the acceptance threshold. Apart from the heuristic approach and missing real-world experiments, is there anything else that would improve the paper in your opinion? We do believe that the application of IfO to self-improvement as well as the insights into the data dependencies are of very high value for the community.

Optimization Schedule

Regarding the training regime, we now understand your original question was about how gradient steps for the value and policy are being coordinated. Currently, it is indeed as you describe, at every training step both value and policy weights are updated concurrently and you are correct that this could be done differently. We initially experimented with a sequential optimization, where first the value would be trained and subsequently the policy. However, this led to slightly worse performance. This is likely due to some optimisation dynamics and is something we want to revisit when we move to the higher data regime.

We very much appreciate your further comments and will provide answers in the following paragraphs (in two OpenReview comments). We have added clarifications regarding these points to the paper as well.

Clarifications for Difference Plots

We believe there is a misunderstanding regarding the difference plots. The x-axis represents the average return in the background data. The returns being the sum of ground-truth rewards over all transitions in a trajectory. For every trajectory in the background data we thus sum rewards for all transitions and then average across trajectories.

Once a policy is trained for a specific background dataset, we compute the average return of the policy in an analogous manner but with simulated trajectories obtained with that policy.

Say that for a background dataset with an average return of 2000, we trained a policy that generates an average return of 2500. We plot the difference of 500 against 2000. This has two advantages: we can clearly see whether the policy improves over the background data (if the difference is > 0) and we can report multiple runs across different background datasets in the same plot without losing resolution.

Clarifications on Background Data for SIBench

For every SIBench experiment the background data originates from a single BC policy only. The data generated from the different BC policies is not mixed. So for every one of the 10 BC policies that are trained we obtain one corresponding background dataset containing 1000 trajectories. We originally looked into mixing data, but this led to a more complicated protocol. For all D4RL experiments the expert dataset is composed of 10 trajectories and the ratio of expert to background trajectories is thus 1:100.

Scalability

Using a concrete example to explain how we envision scaling is a good idea. Lets assume we want to train a policy using the Open-X Embodiment dataset without actions (not relying on action labels enables access to much more data in the long term). We would select all the datasets that have no suboptimal trajectories (which is the majority) and build our expert dataset thereof. Optionally, suboptimal data with action labels could be added to the background data. But if there is no such data, further data would inevitably have to be collected and this could be done using the suggested self-improvement scheme (in the extreme case starting from a random policy). In terms of expert-background ratio, that would bring us into a regime where expert data is abundant (but not very specific). How much background data needs to be collected in this regime is unclear and requires further investigation. As you might be alluding to, this could highlight the importance of being background data efficient.

Regarding the alpha parameter, there may indeed be some dependency on the ratio between the amount of expert and background data. However, as we have shown in our ablation, the range of alpha parameter yielding improvement is fairly wide and we are thus confident that it could be adapted to a different data ratio.

We acknowledge that there are a lot of open questions and variations regarding this approach to scaling in robotics, but so do all other paradigms (BC, prompting, RL). Still, we believe this to be an important direction to explore and we believe the current paper to contribute very valuable initial insights on which future research can be based.

Bad Performance of Baselines

All baselines (except AWR) show bad performance on SIBench and self-improvement. For BC we don’t expect improvement as there is no corrective signal. As mentioned in a previous response, for SMODICE and DILO, we believe there is a lack of signal with the Bellman residual like loss if not used in conjunction with stop gradient and target networks (there is a wider design space to explore and these could be adapted but also need further investigation).

For the D4RL self-improvement experiment, access to 10 expert trajectories was provided (VfO would not work without them either). However, the background data is only self-collected and this may indeed be the issue for SMODICE and DILO: there are no (diluted) expert trajectories in the background data.

We agree that the results raise questions. However, we went further than many previous works in our experiments and explored the impact of different types of data sources, including adding self-improvement and a corresponding proxy across a wide spectrum of background data. To a certain extent it is also natural that this raises further questions.

Comparing SIBench vs Bimodal results

Both observations are correct: for Bimodal high returns can be achieved with moderate background data. The reason for this is that a substantial amount of expert trajectories is diluted in the background data. Thus any algorithm that is able to extract them and copy their behaviour will yield high returns.

For SIBench this is more subtle as there may be no trajectories demonstrating expert behavior in the background which makes it much more difficult to obtain large improvements. However, this also makes it more representative of self-improvement. The 45 degree trend mainly indicates little improvement.

Rather than transferring information differently we would claim that Bimodal and SIBench represent different problems. Bimodal is more about filtering out non-expert data and then applying BC. On the other hand, SIBench is more about locally improving a policy much like RL attempts to do.

Thank you very much for your detailed, constructive feedback. We are glad you were able to follow it well and appreciate your points regarding potential for improved clarity on novelty and additional context. All specific responses are added below. Please do not hesitate to reach out for further open questions or comments.

Clarifying Novelty

“Novelty. This author’s contribution lies in showcasing that SQIL type methods can potentially learn even if not considering actions”

While the method is similar to SQIL, it is applied to a different problem setup for which both an algorithmic adaptation is necessary and for which there are no previous results. Further, we sketch a path for how to learn from observations without access to expert action via self-improvement, and successfully show how VfO can be applied in such a setting where other algorithms fail. Importantly, we also investigate the lack of representativeness of previous offline benchmarks and propose a novel offline evaluation recipe that is more predictive of iterative self-improvement performance. Our findings are very relevant for the research community as they put previous results under a new perspective and provide improved evaluation and baseline for future research.

Display Absolute Returns Including Expert Return

“The choice of plotting return differences vs background data returns, can be confusing.”, “The expert’s returns should be displayed in the plots along the background data.”

We believe the return differences are extremely valuable in order to clearly see whether the policy is able to improve w.r.t. to the employed background data. Also it improves resolution and allows better comparison of algorithms. Nevertheless, we agree that these plots are uncommon and we thus generate plots with absolute performance, including expert returns, in the appendix. Please note, for the difference based plots expert returns would be on a diagonal which could impact readability.

Explain BC Performance

“Why does simple BC seem to improve on the background data?”

This is indeed a peculiar behavior. We believe that this is a similar effect to what has been previously reported in generative modes [1]. In heterogeneous data it may be easier for the policy to focus on what is in common or more predictable. If this coincides with better actions then the policy will achieve higher returns. This effect is particularly strong for the bimodal data and will indeed also have a transient effect during training (we train all BC policies for 1e6 steps though).

[1] Zhang et al., Transcendence: Generative models can outperform the experts that train them, 2024.

Bad Demonstration Data

“What happens when a significant part of the demonstrated data is of very low returns?”

We are not entirely sure what is meant by “significant part of the demonstrated data”. If this refers to background data then such ablation is provided and we observe that we are able to achieve improvement across a wide spectrum of background data quality. If this only refers to the expert data then we do not intend to deal with this case. The goal is to imitate the expert data in all its aspects. The question is rather whether absolute return is a good metric to measure expert imitation.

Box-Tooth Pattern in Figures 6 and 7

“In figures 6 and 7 it can be hard to discern what is happening especially in the later parts of Hopper and Walker. What does this box-tooth behavior mean?”

The saw-tooth pattern is observed when consistent improvement is achieved during iterative self-improvement: the achieved performance of one iteration (y-axis) is used as base performance for the next iteration (x-axis), thus the projections onto the diagonal. A box-tooth pattern is observed when increase and decrease in performance alternate. This might be caused by oscillating effects. One hypothesis was provided by reviewer 5JyX which hinted that stationary regions might emerge temporarily.

I carefully went over the revisions and alterations to the paper. I also reconsidered the authors response to my initial comment about limited novelty. I believe that there could be value to this direction of self-improving VfO, especially in some applications in robotics, where issues of scale can arise; it is indeed not always possible to capture state-action data. I believe there is merit to this direction, more than a niche case of SQIL. I have raised my score to reflect this.

I would like to commend the authors for their improvements to the paper. It reads and motivates the problem better.

Thank you for the very detailed and constructive review. We appreciate your acknowledgement of both the importance of the problem and the novelty of the method. Answers to your comments follow (split in two comments due to length restrictions). Please do not hesitate to reach out for further open questions or comments.

Practical Motivation

“Insufficient Practical Motivation: The paper lacks clear examples of real-world scenarios where the specific data setting (...) is prevalent.”

We thank the reviewer for pointing out the importance of including examples of real world applications to our work. Shared embodiment devices like UMI are indeed a great point. Using VfO could circumvent the use of a policy interface: Given the demonstrations of the hand-held gripper, VfO could directly be used to collect data on a specific embodiment via self-improvement. Another application would be in settings with motion capture data, where VfO could again be applied to overcome the lack of action annotations. We elaborate on this in the paper and further clarify the long term vision where web-scale data is incorporated as part of the expert data and where an agent collects the background data in self-improvement cycles.

About Real World Experiments

“Absence of Real-World Experiments: The lack of real-robot experiments limits the demonstration of the method's effectiveness in practical settings”

We agree that frequently testing algorithms in the real world is very important. However, real world experiments can be extremely time consuming and given that we are moving in a young field subject to high uncertainty we think that short experimental cycles are of high importance. Thus we focus on simulated environments, which is also in line with most prior work (from the implemented baselines only DILO shows a real world experiment). Given that we have found a good offline proxy for self-improvement, we believe the corresponding data should effectively be collected as a next step.

Improved Writing Clarity and Related Work

“Clarity and Writing Quality: Several sentences are difficult to read due to approximative language” “Weakness in Related Work Section: Some citations are not well justified, and the connections to the proposed method are unclear. “

We agree that writing clarity could have been better. We thoroughly proofread the paper, fixed typos, resolved vague statements, reduced redundancy, added context to the related work (including some notes on self-improvement), and improved motivation.

Clarify Reported Results

“Unclear Experimental Results: It is specified that the reported results are from the training set (line 305).”

The reported results are the returns of simulated rollouts. These are continuously generated during training. In order to reduce noise we average across 5 seeds and across the checkpoints of the last 1e5 training steps. Also, we always use rollouts of the stochastic policy to accurately reflect data collection during self-improvement. We further include plots with absolute returns in the appendix.

Clear Takeaways

“Experimental Conclusions: What are the key takeaways from your experiments?”

Again, we agree that this could have been clearer. We refined the key questions posed at the beginning of the experimental section, clarified takeaways when discussing results, and summarized them again in the conclusion. There are as follows:

• There is a high sensitivity of performance w.r.t. the action-labeled background data.
• VfO performs well on the offline self-improvement benchmark approaching the method using oracle reward. Baselines such as SMODICE and DILO perform well on bimodal background data mixing expert data with very low return data.
• There is a correlation between performance on the offline self-improvement benchmark and iterative self-improvement experiments, which means that it can effectively be used as proxy for self-improvement.
• The offline self-improvement benchmark with images is challenging and VfO only achieves partial improvement.

Explain Performance Difference between VfO-bin and VfO-disc

“Discriminator Performance: In Figure 2, what accounts for the large difference between the discriminator and binary results in the walker experiment?”

One possible explanation lies in how VfO-bin and VfO-disc transfer information from expert to background data: VfO-disc transfers both on a reward and value level, but VfO-bin only uses the value function. During policy learning, VfO-bin has thus no immediate reward. Tasks where this may be important could be impacted. This could explain why VfO-disc is better on cyclic tasks, such as D4RL, where immediate reward could be more important than for non-cyclic tasks, such as robomimic. Why this is particularly large for Walker2D is unclear and requires further investigation.

Further Baselines for Self-Improvement

“Lack of Baseline Comparisons in Self-Improvement Experiments.”

Self-improvement experiments are highly costly due to their sequential nature (which is the motivation for an offline self-improvement proxy), explaining the reduced number of experiments. We have however added both DILO and BCO as further baselines to the self-improvement experiments. These confirm the previously observed correlation between offline self-improvement proxy and self-improvement.

Thank you for your further comment. What you describe is correct: training only uses pre-defined expert and background datasets.

In parallel to this an evaluation routine generates rollouts using snapshots of the trained weights at frequent intervals and computes the obtained returns. This generated data is not used for training.

The simulated returns are sensitive w.r.t. to training seed, rollout seed, and when the model weight snapshot is taken. We thus average across all three by using 5 training seeds, generating 10 rollouts per snapshot, and averaging across snapshots of the last 1e5 training steps.

We hope this helps to clarify the experimental evaluation and we will further clarify this in the paper. Please feel free to reach out with any further questions.

About the reported results. I am misunderstanding something: You should be training on two offline datasets: an expert dataset and a background dataset. My understanding was that you used preexisting expert datasets and pre-generated examples to create the background dataset. In my understanding, training on this data mix does not require rollouts so why are they "continuously generated during training" ?

Regardless, it seems that you are averaging together the results of different models. The model changed because it is being trained between different rollouts.

Thank you for the valuable discussion. We appreciate that evaluation and model selection remain big challenges in behavior learning. This also further highlights the importance of real world experiments where ultimately model weights have to be selected and applied on a real embodiment. We will do our best to collect real world data as part of future work to further advance the practical relevance of imitation learning from observations.

This is a justified comment. However, we want to point out that averaging across training seeds for evaluation is a very established practice. E.g., almost all RL papers report averages across seeds (e.g. [1, 2]) and the same practical difficulty of only being able to deploy a single model arises. Nevertheless, comparison between different algorithms is commonly performed in such a way. Furthermore, averaging across temporal model snapshots once steady state has been attained is very similar to averaging across training seeds.

But in contrast to the established practice we did not report standard deviations. We added plots visualizing the standard deviation in the appendix and plan to fully integrate them as part of the final revision. For simplicity we only plot AWR against VfO-disc and we can observe that a) the obtained standard deviation for VfO-disc is similar to that of AWR and b) most obtained model weights are able to achieve improvement (except for hopper as previously observed).

Finally, we also want to highlight that our self-improvement experiments effectively only use one set of model weights to generate rollouts for the next iteration. And this further confirms that, although subject to stochasticity, improvement can be achieved with the suggested method.

[1] Schulman et al. "Proximal policy optimization algorithms." 2017.

[2] Haarnoja et al. "Soft actor-critic algorithms and applications." 2018.

We have reconsidered your comments and looked into an updated evaluation procedure: Only the final model weights from each training run are evaluated, using 1000 simulated rollouts each. We keep training with 5 different seeds to capture the distribution of average returns and report mean and standard deviation across training seeds.

We generated plots for these new values and added them to appendix H for the time being (Figure 18 is new, Figure 17 is with 10 rollouts over a window of model weights). Compared to the previous evaluation, the variance of the returns decreased, and this is due to generating more rollouts per model weights (the environments exhibit high stochasticity). With this reduced variance we can now claim that most VfO model weights attain good performance improvement.

Thank you again for your valuable feedback. We addressed a majority of the comments thus significantly improving the paper, including a clearer motivation, better writing, and a good batch of further experiments. Given that the end of the rebuttal period is approaching quickly, are there any further questions or comments?

Dear reviewers, thank you again for your time and effort in reviewing our paper. Your comments were very valuable in improving the presentation of the paper with a clearer motivation, further explanations, and better plots. The experimental section was further strengthened with additional results confirming the applicability of IfO to self-improvement.

We understand that some of you would like to see further theoretical grounding and more real world experiments. However, we believe that their absence is not uncommon in the field. Furthermore, given that we are exploring a direction with high uncertainty, where a majority of baselines can not achieve improvement, it was essential to investigate a setup with fast experimental cycles. This enabled us to generate novel insights around the applicability of IfO to self-improvement and to introduce VfO as an effective baselines, paving the way for future research in the community.

Having said that we would like to ask you if there are any remaining questions or comments we could address during the remainder of the discussion period?