Video2Demo: Grounding Videos in State-Action Demonstrations

We thank the reviewer for their feedback, the comparison with newer vision model APIs and appreciate the questions raised about the the evaluation of our pipeline

Questions and Weaknesses

Clarification on Prior Work:

We thank the reviewer for their suggestion to consider ProgPrompt[5], Voyager[6] and Voxposer[7] as related works. In our next revision, we will duly acknowledge them.

We wish to emphasize that Video2Demo serves a different purpose from the mentioned works.

1. ProgPrompt is an LLM-based task planner translating language instructions to code
2. Voxposer defines affordances and rewards through code for motion planning
3. Voyager builds a skill library for playing Minecraft using LLMs.

Video2Demo however, is not a planner. We focus on extracting state-action predicates through vision-language demonstrations. We use Demo2Code[8] as an example of how Video2Demo’s output can be used in downstream task planners like [5-7]. We aim to explore this in future work with more embodied planners.

Other VLMs and LLMs:

Our choice for the generative LLM was straightforward: there are currently no other LLMs that match the context capacity, instruction following and logical coherence capabilities that GPT-4 has [1]

Our generative VLM choice was based on recent literature surveys, underlying simplicity of the architecture, and the choice of pre-trained models in LLaVA [2,3]. Due to time and cost limitations, we could not ablate the choice of the VLMs. However, the method is plug and play for both the choice of LLM and VLM, similar to the Socratic Model approach.

Finally, OpenAI has recently enabled GPT4-vision as part of their API[4]. For future work, we are interested in evaluating this in our setting as:

1. An alternative generative VLM in the Video2Demo pipeline, where we can expect better visual grounding compared to LLaVA.
2. A competing method to Video2Demo

On execution-based evaluation

In the global response, we elaborate on how we primarily focus on the problem of extracting rich information from offline vision-language demonstrations. In future work, we plan to evaluate Video2Demo’s generated state-action with task and motion planners [5,11] in simulators [9-10].

References -

[1] OpenAI, R. "GPT-4 technical report." arXiv (2023): 2303-08774.

[2] Liu, Haotian, et al. "Visual instruction tuning." arXiv preprint arXiv:2304.08485 (2023).

[3] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

[4] OpenAI: New models and developer products announced at DevDay. https://openai.com/blog/new-models-and-developer-products-announced-at-devday, 2023

[5] Singh, Ishika, et al. "Progprompt: Generating situated robot task plans using large language models." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

[6] Huang, Wenlong, et al. "Voxposer: Composable 3d value maps for robotic manipulation with language models." arXiv preprint arXiv:2307.05973 (2023).

[7] Wang, Guanzhi, et al. "Voyager: An open-ended embodied agent with large language models." arXiv preprint arXiv:2305.16291 (2023).

[8] Wang, Huaxiaoyue, et al. "Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought." arXiv preprint arXiv:2305.16744 (2023).

[9] Eric Kolve et al. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv:1712.05474, 2022

[10] Todorov, Emanuel, Tom Erez, and Yuval Tassa. "Mujoco: A physics engine for model-based control." 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012.

[11] Yu, Wenhao, et al. "Language to Rewards for Robotic Skill Synthesis." arXiv:2306.08647, 2023

We thank the reviewer for a thorough review of our paper, enumerating its strongest points, asking us clarifying questions and providing new ways to test and evaluate our approach.

Questions:

Q1: Can it scale to beyond egocentric videos in EPIC kitchen to third-person tutorial videos? How would the prompt change, especially in the phase 2 where the prompts and state-action predicates seems to be centered on the human in the videos?

While we did not evaluate our system on third-person videos in this paper, Video2Demo can easily be applied to generate state and action predicates for third-person tutorial videos.

Video2Demo's VLM and LLM prompts are mostly domain-independent. To gain better performance in new environments, one only needs to modify the following:

• Domain information in the LLM's prompt, which could include
  • Overall setting or theme of the vision language demonstrations (e.g. house chores)
  • Examples of the state predicates
  • Examples of the action predicates
  • Domain-specific rules (e.g. we forbid the LLM from directly referring to the human in its state and action predicates).
• Domain information in the VLM
  • Whether the images are first-person view or third-person view

Q2: How does ChrF compare to other code generation metrics?

We thank the reviewer for suggesting CodeBERTscore as another code evaluation metric. We will integrate this in the final version of the paper and strengthen our evaluation of the generated code.

Does the generated code with high ChrF score correlate with human preference?

Our qualitative testing showed a correlation between generalized code with loops and relevant conditions, and a high ChrF score

How much of the generated code follow required syntax and physical feasibility to execute successfully on a simulator?

As we provide in the paper, the results from states and actions generated by baselines and Video2Demo imply that the demonstrations provided by our perception system will be noisy. However, we qualitatively show that the code generated is close to expert-generated code. In addition, we note that:

1. Demo2Code is robust to these noisy states and actions, and manages to capture the user’s preference
2. The code generated is valid Python code without any undefined functions As also mentioned in our global response, in future work, we will test the generated code for validity and feasibility in simulators like AI2-Thor [2] and VirtualHome [3].

Weaknesses:

On more Evaluations

In our global response, we expand more on closing the loop with Video2Demo and evaluate using simulators [1-3]. We thank the reviewer for suggesting the ALFRED [1] benchmark to test our method. Since they also use high-level actions and egocentric vision, it provides the right testing ground. Our aim with this paper is to motivate the need for perception and generation of state-action sequences from task videos, and in future work, we hope to evaluate Video2Demo on the simulator.

On the reliance on prompt engineering

Most section of Video2Demo's prompt is domain-independent, so it can be applied to other activity videos. For domains significantly different EPIC-Kitchens, the only parts of the prompts that need to be changed are about the domain. Please refer to our answer to your 1st question for details.

References -

[1] Mohit Shridhar, et al. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. arXiv:1912.01734, 2020

[2] Eric Kolve et al. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv:1712.05474, 202

[3] Xavier Puig et al. VirtualHome: Simulating Household Activities via Programs. arXiv:1806.07011, 2018

[4] Zhou et al, 2023. CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code. https://arxiv.org/abs/2208.03133

We thank the reviewer for their comprehensive review commending the strengths of the paper as well as constructive criticism about the contributions.

Questions

Q1: How big is the "Human-annotated state and action predicates" in claimed contribution 2? I think this is a nice contribution but from what I read in the paper, this doesn't seem to be very big. Could you clarify?

We annotate 20 videos from the EPIC-Kitchens validation dataset, where each video is 5 minutes in length on average

• an average of 3 predicates per frame
• 1 action per frame, which is already provided by the dataset.
• 1027 image frames in total

We also partially annotate 5 videos to only focus on the dish-washing segment of the videos. All annotations are double-checked by a second person to ensure quality.

We estimated that the time required for manual labeling per video is 2 hours, resulting in a total estimated annotation time of 50 hours. Thus, we built a user-friendly web-based annotation tool that allows users to cross-reference frames and autocomplete predicates based on past annotation and reduces annotation time.

Q2: Could you clearly define "structure and temporal consistency" in the abstract?

The term structure and temporal consistency refers to the importance of understanding past actions and the relevance of objects in past timesteps (i.e. state of the environment) to predict state-actions for the current timestep. Referring to past state-actions to infer the current state and action is evident during video annotations as human annotators as well. For example, if a cook is seen holding a pan in their hand at some time t:

• If the pan was brought over to the stove-top from elsewhere, it is likely to be put down in the next few frames
• If the pan was not in hand earlier, it was picked up right before time t.
• The pan, as part of the environment, may stay relevant in future actions in the demonstration.

If we use a VLM to generate captions without context, it may fail to recognize what is going on in the scene. However, responses from VLMs along with GPT-4’s ability to reason what states precede actions and what actions follow other actions enhances the consistency in actions and the state of the observed environment.

Weaknesses

On the contribution in the paper

The contribution of the paper is not just in the method of LLM/VLM interaction, but also in solving the problem of extracting symbolic states and actions from long vision language demonstration. The baseline result shows that

• The captions that VLM generates are not sufficient in extracting salient states about objects in the frame. (see SM-Caption)
• Despite including history, a predefined, fixed interaction between the LLM and the VLM is also not sufficient in accurately extracting salient information because the LLM has no way to query VLM about inconsistencies in its answer. (see SM-Fixed-Q)

Naive application of LLM and VLM cannot effectively solve this problem. Video2Demo, despite its simplicity, is able to significantly outperform the baselines by extracting symbolic states and actions with higher quality.

Clarification on Evaluation

We use GPT-3.5 to evaluate our method’s generated state and action predicates as they come from:

1. open-vocabulary based predicate generation and
2. expert annotations from different humans meaning that it's possible two predicates match semantically without matching exactly in text string. Since it is taxing to enumerate all possible states in a complex environment like EPIC-Kitchens, we allow GPT-4 to come up with its own predicates. We argue that in this setting a soft match in the semantic space is fair to evaluate the model. In addition to this, a natural-language planner should be able to generate similar plans for semantically similar predicates.

In order to make the evaluation more robust, we ask GPT-3.5 to give reasoning before giving an answer (of whether two predicates are “equal” in semantic space)

Text evaluations are often done using other AI models, especially more recently with reliable models like GPT3.5 or GPT4. [1,2]

Clarification on Outputs

In the global response under “Representation of States and Actions”, we clarify the choice of how we represent states and actions. While they may be text-based, they are structured in predicate format, and a majority of the most commonly used state predicates and actions are provided to GPT-4 as part of its prompt. Examples of predicates:

• state: is_inside(<A>, <B>), human_is_holding(<A>), is_on_top_of(<A>, <B>), etc.
• action: pick_up(<A>), open(<A>), turn_off(<A>), etc.

Since it is taxing to design a comprehensive pddl-like specification for every possible predicate in a complex setting like EPIC-Kitchens, we allow the model to come up with its own predicates similar to the examples. This can be restricted in the prompt if Video2Demo is to be used in a simpler enumerable setting.

Thank you for the clarifications.

I just went through the response written by peer reviewers, as well as the author's response. Unfortunately, I don't think the author's rebuttal would address my concerns, without heavy modifications to the project. I reiterate the need for stronger technical contribution for this paper to be accepted at any top ML conference.

We thank the reviewer for their constructive criticism, and we acknowledge their concerns about our paper’s motivation, evaluation, and practical deployment.

Questions:

Q1: What was the cost of running these experiments?

The costs for these experiments are mainly the cost of OpenAI’s GPT-4 API usage.

• The total cost was approximately $2200.
• Each demonstration costs about $15.

We ran approximately 6 experiments on each of the 26 videos, encompassing several trials with the baselines, Video2Demo, ablations, and qualitative experiments for code generation.

Lower costs for GPT-4 were recently announced by OpenAI [6], and we can accordingly expect the cost of this method to be reduced in the future.

Q2:Do you have one state for every image?

We do not generate states for every frame in the video because two consecutive frames are highly similar. Instead, Video2Demo generates a set of state predicates and one action for each action segment specified in the EPIC-Kitchens data.

Weaknesses

On insufficient evaluation

In quantitative evaluation, we focus on the quality of the generated state and action predicates. In Table 1 and Figure 3 in our paper, we evaluated the approaches on 19 videos that have human state-action annotation. We show how our generated predicates are closest to human annotations compared to other baselines across all kitchen activity categories.

Our evaluation for code generation on 7 dishwashing videos is a qualitative example of a downstream application of Video2Demo. We show that [2], which takes text-based state-action trajectories as input, can generate expert-like code when it uses the state-actions generated by Video2Demo.

Clarification on the choice of action space

In global response, we clarify why we picked a high-level action space. We would like to add that despite the high-level resolution of this task, predicting these high-level actions is still not easy, as is evident from the accuracy and recall scores from the baseline VLM-based methods and even Video2Demo.

While we acknowledge that code generated by task planners may not be feasible or affordable by the embodiment in that environment, we note that it is not the focus of our work. We aim to be able to generate states and actions that can increase the success rate for language-based planners compared to only providing language instructions. Qualitatively, we see that our demonstrations allow a code-generation pipeline based on [1-2] to generate expert-like code with loops and conditions that are non-trivial to infer from simple language instructions.

Clarification on Motivation

In the global response, we clarify our motivation behind this work and why we represent our state-action sequences in text-predicate form.

Our method aims to ground the visual information available in hours of offline demonstration data such as EPIC-Kitchens by converting it into text-based states and actions. We hope that language-based task planners [2-5] can learn the high-level task plan from symbolic state-action trajectory—bridging the gap between language instruction and planning, as seen in [1-2].

On Robotic Evaluation

In the global response, we elaborate on how the paper focuses on the problem of extracting rich information from offline vision-language demonstrations. In future work, we plan to evaluate downstream applications of Video2Demo’s generated state-action in simulations (e.g. ALFRED, AI2-THOR [7-8]) and on real robots. On Cost and Speed of Deployment We acknowledge the importance of fast perception models during the deployment and online rollout of learned policies. However, Video2Demo's applications are for offline settings, so it does not need to be deployed on a robot.

Concretely, Video2Demo provides a framework for learning high-level task plans from few-shot demonstrations. We hope that with future work, we can:

1. Collect one or more demonstrations of a human performing a task.
2. Run Video2Demo to extract the high-level states and actions from these demonstrations
3. Rely on task-planning and motion-planning methods to leverage these generated states and actions to replicate the tasks, e.g., generate code to perform this task, conditioned on the robot embodiment, environment affordances etc.

Thus, Video2Demo can run offline asynchronously before the policy is learned then run on the robot.

(continued)

References -

[1] Huaxiaoyue Wang, et al. Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought. arXiv:2305.16744, 2023

[2] Wu, Jimmy, et al. "Tidybot: Personalized robot assistance with large language models." arXiv preprint arXiv:2305.05658, 2023

[3] Wenlong Huang, et al. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv:2307.05973, 2023

[4] Tianbao Xie, et al. Text2Reward: Automated dense reward function generation for reinforcement learning. arXiv:2309.11489, 2023

[5] Yu, Wenhao, et al. "Language to Rewards for Robotic Skill Synthesis." arXiv:2306.08647, 2023

[6] Eric Kolve et al. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv:1712.05474, 2022

[7] Xavier Puig et al. VirtualHome: Simulating Household Activities via Programs. arXiv:1806.07011, 2018

[8] Mohit Shridhar, et al. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. arXiv:1912.01734, 2020

We once again thank reviewer 51er for suggesting we use CodeBERTScore[9] to additionally evaluating the quality of the code generated by Video2Demo using [1].

The following table summarizes the qualitative evaluation of code generated using our baselines, Video2Demo and the human-annotated state-action demonstrations, as described in Table 2 in our paper, replacing ChrF with codeBERTScore as the metric.

We see that codeBERT generally agrees with ChrF scores, and favors code written by Video2Demo over baselines.

added reference - [9] Zhou, Shuyan, et al. "Codebertscore: Evaluating code generation with pretrained models of code." arXiv preprint arXiv:2302.05527 (2023).