VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks

We thank the reviewer for the insightful comments. We address concerns below.

Concern re: contributions of VideoWA

• We believe that video as an input modality has value on its own especially given how relatively few evaluative works, as of the time of writing, have been done on how large foundation model-based agents perform with videos, together with text etc., as inputs under dynamic agentic settings:
  • how an agent model understands an instruction via video vs. text can be considerably different as each modality has different kinds/amounts of informational context.
  • e.g., humans sometimes prefer to use videos to learn how to do certain tasks (vs reading docs) as they may more "easier" or contain additional information.
• Our work demonstrates that these video tutorials may not be used/processed optimally by long-context models in the manner humans use videos to complete tasks, and there are innovations to be made on that front.
• Re: the necessity of video input; agent performance on the original WebArena and VisualWebArena tasks we provide tutorials for is 14.9% and 19.78%, respectively. Therefore, the agents are not able to perform these tasks at a prolific rate, especially given that human performance sit at 82.6% and 72.7%.
  • This performance gap inspired the creation of video tutorials and their ability to help the agents. We also show in our human evaluations that providing a tutorial to humans gives a boost over these numbers in success rate and efficiency.
  • However, our experiments of providing tutorials to agents show they actually perform worse, which is an interesting and possibly new finding.

Low human success rates

• In human evaluations, participants had no prior knowledge of the tasks or any assistance (e.g., internet access, search tools, replay). We consider 73.9% a reasonable success rate and note that many errors came from inconsistent human attentiveness (especially for longer videos, etc.).
• Considering many of the tasks require watching a full video to complete a task, this can become tedious for human evaluators, which can take over 5 minutes in one contiguous sitting. Several cases also appeared to result from human error (e.g., carelessness).
• We also note that these success rates are, on average, in line with human performance levels among other agentic benchmarks such as WebArena (~78%), OSWorld (~72%), Windows Agent Arena (~74%), etc.
• Additionally, given the range of difficulty levels in our benchmark’s tasks, we expected that the non-uniformity of task difficulty will result in less than perfect, or even very high levels of, human performance.

"Noise" introduced by video information or ineffective agentic reasoning

• We observed that the "noise" from introducing video causes issues with skill-retention tasks; for example, this, at times, causes the agent to fail at following action generating instructions.
  • e.g., under the Set-of-Marks elements, a click action must be generated using click [elem] where elem is the numeric ID of the SoM element. However, the agent would return click [elem] where elem was the name of the element. This formatting issue persisted for other actions with the longer prompt.
  • e.g., a common issue for skill retention tasks was the agent generating multi-action responses when the prompt explicitly says to generate one action.
  • e.g., Given the tutorial or summary to complete a similar task, the agent would get distracted by the comprehensive plan and generate multiple actions from the video information, straying away from the prompt guidelines. Many times, the agent became distracted by the video and began generating actions unrelated to the task or its previous actions.
• One hypotheses relates to the backbone of these agents (multi-modal LLMs) as they were first grounded in language and have recently been adapted to other modalities, resulting in "imperfect" or lossy learning of, and across, them.
  • As such, instead of being first grounded in vision (e.g., learning/pre-training with images), this adaptation to image/video via its language/text understanding may introduce “noise” in its learning, causing unwarranted and unexpected behavior.
• From analyzing agent trajectories, we found that agents show inconsistencies in both their reasoning and planning.
  • e.g., agents can get stuck in a loop, repeating the same action/step even when the observed state or information does not change. Agents also undo correct tasks or partial progress or simply hallucinate.
  • Additionally, similar issues with visual grounding/understanding of the webpage remains an issue seen even in VisualWebArena; the agent will generate a correct plan but choose the wrong element to interact with.
• We will include a section with more details of failure modes in the revised manuscript.

Please let us know of any other questions!

We are very encouraged that the reviewer finds our benchmark and agent experiments novel and interesting!

Source code for this paper

• Yes, we plan on making our benchmark and agent code available with robust documentation and details regarding installation, setup, and implementation along with examples and common problems/solutions. However, due to ICLR’s double-blind policy, our current repo is private. We will make our repo public immediately afterwards.

Please let us know of any other concerns/questions!

We thank the reviewer for their insightful comments and suggestions. We address concerns below.

References in introduction

• We will add references to better contextualize our paper, focusing on the development of video-capable and multi-modal agents as well as their benchmarks.

Dataset comparison table of existing benchmarks

• Great suggestion. We will include a table highlighting a comparison of related benchmarks in our revised manuscript in Section 2 so readers can better visualize and compare benchmarks with ours.

Differences b/w VideoWebArena and VisualWebArena and WebArena

• First, although we use similar environments as VisualWebArena/WebArena do, we introduce intermediate evaluation into our task design for our bechmark.
  • As noted in our Task Design section, each video-based task has an intermediate question/answer pair that must be answered correctly for the task to be completed successfully.
  • e.g., if the video-based task is “Block the user that made the post about reindeers in the video”, the intermediate question would be “What was the name of the user that made the post about reindeers in the video”. All of our 400 factual retention tasks include this intermediate intent, that was manually created and verified by the papers’ authors.
• Second, we create ~4 hours of video content for tutorials/tasks; while we use these for our benchmark, this considerable amount of content can also be used for various modeling, planning, and behavior cloning applications in WebArena/VisualWebArena, larger web agent studies, and more.

Differences among video/frames/summary agent in terms of their input

• In short, the video agent takes in a MP4 or MOV file (without extracting audio), while the video frame agent takes in N frames and the audio transcription of the video using Whisper (OpenAI). We include these details in the Baseline Agents section of the paper.
• In comparison, the summary agent takes in a summary of the video with respect to the task as well as the summary text at every step.
  • The summary is generated by feeding the video or the video frames + audio as input and prompting the VLM to give a summary of the task at hand. Sample prompts are in the Appendix.

In Table 4: a clearer example focused on temporal reasoning

• Thank you for pointing this out. We will instead change the example to a task in the Gitlab domain, "Can you assign the issue with the critical priority label which showed up earliest in the video to @earlev4?". This task does not explicitly require counting and hopefully highlights the temporal nature of the task!

Inclusion of open-source VLMs such as LongVILA, and LLAVANextVideo

• Due to memory/compute resource limits, as well as the short timeframe for discussion period (with experiments for even a closed-source model taking ~2weeks), we unfortunately have been unable to test large open-source models. Examples include:
  • Memory: Many commercial APIs that host open-source multi-modal models support multi-modal input but have restrictions on input format. For example, on Azure, both LLaMA3 vision and Phi3 vision only accept a single image input (videos need several images/frames); hoever, we require ~60 images per inference.
    • Others like Replicate limit input size to 20MB per request while our inputs can easily exceed 50MB due to the video content.
  • Compute: Our experiments are also computationally intensive and time-consuming as they involve processing up to 60 images, along with other content, per inference.
• As such, we test a smaller open-source multi-modal model, Phi-3.5V using 30 frames. Its predecessor Phi3 has also shown strong performance in long video understanding, at times better than 7B/13B models (source: Long Video Bench https://arxiv.org/pdf/2407.15754).
  • Results of Phi-3.5V are below which will be included in our revised manuscript as they serve as a good contrast to our existing results (Phi-3.5V is a smaller open-sourced multi-modal model vs. Gemini and GPT which are large closed-source ones).

Please let us know if any other questions/concerns arise.

Thank you again for taking the time to review our changes/revisions -- we very much appreciate the reviewer's time and feedback in helping improve our work!

We thank the reviewer for their insightful questions. We address your concerns below.

Adding testing/evaluation of open-sourced models

• Given our focus on constructing, scaling, and testing the benchmark’s tasks and video creation, our efforts on model evaluation prioritized popular state-of-the-art foundation models that take in video modality as input (e.g., GPT, Gemini) as well as their ablations.

• Due to limits on memory and compute, as well as the short timeframe for discussion period (with experiments for even a closed-source model taking ~2weeks), we unfortunately have been unable to test large open-source models. Examples include:

  • Memory: Many commercial APIs that host open-source multi-modal models support multi-modal input but have restrictions on input format. For example, on Azure, both LLaMA3 vision and Phi3 vision only accept a single image input (videos need several images/frames); hoever, we require ~60 images per inference. Others like Replicate limit input size to 20MB per request while our inputs can easily exceed 50MB due to the video content.
  • Compute: Our experiments are also computationally intensive and time-consuming as they involve processing up to 60 images, along with other content, per inference.

• As such, we test a smaller open-source multi-modal model, Phi-3.5V. Its predecessor Phi3 has also shown strong performance in long video understanding, at times better than 7B/13B models (source: Long Video Bench https://arxiv.org/pdf/2407.15754).

• Results of Phi-3.5V are below which will be included in our revised manuscript as they serve as a good contrast to our existing results (Phi-3.5V is a smaller open-sourced multi-modal model vs. Gemini and GPT which are large closed-source ones).

Lack of discussion of other video benchmarks

• We will include a fuller discussion and comparison of other benchmarks, including MVBench, to highlight differences and contextualize the contributions of each.
• Our main difference/novelty/contribution lies in the agentic nature of our video benchmark, which requires an agent to generate actions in an interactive environment on an open-ended task in addition to answering video-understanding-based questions.

Adding new tasks/domains to the benchmark

• Yes, new tasks/domains can be easily added. We will include instructions on creating new tasks, along with examples, in our readme and repo which will give a comprehensive walkthrough.
• Adding new domains is also very easy simply by specifying the starting URL in the task JSON. We will also add an example task json to the appendix to further clarify.

Possible aspects of improving existing agents

• Our analysis of failure modes found similar agent errors also seen in other benchmarks:

  • Hallucinations, failures in visual grounding, infinite loops, undoing correctly done tasks, and failing to adhere to the prompt instructions for generating actions.
    • e.g., under the Set-of-Marks elements, a click action must be generated using click [elem] where elem is the numeric ID of the element. However, the agent would return the name of the element.
    • e.g., the agent generated multi-action responses when the prompt explicitly instructed the agent to generate one action. When given the tutorial/summary to complete a similar task, the agent became distracted by the summary or overall task plan and began generating multiple actions from the video information.

• Regarding improvement, we propose several possibilities given our observations:

  • Better distill/coordinate multimodal chain of thought reasoning so that models can improve reasoning and planning performance in the presence of an increasing number of modalities.
  • Teach the agent to prioritize relevant/mandatory instructions regarding the task over less important, dense in-context information.
  • Have the agent learn when to invoke the video modality (or any modalities): if some modalities are helpful for certain tasks, the agent should then extract/process that information.

Please let us know of any other questions!

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?

We thank the reviewer for their time and insightful comments. We address each concern in turn below.

Re: dynamic, interactive challenges

• We agree that an interactive context is important in an agent’s ability to dynamically plan and accomplish tasks, which is exactly why our benchmark:

  • (1) evaluates the ability of agents in interactive environments and
  • (2) evaluates the ability of agentic foundation models to process and understand information from a video modality in these dynamic settings.

• We emphasize that our tasks all require deploying agents in a dynamic, interactive environment. More information on our environment’s interactivity along with other details of the interactive environment and the tasks can be found in Section 3.

• The live environments are all locally-hosted real websites that are populated with real data that can be modified by the agents’ actions.

• These sites are responsive, changing in response to agent actions—just like how human users interact with websites.

• The video information is provided in-context to the LLM agent at each time step as part of its input. Each of the agent’s actions causes a change in the webpage state; for example, clicking on a link will change the webpage while the agent is doing a task. We will make this clearer in our paper.

Re: potential biases

• We agree with the reviewer on the importance of inclusivity/fairness in benchmark datasets—while no benchmark can perfectly represent all global variations, we have made considerable efforts to ensure that ours is sufficiently broad to test robustness in our area of focus across realistic scenarios.
• To clarify, our main focus is studying the ability of agents to make use of video information to accomplish tasks in dynamic settings. As the issues of multi-modal video-capable agent performance and in-context video understanding pose huge obstacles, we made this the main focus of our data/evaluation set-up.
• Nonetheless, to better address these issues, we will make several changes:
  • We will add Portuguese/Chinese/Korean variations to the video dataset---though not comprehensive, we intend this to be a start.
    • Given the significant time/labor required to implement new websites and the main focus of our paper, adding non-English websites unfortunately is beyond the scope of this paper. We have, however, made it so that our benchmark and its videos can be easily accessed and spun-up by the open-source community to test/study potential biases.
  • We will include an ethics statement along with a discussion on the limitations and disclaimers about the reviewer’s points to our paper, citing the lack of diversity in tasks and video data that may not span the global use cases mentioned above.
  • We also recognize that benchmarks evolve with the community’s needs, identification of new flaws/biases, etc., and welcome feedback. Benchmarks are iterative tools and we encourage future work to build upon our efforts to further address issues of inclusivity and global applicability.
    • We have intentionally left room for updates and extensions to the dataset to incorporate additional underrepresented content or address gaps identified by the community.
    • Our work and its documentation make it very easy for users and the open-source community to create their own tasks customized with their own videos, websites, etc. that reflect specific variations of interest that may be underrepresented here. Additionally, existing websites can be used to create new tasks.
    • Our repo will be available and monitored to also address potential issues and problems as they arise with implementation, testing, etc.

Please let us know of any further questions/concerns!

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?