Thank you for the detailed review. Please see our response below.

W1. Originality of self-conditioning consistency

Thank you for highlighting this point. Self-conditioning consistency for videos is indeed an original contribution of this work, designed to enable iterative refinement of generated videos. In VideoAgent, we use "consistency" in a specific context to mean mapping any hallucinated or incoherent video to a realistic, non-hallucinatory one. We do not employ traditional consistency models, our iterative refinement process serves a similar purpose: it aligns multiple flawed versions of a generated video into a progressively more coherent final output. We understand that the current writing may have created some ambiguity about its novelty. In particular, equation 6 adapts the DDIM equation for generating $\mathbf{x}^{(t−1​)}$ from $\mathbf{x}^{(t)}$​. In our approach, we replace the "predicted $\mathbf{x}^{(0)}$" term in the original DDIM equation with $\mathbf{\hat{x}}$, which is the previously generated video sample. During training, DDIM predicts $\mathbf{x}^{(t−1)}$​ based on this $\mathbf{x}^{(0)}$, while in VideoAgent, we predict the next video based on $\mathbf{\hat{x}}$ for a set of parameters $\theta$. This adaptation is what we mean by "mimicking ODE solvers," as our method similarly refines video predictions iteratively based on previous samples.

We recognize that this intuition may not be fully clear in the current text. To address this, we have revised Section 3.1 to explicitly highlight this chain of ideas and clearly delineate our original contributions.

W2. Use of VLM feedback in Equation 10

Thank you for the suggestion. After reviewing your feedback, we have updated the paper to explicitly incorporate how feedback is conditioned into Algorithm 2. Specifically, as outlined in Equation 10 and Section 3.1, the feedback is provided in two forms: (1) binary feedback from the VLM indicating success or failure, and (2) descriptive feedback where the VLM identifies inconsistencies and errors in the generated video plan. This textual feedback is concatenated with the task condition and used to refine the video plan generation. We appreciate your suggestion, which has helped improve the clarity of our work.

W3. Novelty of using VLM feedback for video generation

Thank you for your comment. To the best of our knowledge,the use of VLM feedback to improve video generation models is indeed a novel aspect of our work. We have updated the paper to explicitly highlight this novelty, along with the introduction of self-conditioning consistency, as the core contributions of our approach. Section 4.3.1 presents the results validating the effectiveness of these contributions.

W4. Contributions of key modules versus Online-and-Replan

Thank you for your valuable feedback. One of the key contributions of our work is the use of robotic tasks to ground video generation, providing a practical framework for refining video plans in a task-specific manner. While the improvements from Online-and-Replan are significant, our refinement modules play a crucial role in offline contexts, where replanning is not feasible. This enables iterative enhancements to video plans without requiring task resets, making the approach broadly applicable across various embodiments and data inputs. Additionally, our model demonstrates strong zero-shot generalization on real-world tasks (shown in Appendix H1 of the updated version of our paper), further highlighting its robustness and versatility in diverse settings.

a) Online Environment Access: Unlike traditional reinforcement learning, which relies on exploration through trial and error, our method focuses on goal-conditioned video generation to enhance agent performance. By leveraging real-world feedback, we refine video plans to align closely with intended goals, minimizing the need for computationally expensive trial and error. This approach turns online interaction into a direct optimization of video fidelity and task success, offering a more efficient use of environment access. Moreover this approach is scalable and adaptable to various robot embodiments, and potentially even internet scale human data.

b) Architecture and Objectives: Our architecture addresses a critical challenge: as video generation models improve, resolving inaccuracies within video plans will become the primary bottleneck. We proactively address this with self-conditioning consistency and VLM feedback, refining video quality iteratively and enabling high-precision plan generation. This forward-looking approach positions our method to adapt as video generation technology advances, providing impactful, scalable improvements in video-agent learning.

Thank you for recognizing the significance of this work! Please see our response to your questions below.

W1. Inclusion of the self-conditioning consistency loss

Thank you for your question. The second term in equation 7, the consistency loss, is crucial for ensuring that the generated $\mathbf{x}^{(0)}$ remains consistent when conditioned on different $\mathbf{x}_{i}$. This loss encourages the refinement process to align outputs across different intermediate inputs, improving the coherence and quality of the refined video.

Including this term allows VideoAgent to handle variations in the input video plans during refinement, ensuring that the model does not diverge when faced with slightly different initial conditions. Without this consistency term, the refinement process may fail to converge or produce results with greater inconsistencies.

W2. Figure 2 and the target of the diffusion process

Thank you for your feedback. After reviewing the original figure, we recognized that it did not fully capture the denoising and refinement processes accurately, which could cause confusion. To address this, we have updated the figure to better represent the methodology. The updated figure explicitly separates the denoising process and the refinement iterations. The denoising process progresses from $x^{(T)}$ to $x^{(t)}$, eventually generating $\mathbf{x}_i$.

This represents the diffusion model's role in producing an initial video plan. In the refinement process, shown below, $\mathbf{x}_{i+1}$ is generated by conditioning on $\mathbf{x}_i$, effectively refining the video output iteratively. This iterative process ensures that each subsequent video sample incorporates corrections, guided by feedback, to achieve improved quality and coherence. This updated figure more accurately reflects the methodology described in the paper, and we have included it in the revised version for improved clarity.

W3. Absence of real-robot experiments

Thank you for this insightful suggestion. We acknowledge that our work does not include real-robot experiments due to a lack of access to real-robot systems, as noted in the limitations section of the paper. However, our primary focus is to establish a stepping stone for improved video generation models, with the goal of facilitating future transfer to real-world robotic policy learning.

The proposed video refinement process is designed to address challenges in video plan generation and self-improvement, which are critical for real-robot applications. By demonstrating significant improvements in video generation and refinement, our method lays the groundwork for future research to integrate these advancements into real-robot systems. This work aims to bridge the gap between video generation and real-world policy learning, making future real-robot deployment more feasible and effective.

Thank you for recognizing the significance of this work! Please see our response to your questions below.

W1. Computational overhead, marginal gains after two iterations (Figure 4)

Robotic control from videos, as demonstrated by AVDC, is an exceptionally challenging problem with high relevance due to its potential for leveraging internet-scale knowledge transfer. While this task remains difficult, our method demonstrates significant improvements in specific tasks, including door-open, button-press, and faucet-close, which represent meaningful progress compared to prior work. Notably, little work has demonstrated improvements over AVDC, highlighting the significance of our results.

Additionally, both The bitter lesson[1] and recent advancements in large language models (LLMs)[2] have shown that trading computational overhead for better performance has lots of potential. Our approach follows this principle, prioritizing performance gains while leveraging additional computational resources.

W2. Necessity of Refinement and Self-Improvement for Task Success

We appreciate the reviewer’s valuable point about balancing refinement with replanning for cost-effective success. Refinement proves advantageous in precision-dependent tasks (e.g., 'faucet-close' in MetaWorld), where compounding errors from early frame inaccuracies could require frequent resets with replanning alone.
a) Offline Context: In offline settings, where replanning is not feasible, refinement is essential for iteratively enhancing video plans without task-specific reinitialization.

b) Broad Applicability and Generalizability: Refinement allows VideoAgent to integrate data and insights from other tasks (e.g., from refining human videos), creating broadly shareable improvements across tasks, embodiments, and potentially even human settings. Replanning, by contrast, is inherently task-specific, making refinement a more versatile and reusable enhancement strategy.

c) The contribution of this work goes beyond better task success. A main contribution of this work is using robotic tasks to ground and improve video generation.

Q1. Missing Term in Algorithm 1 from Eqn 7

We would like to clarify that Equation 7 and Algorithm 1 represent the same process. Equation 7 does not depict the standard diffusion loss but instead reflects the self-consistency loss, with both terms emphasizing self-consistency. During training, we set $\lambda = 0$. The consistency mechanism in VideoAgent is captured in the second term of Equation 7, where the loss function minimizes differences between videos generated from two individual samples at different timesteps. This mechanism promotes frame-to-frame coherence, enabling VideoAgent to iteratively refine its outputs.

To address any potential confusion, we have updated the paper to explicitly state this connection and clarify the role of self-consistency in our method. Thank you for pointing this out and helping us improve the presentation of this concept.

Q2. Open-Loop Execution and Planning Horizon

Each video plan is executed in an open-loop manner. The plan is first passed through a flow model, which processes it to determine grasp positions. Each video plan typically corresponds to approximately 60–100 time steps. The task horizon is set to a maximum length of 500 steps.

Q3. Additional Trajectories per Iteration

To achieve this improvement, we collect 15 successful trajectories for each task during every iteration and perform online finetuning on them. This standardization helps address task imbalance, as task success rates are higher for certain tasks compared to others. By ensuring a fixed number of successful trajectories per task, we prevent overfitting to easier tasks and maintain balanced model performance across the entire task set. We have incorporated this into the paper to clarify how we mitigate task imbalance during finetuning.

Q4. Definition of Task Success in Table 5

In Table 5, "task success" measures only the proper completion of the task as defined by the initial goal-conditioning textual prompt, ignoring the relative differences in video generation quality. In Table 8, presented in the Appendix, we report both this task completion success rate and additional human evaluation metrics on fine-grained aspects such as video quality, temporal consistency, and more.

Q5. Include Conditioning Variable for Clarity

Thank you for the suggestion. We have detailed how we condition on feedback in Equation 10 and in Section 3.1. We will ensure this is explicitly incorporated into Algorithm 2.

Q6. Typo in line 342

Thank you for pointing this out. We have corrected this in the updated paper.

[1] Sutton, Richard. "The bitter lesson." Incomplete Ideas (blog) 13.1 (2019): 38.
[2] Snell, Charlie, et al. "Scaling llm test-time compute optimally can be more effective than scaling model parameters." arXiv preprint arXiv:2408.03314 (2024).

Thank you to the authors for their efforts in addressing my concerns. Revised section 3.1 improves the clarity of the proposed method, and most of my questions have been addressed. I will maintain my positive rating. However, I do have a few suggestions for further improvement on the paper:

• Comparing the "AVDC" and "VideoAgent" rows in Table 1, apart from "door-open" and "faucet-close," which clearly benefit from the refinement process, performance on some tasks (e.g., basketball, faucet-open, handle-press, and assembly) has also decreased. I believe including the standard deviation of these numbers may better clarify the performance variance.

• In Section 3.1, the same $\lambda$ was overloaded for both Eq. 7 and Eq.9, it would be clearer to use different symbols to denote the coefficients

Thank you for the detailed review. Please see our response below.

W1. Consistency model vs. DDPM

In VideoAgent, we use "consistency" in a specific context to mean mapping any hallucinated or incoherent video to a realistic, non-hallucinatory one. Although we do not employ traditional consistency models, our iterative refinement process serves a similar purpose: it aligns multiple flawed versions of a generated video into a progressively more coherent final output.

Each iteration in VideoAgent can be viewed as an analogue to a consistency model timestep. Where traditional consistency models aim to map noisy representations at any given timestep to a clean data point, our approach uses each iteration to refine and align previously generated, flawed video frames toward a coherent, realistic final video.

While we adopt the DDPM + DDIM structure during training and inference, our unique refinement goal and use of consistency distinguish VideoAgent from these frameworks. Traditional DDPM and DDIM models do not perform iterative refinement—if a sampled video is flawed, the output remains flawed, as these models lack mechanisms to self-correct.

In contrast, the consistency mechanism in VideoAgent, captured in the second part of equation 7, applies a loss function to minimize differences between videos generated by conditioning on two individual samples at different timesteps. This iterative process promotes frame-to-frame coherence and allows VideoAgent to refine outputs progressively, addressing errors and improving video quality—a capability that DDPM and DDIM structures alone cannot achieve.

We recognize that this intuition may not be fully clear in the current text. To address this, we will revise Section 3.1 to explicitly highlight this chain of ideas and clearly delineate our original contributions.

W2. Addressing the need for comprehensive baseline comparisons

Thank you for your valuable feedback. After reading your comments, we conducted additional experiments on the suggested baselines, including VLP and Behavioral Cloning (BC). AVDC is already state-of-the-art compared to BC (16.2%) and UniPi (6.1%). However, it falls short of the overall success rate achieved by Diffusion Policy (24.1%) without replanning. In contrast, VideoAgent surpasses Diffusion Policy during its online iterations, even without employing replanning. This demonstrates the effectiveness of our iterative refinement approach in improving video plan quality and task success, highlighting the robustness of VideoAgent in comparison to existing methods. The results from these experiments have been included in the updated manuscript and have been uploaded alongside this response for your reference.

We would like to highlight key distinctions between our approach and VLP. While VLP uses language for planning, our method leverages language as feedback to iteratively refine video plans—a fundamental difference in methodology. Additionally, VLP does not perform refinement, which is central to our approach. Despite these, we include VLP results for comparison. To ensure fairness, we also present results from a modified version of AVDC integrated with a VLM to select the best video plan without using refinement iterations.

Meta-World Results

Thank you for your response. Most of my questions were based on confusion regarding the consistency model, which has now been resolved.

I would like to summarize my thoughts on this paper:

1. The paper introduces a novel method to refine the generated videos through a refine model.
2. Experiments on metaworld demonstrate its superiority compared to AVDC (ICLR 2024) and Diffusion Policy.

While the authors' improvements have enhanced the quality of the paper, I believe it lacks a comparison in real-world robotic settings, such as Calvin[1] or Simpler, or real robots. The experiments are limited to metaworld. Recent works [2][3] increasingly focus on studies based on real-world robotic setups. Compared to these works, Video-Agent is conceptually novel but may lack practical applicability, particularly since this paper is fundamentally an application-focused study. Therefore, I have decided to increase my score to 5, but I remain hesitant about accepting this paper. Due to the low action FPS, it may struggle to be applied to real robotic systems. In the future, if video generation becomes real-time, it might hold more potential.

References:
[1] CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks

[2] Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers

[3] RDT-1B: A Diffusion Foundation Model for Bimanual Manipulation