Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation

Thanks for your valuable review. We address your concerns below.

${\color{BrickRed}Question 1:}$ Generalization: how this framework can be generalized across different task/robot platforms/datasets? Especially for the sub-goal replan/transition, which seems to be very specific for each task. Can the threshold be determined during training?

Generalize across platforms and datasets: The idea of CLOVER (the closed-loop visuomotor control framework) itself is embodiment-agnostic and can be generalized across different platforms. For instance, the robot in CALVIN simulation is Franka Emika Panda, whereas ALOHA is employed for conducting real-world experiments. Therefore, the generalization is more related to data. In our work, we train CLOVER on the respective data from the two platforms and successfully conduct verification. It is also expected that training on large-scale cross-embodiment datasets may lead to zero-shot generalization.

Sub-goal replan/transition generalization across tasks and models: The thresholds for sub-goal replan/transition empirically generalize well across tasks. For the CALVIN benchmark, there are 34 different tasks in total and the evaluation suite involves 1,000 distinct instruction chains. We use the same set of hyperparameters for sub-goal replan and transition, which works well for all tasks and different visual encoders (Fig. 6(c)). Please also refer to Fig. R3 in the rebuttal PDF.

Threshold determination during training: In our current implementation, the visual planner and the subsequent inverse dynamics-based policy are trained separately. Consequently, the required threshold can only be measured during the inference stage, when the two components are integrated. Future research will focus on training these models jointly in an end-to-end manner, potentially facilitating the threshold determination process during training.

${\color{BrickRed}Question 2:}$ Discussions on computational overhead

Thanks for the question. Please refer detailed discussion in the Q2 of Global Rebuttal session.

${\color{BrickRed}Question 3:}$ Do you expect your proposed architecture to scale well when pre-trained on a large dataset; what additional design might be needed?

Thanks for the insightful question.

• Diffusion-based video generation models have been proven to be scalable with the size of the dataset and network. All we need to do to scale up CLOVER is to collect more videos and extend the model with more channels and layers. Notably, the videos can be free of action labels, and even human videos would help as well. Works like UniPi and UniSim [16] have made very successful attempts towards building world simulators by scaling up the pre-training dataset, and it would be similar for CLOVER to perform such extensions as well.
• Feedback-driven policy: Its training is grounded in an inverse dynamics objective. While it necessitates action labels, it does not require high-level, task-specific knowledge for policy training. This characteristic facilitates the potential for training the policy on extensive, cross-embodiment datasets [38], thereby enabling few-shot cross-embodiment generalization.

${\color{BrickRed}Question 4:}$ How the dataset in CLOVER is obtained? What data helps train the CLOVER? More details will be welcomed.

We train our models exclusively on in-domain datasets. For simulation experiments, we utilize the ABC split of the CALVIN training dataset, which includes language instruction labels, to train the video diffusion model. Consecutive frames and their action labels are randomly sampled to train the low-level policy. Similar procedures are applied in real-world experiments. Additional training details are available in Appendix B2.

By decoupling the training of the visual planner from the feedback-driven policy, we allow the visual planner to be trained on diverse video datasets without action labels. As discussed in Question 3, this approach enables us to leverage internet-scale videos without action labels for training a robust and generalizable visual planner and to utilize large-scale cross-embodiment robot datasets (Open X-Embodiment [38]) for training a more effective low-level policy.

${\color{BrickRed}Question 5:}$ How does CLOVER perform in environments with significant variations from the training scenarios? Have you tested the framework on completely new tasks or objects not seen during training?

Thanks for the question. Zero-shot generalization to completely new tasks or objects stands as a significant challenge for current robotic research. Following the common experimental design, we mainly adopt certain variations during testing to demonstrate the generalization ability of our method.

As requested, we conduct additional real-world experiments in the rebuttal, by introducing entirely new objects — a yellow clay and a doll — alongside the primary interaction object. Please refer to the Global Rebuttal session for detailed results.

Additional Analysis:

• Our simulation experiment setup necessitates the policy's generalizability. In the CALVIN benchmark, the textures of the table and the positions of buttons, drawers, and sliders in the test environment differ from those in the training set, posing a substantial challenge to the generalizability of various policies. CLOVER demonstrates exceptional generalizability in this context.
• CLOVER relies on a video generation model that must reliably follow task instructions to produce corresponding visual plans. Currently, video generation models struggle to generalize to completely novel tasks (instructions) outside the training set. This challenge could potentially be mitigated by extensive pre-training on large-scale internet datasets, which we hope future open-sourced video foundation models will provide to the community. Our inverse dynamics-based policy is inherently task-agnostic and capable of generalizing to new tasks.

Thanks for your careful review and valuable comments. We address each question below.

${\color{BrickRed}Question 1:}$ Limited evaluation to real robot environment.

We conduct new real-world experiments to further validate the effectiveness and generalizability of CLOVER. Please refer to the Global Rebuttal session Q1.(1) and (2) above, and the qualitative analysis in Fig. R2 of our rebuttal PDF.

${\color{BrickRed}Question 2:}$ The measurable embedding space for error quantification seems effective not be sufficiently validated. This component’s effectiveness could vary significantly in different scenarios, and the paper does not provide extensive evidence of its robustness.

We address the concern on robustness in the following perspectives and will update them in our revision:

• Empirically robust to various tasks and scenes: The CALVIN benchmark comprises 1,000 unique instruction chains across 34 different tasks. We select the most challenging scenario where the texture and position of objects to interact during testing is distinct from the training samples (ABC -> D). As outlined in Algorithm 1, our sub-goal transitions and replanning depend entirely on measurable embeddings. Thus, the performance improvement on CALVIN can be a reflection of the robustness of error qualification. For instance, using embeddings with poor robustness can cause the policy to get stuck on certain goals, as the measured distance keeps failing to meet the threshold required to trigger a sub-goal transition.

• Robustness comparison with different embeddings for error quantification: In the table below, We provide quantitative results of LIV which introduces tailored training objectives for dense reward learning (error measurement). CLOVER yields notable robustness compared with previous works. Further qualitative illustrations are provided in Appendix A. \indent |Methods|Task 1|Task 2|Task 3|Task 4|Task 5|Avg. Len.| |-|-|-|-|-|-|-| |LIV [61] (newly added)|70.8|48.2|29.2|18.2|10.2|1.77| |CLIP Feature (Fig. 3 (a))|72.4|46.8|25.0|13.7|5.1|1.63| |State Embedding (Ours, Fig. 3 (c))|96.0|83.5|70.8|57.5|45.4|3.53|

• Model and parameter insensitive: Figure 6(c) shows that we do not need to carefully cherry-pick hyperparameters about the measurement even for models with different architectures.

${\color{BrickRed}Question 3:}$ Computational overhead.

Thanks for the question. We provide a detailed discussion in the Q2 of Global Rebuttal session above.

${\color{BrickRed}Question 4:}$ Further validation on a wider range of benchmarks and task environments is necessary to confirm the framework's generalizability.

Thanks for the comment. On the one hand, the tested simulation benchmark CALVIN ABC-> D considers certain scene generalizations. The texture of the table and the position of buttons, drawers, and sliders in the tested environment is different from what the model has seen in the training set, which is quite challenging to the generalizability of different policies.

To further validate the generalizability of our framework, we conduct more real-world experiments under certain perturbations. Please refer to the submitted PDF for detailed experiment setting and in the Global Rebuttal session.

${\color{BrickRed}Question 5:}$ Minor, in Figure 5, the authors show generated videos of four tasks conditioned on the same initial frame, but the first figure in each task looks different, better to make them the same.

Thanks for the suggestion. In Fig. 5, we show the 2nd, 4th, 6th, and 8th frames of the generated video for simplicity. We have revised our manuscript where the initial frame has been positioned at the forefront.

${\color{BrickRed}Question 6:}$ In Figure 3, the authors mention is obtained during a roll-out, what task and what environment are you using? How many tasks and environments did you test?

The visualization in Fig. 3 is based on a randomly sampled task ("open the drawer") within the CALVIN D environment. We only show one task in Fig. 3 for simplicity and clarity. However, we note that similar patterns are observed across all tasks, where our measurable embedding space consistently shows monotonicity when approaching each sub-goal, compared to the other two counterparts. We also provide a similar plot to Fig. 3, but encompassing all possible tasks within over 300+ independent rollouts, in Fig. R3 of our rebuttal PDF. As shown in the figure, the measurability remains and generalizes well under all scenarios.

${\color{BrickRed}Question 7:}$ How long does it take to execute one task in simulation and real robot?

• Simulation: Our experiments are conducted using NVIDIA RTX 3090 GPUs. The low-level policy, with an input size of 196, operates at a frequency exceeding 70 Hz. The video diffusion model requires approximately 5 seconds to generate 8 video frames with a resolution of 128x128. It only takes around 11 seconds on average to complete a task in CALVIN simulation.
• Real-world: We perform experiments on an NVIDIA RTX 5000 Ada GPU with the input size of 128*96, where the low-level policy runs at more than 25Hz and the visual plan generation takes around 4.3 seconds. Overall, it takes around 38 seconds to complete the 3 consecutive tasks with our real-world robot.

We have added the detailed execution time in the revised version. Please also refer to our reply in Global Rebuttal for further information and comparison with other methods.

${\color{BrickRed}Question 8:}$ Video demos?

Thanks for the advice. Regrettably, we are not allowed to provide an external link to present the demonstration videos according to the NeurIPS rebuttal guidelines. We will release a project page with video demos when publicly releasing the work.

Thanks for your detailed review. We address your questions below.

${\color{BrickRed}Question 1:}$ Novelty of the error measurement approach. Comparison with other state-of-the-art error measurement methods.

As discussed in Sec. 2, existing works rely on additional detection models with manually set rules or high-cost VLMs to understand tasks' completion state. These impose restrictions on their applicability and efficiency.

Compared to works like LIV [61] which also tends to learn measurable embeddings, we do not incorporate additional contrastive objectives but investigate the inherent property of inverse dynamics-based policy. In the new results below, our method shows more robust performance for consecutive tasks. We will update the analysis in our revision.

${\color{BrickRed}Question 2:}$ Clarification of the effect of closed-loop system (long-horizon manipulation).

Each test rollout in CALVIN consists of five distinct instructions (sub-tasks), which requires the policy to succeed in chained sub-tasks. Our framework's improved performance on CALVIN and long-horizon real-world tasks demonstrates its effectiveness in such settings. We acknowledge that current settings are limited compared to tasks like "making a cuisine". We will work to develop more complex long-horizon manipulation tasks in the future.

${\color{BrickRed}Question 3:}$ Scalability of the feedback-driven controller. Evaluating the controller's performance across a wider range of scenarios.

We provide additional real-world generalization experiments, including background distraction, object variation, and dynamic scenarios in Global Rebuttal.

Scalability: As discussed in Secs. 3.1 and 5, by decoupling planning and low-level control into a two-level hierarchy, the visual planner can learn from massive human videos to be robust world simulators [16]; our inverse dynamics-based policy offers greater robustness for multitask IL compared to BC-based policies [59]. Fig. R4 shows that the BC-based method RT-1 struggles with scene variation.

${\color{BrickRed}Question 4:}$ Lacked methodology clarification. Detailed training process used for the text-conditioned video diffusion model and the error measurement strategy.

We have detailed our model architecture and training protocol in Appendix B1 & B2, and we will publicly release all materials. In the rebuttal, we provide details below to address the question.

• Video diffusion model: Extending Imagen [32], we add input/output channels and separate noise injection for RGB and depth, enabling depth generation. Optical flow-based regularization is introduced using diffusion latent embeddings to create cost volumes, with a lightweight CNN serving as ContextNet, inspired by RAFT [39]. Training data consists of 8 randomly sampled frames with fixed intervals (5 in CALVIN, 20 in real-world experiments).
• Feedback-driven policy: We use MLPs for the action decoder, but Transformers or Diffusion Policy could be alternatives. The policy is trained with the inverse dynamics objective, using frames from random intervals in demonstrations to enhance its robustness.

${\color{BrickRed}Question 5:}$ Relatively limited experiments.

Thanks for the advice. RLBench and Robosuite focus on few-shot learning with tasks' horizons comparable to a single subtask in CALVIN. Thus, CALVIN is the most suitable benchmark for validating long-horizon capabilities. We will consider involving additional simulation experiments.

Please also refer to Global Rebuttal Q1(3) for additional real-world experiments.

${\color{BrickRed}Question 6:}$ Computational complexity.

We provide detailed discussion in Global Rebuttal Q2.

${\color{BrickRed}Question 7:}$ How does CLOVER handle highly dynamic environments?

Most existing works test in static environments where the interaction object remains stationary. Our work also focuses on similar settings and does not address environmental dynamics. We envision this as an important future direction and have incorporated the discussion below into the revision.

• How to adapt to dynamic environments: CLOVER could be adapted for dynamic settings by using multi-frame conditioning to capture velocity and acceleration information. Besides, incorporating the replanning mechanism (Appendix A) at each inference step could help handle new scenarios.
• Additional experiments with unpredictable changes: In the added tests, we randomly place and pick up a doll to create unpredictable visual changes. Thanks to the robustness of visual misalignment of our inverse dynamics-based policy, CLOVER surpasses RT-1 by a large margin.

${\color{BrickRed}Question 8:}$ How is the training data organized during the diffusion model generation phase of the visual plans?

During training, CLOVER extracts 8 frames at five-frame intervals, covering key task segments. This ensures better task alignment and fewer generation rounds during a test (1-2 rounds per task). As shown in Figs. 6(a) and 9, CLOVER effectively achieves sub-goals with adaptive steps and generalizes to different visual planners without specific adjustments. Detailed protocol is in Appendix B2 and will be further clarified.

${\color{BrickRed}Question 9:}$ Computational requirements. How does it scale with more complex tasks?

Please refer to Global Rebuttal for the discussion on computational requirements.

For more complex tasks, we may introduce LLMs to decompose the tasks into more manageable subtasks similar to HiP [29]. CLOVER can then generate visual plans for simplified tasks and perform real-time low-level control. The computational cost of this system will potentially scale linearly with the number of subtasks derived.

Thank you for your response. I've read the other reviews and the rebuttal. I’m keeping my initial score.

Thanks for your careful review and we really appreciate your comments. We address your questions below.

${\color{BrickRed}Question 1:}$ How are you different from AVDC?

Our work differentiates from AVDC in the following aspects:

• Model structure: For the visual planning part, we introduce a novel constraint term to enhance its temporal consistency and endow the model with additional depth generation ability.
• Action output mechanisms: As discussed in the related work, AVDC infers actions from predicted video content with dense correspondence. It combines an off-the-shelf optical flow estimator with depth information to compute SE(3) transformations of the end-effector and transfer it into action control signals. However, CLOVER learns an inverse dynamic model to output actions and adaptively reach given goals. Paired with error measurement and feedback mechanism, CLOVER is more robust to the inherent instability of video diffusion models and can be adapted to broader scenarios.
• Feedback policy: The AVDC framework lacks a feedback mechanism within its visuomotor control system. In contrast, our work aims to explicitly quantify errors and incorporate them into a unified framework for long-horizon manipulation.

We have added the above discussion to the revision.

${\color{BrickRed}Question 2:}$ Could the authors elaborate more on the superiority of their design which use diffusion model generate reference frames and then use inverse dynamics model to get the actions, over the design like diffusion policy which directly generate actions from diffusion model with a closed-loop style?

Thanks for the question. Our decoupled paradigm has two main benefits:

• As mentioned in Section 4.2, our diffusion-based video model can effectively understand high-level instructions and generate corresponding plans. This alleviates the learning complexity of policy by decoupling the planning and control into a two-level hierarchy, where the low-level policy can be trained in a task-agnostic manner (no need for high-level instruction labels).
• The inverse dynamic model (IDM) can be established based on the generated sub-goals, which maps the state transitions to actions. In contrast, diffusion policy uses the current observation only following the paradigm of behavior cloning (BC). Previous work [59] proves that IDM could be more performant and robust than BC, and scales best with the pretraining dataset.

With the above benefits, we show the stronger performance of our method in Table 1, compared to the diffusion policy-based model "3D diffusion actor" [52], even though it utilizes more sensor input.

${\color{BrickRed}Question 3:}$ What is the orange dash line meaning in Figure 6(a)?

We intend to highlight the highest performance achieved by open-loop methods for clearer comparison to closed-loop counterparts. We have revised the subfigure.

${\color{BrickRed}Question 4:}$ It seems author defines sub-goals as unreachable when the cosine distance between the state embeddings of consecutive are too far. Would there be circumstances when the embedding is close but the sub-goal is actually not reachable due to singularity or other robot limits? Especially when testing the framework on real robots?

Thanks for the insightful question. Injecting physical constraints or considering potential robot limitations in video generation models (world models) remains a longstanding challenge in visual plan generation. Even models like Sora, which are trained with extensive data and computational resources, have been found to fail in fully adhering to physical laws. However, the reliability of visual planners can be significantly improved by training on large-scale, real-world robot demonstrations that are inherently physically plausible. The generalizability of such models to unseen embodiments during the training phase remains an area that requires further exploration.

In practice, the lack of physical feasibility in visual planning can result in scenarios where a robotic arm stops at a certain position for an extended period without the error reaching the predefined threshold. To empirically enhance the reliability of visual planning, we can implement human-defined rules to detect these conditions and trigger replanning across multiple rounds of generation, thereby addressing this limitation.

${\color{BrickRed}Question 5:}$ Did not test on generalization ability.

Thank you for the comment. In the current draft, the benchmark CALVIN ABC-> D itself certifies the scene generalization ability of the learned policy [45]. Specifically, the texture of the table and the position of buttons, drawers, and sliders in the tested environment are different from those in the training set, which challenges the model's generalization in unseen scenarios. In real-world robot experiments, we further validate the position generalization by putting the fish to be grasped at different positions for each rollout.

During the rebuttal, we provide additional real-world results to verify CLOVER's robustness against background distraction and object variation. Please refer to Fig. R1 and R2 in submitted PDF for detailed experiment settings and the results given in the Global Rebuttal session Q1(1) and (2).