Any2Policy: Learning Visuomotor Policy with Any-Modality

We thank reviewer for the comments and reference reviewer with identifier 5TLy as R4. Comment n of reviewr m is denoted as RmCn.

[R4C1]

While the authors have conducted extensive ablations on real-world experiments, replicating these results in practice may be challenging for follow-up researchers. To facilitate better benchmarking, it would be advantageous if the effectiveness of each proposed method could also be demonstrated in simulated experiments.

We share the reviewer's sentiment that building a simulation environment is a paramount topic that could improve productivity and enhance future research. We are currently developing a simulation based on Isaac Sim, which we hope will facilitate further research in this domain.

[R4C2]

Due to the high cost associated with evaluating real-world tasks across different random seeds, reporting variance is difficult. Nonetheless, it would be beneficial if the authors could assess the impact of multiple random seeds in simulation tasks to provide more robust statistical insights.

Since our simulation environments were not ready, we performed a robustness study on random seeds in real-world experiments. We evaluated four tasks in the real world, running 3 seeds for each experiment with seed numbers 0, 1, and 2. For each seed, we conducted 10 evaluations across the four real-world tasks. Both methods were evaluated 10 times, and we report the average success rate for each method.

Our experiments indicate that the training of our approach is stable, and the results are consistent across different seeds.

[R4C3]

Missing MetaWorld Results.

We thank Reviewer 4 for the careful review and for pointing out the missing experimental results. We sincerely apologize for this oversight. Here is a brief summary of the experimental results. We present the results based on three task levels, following the settings in Masked World Model (CoRL'23). We report the average success rate across three task levels: easy, medium, and hard. In total, there are 45 tasks. All experiments were trained with 20 demonstrations, evaluated with 3 seeds, and for each seed, the success rate was averaged over five different iterations. We used all available modalities for our method. The experimental results are shown below:

The numbers in parentheses indicate the number of tasks for each specific task level. The experimental results support our conclusion that Any2Policy are able to achieve better performance than the baseline across different levels of task difficulty.

[R4C4]

Comparison with Mutex.

We thank the reviewer for raising this point. Mutex is indeed a significant work that explores multimodal task specification. To strengthen our work, we conducted a comparative study with Mutex. Specifically, we selected four real-robot tasks and compared them with Mutex using the same training data. Both methods were evaluated 10 times, and we report the average success rate of each method.

Our experimental results indicate that Any2Policy performs better than Mutex. We believe that the performance gain comes from the additional modalities introduced by our method.

[R4C5]

I'm curious to know whether the video instructions were recorded using the same embodiment or if they include cross-embodiment examples, such as demonstrations by humans?

Yes, the video instructions were recorded through human demonstration.

Thanks for the careful comments. My major concerns are well-resolved. So, I'm happy to increase my confidence to reflect this (from 3 to 4).

Rebuttal:

We thank reviewer for the comments and reference reviewer with identifier RRf1 as R3. Comment n of reviewr m is denoted as RmCn.

[R3C1]

The technical contribution of this paper is limited. Although the paper analyzes some difficulties in the process of fusing multimodal information and embodied alignment, the overall model architecture is still relatively straightforward and simple.

We thank the reviewer for raising this point. While the technical novelty may seem limited, our primary contribution lies in providing a straightforward yet effective solution that, for the first time, successfully integrates multiple modalities for both task specification and observation, enabling seamless robotic manipulation. Our method demonstrates significant performance gains over the baselines in new settings, underscoring the necessity of our approach

[R3C2]

The proposed dataset seems to lie in the addition of rich paired audio data? Would it be a better choice to augment the existing datasets with information from modalities such as audio?

We appreciate the reviewer's concern about our dataset. Notably, our dataset goes beyond merely adding rich paired audio data. We offer a comparison with existing datasets in the following table. We focus on the types of modalities in existing data on both observation and task specification.

Most previous datasets provide point clouds and images for observation, and text for task specification. There are other modalities, such as image goals and audio, for task specification. Despite being smaller than Open-X and other datasets, our dataset is more comprehensive in the domain we aim to study, specifically aligning multiple modalities for both instruction and observation to facilitate the training of robotic manipulation. It highlights the unique value our dataset brings to the community.

Secondly, while it is possible to expand existing datasets with this additional information, it would require significant effort and time. This presents a valuable future research direction for expanding datasets to include richer multimodal information.

[R3C3]

The experimental section does not yield particularly new conclusions. Generally speaking, it is not surprising that the addition of new modalities, especially for robotic manipulation tasks, typically leads to better performance.

We agree with the reviewer's point. Intuitively, adding new modalities should improve model performance for manipulation tasks. However, prior works have not fully explored this area. The main contribution of our work lies in building a robust system for multimodal settings, providing a benchmark to evaluate methods in this context, and demonstrating the value and potential of this approach for future research.

[R3C4]

Considering the excellent development of current multimodal pre-trained models, and that the models in this paper are trained from scratch, is it feasible to incorporate existing pre-trained models such as text-image, audio-image, image-point cloud, and text-point cloud models into the current training framework?

The challenge here is that an embodied agent needs to observe and interact with the world using different modalities. Not only do we need to manage diverse modalities for tasks such as observation and task specification, but we also need to align these two components together. Currently, no work addresses this comprehensive challenge, and it is infeasible to incorporate existing pre-trained models to resolve it.

We thank reviewer for the comments and reference reviewer with identifier QY6x as R2. Comment n of reviewr m is denoted as QY6x.

[R2C1]

The performance of AnyPolicy looks very similar to EmbodiedGPT in the Franka kitchen setting in Figure 4. The Figure 4 caption appears to be wrong

Thank you for pointing out our mistake. The caption is indeed incorrect, as Any2Policy performs slightly worse than (or on par with) EmbodiedGPT on two tasks. However, we respectfully disagree that the overall performance of Any2Policy is similar to EmbodiedGPT. For the Franka Kitchen experiments, we demonstrate that:

1. For challenging tasks, such as Micro Right and Cabinet Right, our method significantly outperforms EmbodiedGPT.
2. In low-data scenarios (e.g., 10 demonstrations), Any2Policy achieves much better performance than EmbodiedGPT and other baselines.
3. Despite being developed to handle multiple modalities, our method outperforms those trained specifically for a single modality.

[R2C2]

The experiments of FrankaKitchen is an easy benchmark.

We agree that FrankaKitchen is an easy benchmark and may introduce bias in evaluation. Therefore, we have provided further evaluation on Meta-World (results are included in the rebuttal), ManiSkill-2, and real robots. To further address the reviewers' concerns, we conducted additional experiments on RLBench and Adroit, which include more challenging tasks with a 24 DoF ShadowHand and a 4 DoF arm. All experiments were conducted with 200 demonstrations.

We provide multiple baselines, including Diffusion Policy, IBC, EmbodiedGPT, and also present error bars. We ran each experiment with 3 seeds and evaluated 200 episodes every 200 training epochs. We computed the average of the highest 5 success rates, which is a typical setting in this simulation. Below are the experimental results.

We demonstrate that even on challenging benchmarks, our method consistently outperforms state-of-the-art baselines.

[R2C3]

Metaworld Results are missing.

We thank Reviewer 4 for the careful review and for pointing out the missing experimental results. We sincerely apologize for this oversight. Due to limited space, please see [R4C3] for details.

[R2C4]

Name Roboany is not used anywhere else.

We thank R4 for careful reviews and for pointing out this typo. We will include this in the revision.

[R2C5]

Can there be more clarification around how the ablation of no embodied alignment module works?

We appreciate the reviewer's concern regarding the details of our ablation study. We have provided a detailed illustration of our methodology in lines 254-258. Specifically, we use MLP layers to align the observation tokens with the instruction tokens, ensuring the dimensions are correct. Additionally, we employ TokenLearner to reduce the number of tokens and maintain consistency, even when some modalities are missing.

[R2C6]

Is AnyPolicy trained on data from all benchmarks or is a separate policy trained per benchmark. In other words, is the multi-task policy multi-task over all tasks in a benchmark, or all tasks in all benchmarks?

In our experiments, for each benchmark, including real-world scenarios, our model is trained on all collected tasks to perform multi-tasking. This is a typical setting in robot learning, as seen in works like Diffusion Policy (RSS '23) and EmbodiedGPT (NeurIPS '23).

[R2C7]

How does the modality-specific model work, what is its architecture compared to AnyPolicy?

Due to our systematic design, our method can naturally handle missing modalities. The architecture of our modality-specific model is identical to AnyPolicy. During inference, we pass empty tokens to account for the missing modalities.

[R2C8]

Since PickCube only needs to place a cube on a random place, which is much easier than StackCube, why does StackCube have such a higher success rate than PickCube?

We thank Reviewer 4 for their careful review and for pointing out this observation. To explain, first of all, PickCube requires picking up a cube and placing it at a specific goal position. This task is not easier than StackCube. For instance, as shown in Table 3 of the ManiSkill-2 paper, the success rate for StackCube is higher than for PickCube when using point cloud observations. Secondly, since our method is trained in a multi-task fashion (as opposed to the single-task training in ManiSkill-2), the learning behavior could differ from that of a single-task policy.

[R2C9]

How many demonstrations are used for Maniskill2?

We used 1K demonstrations for all tasks for Maniskill2.

[R2C10]

How is R3M used in the Video-Image results?

For R3M, we select the keyframe with images of resolution $224 \times 224$ as input and pass to R3M. All other settings are kept the same as the default R3M.

We thank you for the precious review time and valuable comments. We have provided corresponding responses, which we believe have covered your concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work.

We thank reviewer for the comments and reference reviewer with identifier h8z5 as R1. Comment n of reviewr m is denoted as RmCn.

[R1C1]

More details on the datasets. We concur with the reviewer's suggestion to provide additional details in our paper. To address this, we have included in the Appendix examples of 8 tasks from our real-world dataset. This dataset comprises 20 short-horizon tasks, such as pick-and-place [item], and 10 medium-to-long-horizon tasks, which involve multiple steps to complete (e.g., opening a drawer, retrieving an item, placing it back, and closing the drawer). Our setup features a variety of real-world objects, including microwaves, drawers, laptops, boxes, cubes, bottles, etc. These objects range from deformable to rigid and articulated, encompassing a wide array of real-world scenarios. We will provide a detailed summary of our dataset in the revised manuscript.

[R1C2]

Extensive comparative analysis would be beneficial, such as RT-1/RT-2/RT-X, Octo, OpenVLA.

We appreciate the reviewer for highlighting these issues. Since RT-2/RT-X are not open-sourced, and OpenVLA is released recently which we do not have time to implement in our environment, we provide additional results of Octo and RT-1 in our real world experiments. Given the limited time available for the rebuttal, we conducted experiments on four selected tasks. The same training data were used for all methods. For Octo, we present two versions: one that uses pretrained weights from the Open-X framework, and another that is trained from scratch using our dataset. Both versions were evaluated 10 times to calculate the average success rate for each method. Consistent training hyperparameters, including learning rate, learning rate scheduler, and training epochs, were maintained across all methods.

We demonstrate that Octo, pre-trained on the Open-X framework, achieves comparable results to our method on two tasks. However, for more challenging tasks, our method outperforms the pre-trained Octo. When Octo is trained with the same amount of data as our method, our approach achieves a significantly higher success rate.

[R1C3]

The author could investigate the impact of different encoder architectures or the role of specific modalities in greater detail.

We thank the reviewer for raising these points. Unfortunately, due to the limited time available during the rebuttal period, we were unable to complete these experiments. However, we plan to conduct additional experiments in the future, focusing on using different encoders and replacing the policy head.