The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning

#1: The tactile simulation employed in this study lacks the desired accuracy. Moreover, the introduced method hasn't been empirically tested or combined with tactile sensors in real-world experiments.

MuJoCo’s contact force optimization is equivalent to a penalty-based approach, which is essentially the same used to compute force maps in [A] and [B], with those works showing zero-shot transfer to the real-world for the specific tasks considered. Additionally, there are other remarkable examples of sim-to-real transfer using MuJoCo [C], [D]. Based on such evidence, we would like to highlight that MuJoCo’s touch grid is currently the most realistic general-purpose simulator of tactile maps, especially given the wide range of possibilities for proper system identification. We expect this very important clarification to change the reviewer’s evaluation of the contribution.

While the main focus of our work was on studying how to extract stronger representations when visual and tactile data are available, we also showed some promising direction (e.g., improvement of vision-only policies with tactile-enabled representation training) for real-world deployment, which we leave as future work.

#2: The ViT model, as utilized, is computationally intensive and has a propensity to overfit to specific problems. The potential benefits of ViT's self-supervised pre-training techniques, such as MAE, ought to be realized across a diverse range of datasets or environments.

Thanks for suggesting this comparison! We indeed compared to MVP [E], which is basically relying on ViT representations pretrained across datasets, and on an additional baseline using CLIP [F] visual encoder. As shown in Figure 10b of the updated appendix, both M3L and its vision-only MAE variation outperform MVP and CLIP representations on the in-domain learning task (tactile insertion). We believe this is due to the fact that we let the representations further adapt during task learning. Note that training such MVP and CLIP baselines was even more computationally intensive than training our all-in-one approach, due to the inference of the much larger MVP (22M) and CLIP (88M) encoders, compared to the small ViT used in M3L (5M).

We omit generalization results on the new baselines due to the low success rates on the in-domain task.

#3: There's an inconsistency in the results between the baseline and the proposed method across various tasks. It would be advisable to extend the scope by implementing over 10 distinct tasks and undertaking comprehensive statistical analyses to more convincingly demonstrate the method's effectiveness.

Our results show that 1) vision+touch policies perform best when investigating zero-shot generalization, 2) training representations with vision+touch also improve vision policies, 3) representation learning approaches outperform end-to-end approaches when learning in-domain. 1) and 2) are shown in Figure 5 in the paper, while 3) is shown in Figure 6 in the paper. These results are consistent across all tasks, so we disagree with the reviewer comment.

As the reviewer pointed out in #1, the use of tactile simulation is still at an early stage. While we agree that evaluation on a large variety of tasks would be meaningful, note that our three environments are the first available implementations of RL environments with availability of both visual and high-resolution tactile sensing on a general-purpose like MuJoCo. This required carefully designing and adapting each of the tasks for this work, as opposed to other fields in RL that have the possibility of benchmarking on existing environments. To give an idea of the process involved, to increase the number of contact points on the Shadow Hand, we needed to split each collision mesh in a much higher number of meshes making sure that each of these was still convex (as required by MuJoCo), which is a non-trivial process for each robot embodiment. Ours is an effort in this direction, and open-sourcing such environments will hopefully give the opportunity to fellow researchers to further extend the benchmarking opportunities in the area. As the reviewer pointed out, however, we purposely focused on tasks that are very different to cover a wider spectrum of applications. We are currently working on further extending the number of tasks.

Additional tasks

We performed additional experiments on two new in-hand rotation tasks, namely, egg and pen rotation tasks.

These tasks confirm that M3L outperforms the baselines in terms of generalization, particularly by a large margin when altering the pen color in the corresponding rotation task. Please find the corresponding discussion in Appendix I of the paper. We hope that this clarifies your related concern. We look forward to hearing your feedback before the end of the rebuttal period.

#1: The paper ignores several different related works [A], [B], [C], both in terms of tactile sensors used for robotics, and machine learning approaches that use touch as a modality for manipulation tasks.

Thank you for pointing out relevant works. We have included these works in our updated related work section and also compared our method with [A] in Figure 10, Appendix. However, please note that [A] is a contemporaneous work according to ICLR guidelines, having been arxived only 10 days prior to the submission of this paper.

#2: Is it possible to have more than three simulation tasks?

We agree that evaluation on a large variety of tasks would be meaningful and thus, we are currently working on further extending the number of tasks.

We want to emphasize that our three environments are the first RL environments both with visual and high-resolution tactile sensing on a general-purpose physics engine like MuJoCo. This required carefully designing and adapting each of the tasks for this work, as opposed to other fields in RL that have the possibility of benchmarking on existing environments. To give an idea of the process involved, to increase the number of contact points on the Shadow Hand, we needed to split each collision mesh in a much higher number of meshes making sure that each of these was still convex (as required by MuJoCo), which is a non-trivial process for each robot embodiment. Ours is an effort in this direction, and open-sourcing such environments will hopefully give the opportunity to fellow researchers to further extend the benchmarking opportunities in the area. As the reviewer pointed out, however, we purposely focused on tasks that are very different amon each other to cover a wider spectrum of applications.

#3: There are no external baselines compared against. All the comparisons are with variants of the proposed M3L algorithm.

As suggested by the reviewer, we have included comparisons to other approaches in Figure 10 of the updated version: contact location-based tactile representations [A] and vision-only pre-trained representations (MVP [D], CLIP [E]).

While [A] is contemporaneous work, we found it interesting to compare with a similar transformer architecture, trained in a single stage from vision and touch. In particular, we also investigate the tactile representation used in [A], that is, discretized contact locations. In contrast, our approach uses intensity and directions of spatially distributed force vectors (force maps). The results are shown in Figure 10a of the updated appendix, with the discretized contact location approach outperformed by both our M3L and end-to-end baselines even for the in-domain learning task (tactile insertion).

We also added comparisons to baselines based on vision-only representations extracted through MVP [D] and CLIP [E] visual encoders, kept frozen during PPO training. Note that both encoders are much larger than the one used in M3L (5M for M3L, 22M for MVP, and 88M for CLIP), leading to considerably slower training. The results are shown in Figure 10b of the updated appendix, with both M3L and its vision-only MAE variation outperforming MVP and CLIP representation on the in-domain learning task (tactile insertion).

We omit generalization results on the new baselines due to the low success rates on the in-domain task.

#4: The experiments show only very weak generalization to minor changes in the object (in particular mass, damping, friction variations). It is important to show generalization to actually different objects.

In the tactile insertion task, we indeed test the generalization capability to different objects, as illustrated in Figure 4 and Figure 8.

Moreover, we would argue that other perturbations in the objects are not minor, i.e., 10x friction and damping, 2x weight. In addition, we test on different object poses in Door Opening and camera poses in In-Hand Rotation.

#5: Real-World tactile data is usually very noisy, but simulation data is clean. Is there anything that will need to be changed in the approach for bridging this gap or making the representations more robust to noise, when real-world deployment is attempted?

Thanks for the great suggestion! We added a noise robustness analysis in Figure 12 of the updated appendix, where we add zero-mean Gaussian noise to both visual and tactile data at inference time. The figure shows the performance of M3L vs the end-to-end for increasing noise variance, highlighting how M3L success rates do not degrade up to a reasonable degree of disturbances and also consistently outperforms the end-to-end approach also in terms of robustness.

Also, note that force maps have already been shown to be transferable to the real-world, either by estimating them [F] or by mapping them to optical flow [G].

Additional tasks

We performed additional experiments on two new in-hand rotation tasks, namely, egg and pen rotation tasks.

These tasks confirm that M3L outperforms the baselines in terms of generalization, particularly by a large margin when altering the pen color in the corresponding rotation task. Please find the corresponding discussion in Appendix I of the paper. We hope that this clarifies your related concern. We look forward to hearing your feedback before the end of the rebuttal period.

#1: Manipulation with tactile sensor and visual input has been evaluated in the real world for different tasks [A]. To actually validate the performance of the proposed method, the real world evaluation is required. Especially for noisy tactile sensory information.

We would like to highlight that [A] is a contemporaneous work according to ICLR guidelines, having been arxived only 10 days prior to the submission of this paper. We are however happy to cite and compare to [A] (see also point #2).

[A] trained an end-to-end baseline (in two privileged learning stages) and achieved sim-to-real transfer through large-scale GPU parallelization and domain randomization. The focus of our work was instead on studying how to extract stronger representations when visual and tactile data are available. We also showed some promising direction (e.g., improvement of vision-only policies with tactile-enabled representation training) for real-world deployment, which we leave as future work.

Regarding the robustness to noise, please see our response to #7.

#2: Additional baselines need to be compared.

Thank you for your suggestion. We have compared our method with additional baselines: contact location-based tactile representations [A] and vision-only pre-trained representations (MVP [B], CLIP [C]).

Figure 10a of the updated appendix shows that both our M3L and end-to-end baselines outperform the contemporaneous work that uses discretized contact location approach [A] for the in-domain learning task (tactile insertion), confirming the advantages of using high-resolution force maps as opposed to low dimensional contact quantities.

The pre-trained representations, MVP [B] and CLIP [C], also show slower training curves in Figure 10b. Note that the encoders used in MVP (22M parameters) and CLIP (88M parameters) are much larger than our M3L (5M parameters).

We omit generalization results on the new baselines due to the low success rates on the in-domain task.

#3: Missing tactile-only baseline.

This is a great point! We have run additional experiments with the tactile-only baseline. Interestingly, although the Door Opening task can be fully solved without vision, M3L (vision + tactile) achieves much faster and better learning results than the tactile-only baseline. The benefit of M3L could come from (1) our multimodal representation learning and (2) the use of visual observations. We included this result in Figure 11 of the updated paper.

Note that the insertion and hand tasks are misspecified when vision is missing, because of missing target and incomplete state information. We ran such a baseline on the insertion task to confirm such a claim (see Figure 11).

#4: Adding proprioception information.

We agree that proprioception (and any other sensing modalities) can certainly help robotic manipulation. However, this study aimed to isolate the contributions of vision and touch for representation learning, which is why we focus on these two modalities. We leave integration of proprioception and other modalities into our pipeline as future work.

#5: Out of three tasks, the proposed framework only outperforms vision-only baselines in In-hand rotation tasks.

Our results show that 1) vision+touch policies perform best when investigating zero-shot generalization, 2) training representations with vision+touch also improve vision policies, 3) representation learning approaches outperform end-to-end approaches when learning in-domain. 1) and 2) are shown in Figure 5 in the paper, while 3) is shown in Figure 6 in the paper. These results are consistent across all tasks, so we disagree with the reviewer comment. Please let us know if you have any other concerns regarding this.

#6: For generalization experiments, randomization on physical parameters might not be enough.

We do not only have randomization on physical parameters, but also variation of object pose (door) and camera pose (hand) in the generalization experiments. On all such randomizations, M3L shows the smallest performance degradation compared to baselines.

#7: Can the learned model show the same level of performance if there is noise in visual observation or tactile reading?

Thanks for the great suggestion! We added a noise robustness analysis in Figure 12 of the update appendix, where we add zero-mean Gaussian noise to both visual and tactile data at inference time. The figure shows the performance of M3L vs the end-to-end for increasing noise variance, highlighting how M3L success rates do not degrade up to a reasonable degree of disturbances and also how M3L consistently outperforms the end-to-end approach in terms of robustness.