AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors

We would like to express our sincere appreciation for your insightful comments! We are very excited to receive your encouraging recognition of our "good contribution to the community", "novel framework" and "comprehensive experiments".

Though vision based tactile sensors are very popular in robot learning community, it might not be the ultimate solution that people agree on. Tactile perception are more than images. Humans feel temperature, texture, shape, vibrations, etc. when interacting with objects. This is still an open question and is out of scope of this paper. But it will be interesting to see if how other tactile sensors align or match with the vision-based tactile sensors.

Thank you for the insightful suggestion. We agree that tactile perception is not limited to images. Some tactile properties, such as temperature and torque, are difficult to obtain from tactile images alone, requiring the use of other types of tactile sensors. This issue presents challenges from both hardware and algorithmic perspectives.

From a hardware perspective, an ideal tactile sensor should be capable of gathering various types of tactile information, effectively integrating multiple existing tactile sensors into a single unit. This may be very challenging, and a more practical solution might involve equipping different fingers of a robotic hand with different types of sensors. This would allow for the simultaneous collection of diverse tactile data, maximizing the range of information captured.

From an algorithmic perspective, when vision-based tactile sensors are replaced with other types of tactile sensors (e.g., tactile sensor arrays), the multi-sensor data alignment method proposed in this paper can still be applied. Aligned data can then be used to perform alignment or to distill knowledge from the visuo-tactile model to models for other types of tactile sensors. For lower-resolution tactile sensors, the aligned data can facilitate tactile super-resolution learning, enabling knowledge transfer from vision-based tactile sensor models to enhance their performance.

If both vision-based tactile sensors and other tactile sensors (e.g., those capturing temperature or other non-visual properties) are used simultaneously, a possible approach is to fuse their outputs into a unified, comprehensive tactile feature. This enriched representation can then be aligned with other modalities in a unified manner.

We have included the related discussion in the revised version. Thank you again for your insightful comments!

I think there can be some study on the effect from the stage 1, the pixel level learning, on the downstream tasks. How does learning the pixel-level details help the downstream tasks? I think material classification and grasp prediction might not need that level of detail to achieve good performance.

Thank you for raising this point. In fact, we have already provided a detailed analysis of how the modules and the two stages influence the performance of our method in Table 6 of the appendix. By comparing the rows removing stage 1 and stage 2, we observe that learning fine-grained tactile details can indeed improve performance on the TAG (material) and Feel (grasp) datasets. However, this improvement is much smaller than the gain achieved by learning coarse-grained tactile features that are more closely related to the tasks. Thank you again for your comment.

In data collection process, how do you localize the contact position in world coordinate? How to ensure the contact position is the same for different sensors?

Thank you for raising this important point. The movable end on our calibration platform can be programmed to move at a specified speed to a designated position within the coordinate system defined by the base. Therefore, as long as we pre-measure the relative positions of the centers of the four sensor surfaces within the container and compensate for the relative positions during each set of data collection, we can ensure that all four sensors make contact with the object from the same initial position and at the same speed, thereby achieving both temporal and spatial alignment. We have provided a more detailed explanation of the data collection in the revised version. Thank you again for your reminder!

We want to thank you for the helpful comments and valuable feedback! We appreciate the comments that the paper is well-motivated and well-written, and the proposed method is creative.

Given you claimed the model has the ability to extract sensor-agnostic features from unseen sensors, it would be more convincing to show some downstream task performance on the new sensor, aside from qualitatively showing t-SNE visualization.

Thank you for your valuable comments! In fact, we have already compared and discussed the performance of our model on unseen sensor datasets in Section 5.5. We conducted comparisons with the previous SOTA multi-sensor model, UniTouch, on two unseen sensor datasets: ObjectFolder 1.0 and ObjectFolder 2.0. As shown in Table 3, the UltraTouch trained on the same data as Unitouch outperforms it on both datasets, demonstrating the static perception capability of our method on unseen sensors.

To further validate our model's ability to extract sensor-agnostic features and demonstrate the value of our proposed dataset, we trained the model on our collected fine-grained spatio-temporal aligned data for cross-sensor generation. Specifically, we trained models to generate the 20x20 force fields captured by the Tac3D sensor from DIGIT and GelSight Mini data. We compared the performance of our model with the T3 model, which used more training data than ours (3.08M compared to our 2.48M) for pretraining. We treat this task as a regression task and use an MLP to reconstruct the force field based on the features extracted by the encoder. To further ensure fairness, we also removed the overlapping portions of the coarse-grained aligned data from the training data that overlapped with this dataset. Note that Tac3D is an unseen sensor for both of the models. We use mean-square error (MSE) (↓) between the generated data and the ground truth as the metric. The results are as follows:

Table II: Performance comparison with T3 on the cross-sensor generation task using the fine-grained spatio-temporal aligned data (MSE as metric).

The results show that our method outperforms T3 in terms of generation quality. This demonstrates the superior ability of our model to extract sensor-agnostic features. This supports our motivation to obtain a unified tactile multi-sensor representation that is applicable to a variety of tasks and sensors. We also look forward to further exploring a wider range of task types in future work. In particular, using unseen sensors for dynamic perception tasks could be an interesting challenge. We have added the experimental results and related discussions to the revised version. Thank you again for your constructive suggestions!

Regarding Fig 7 and 8, is there a significant gap between GPT-4o outputs for OF-Real and TVL/SSVTP, given you didn't use tactile data for the latter? If so, will this affect your training? If not, do you think GPT-4o is making use of tactile input, given it may not be capable of analyzing images from tactile sensors? Can you show some raw output examples?

Thank you for your valuable questions. We additionally input tactile images when using GPT-4o to annotate the OF Real dataset because the visual images in the OF Real dataset were captured by two cameras positioned relatively far from the objects. As a result, the camera viewpoints are sometimes obstructed or fail to capture the tactile details at the contact points effectively. As illustrated in Figure 7 of the appendix, the images captured by both cameras fail to reveal the protrusions on the lid's surface. Without the guidance of tactile images, GPT-4o might incorrectly assume that the contact point on the lid’s surface is smooth, resulting in inaccurate annotations. This issue is not present in other datasets, allowing high-quality annotations to be achieved using only visual images. Therefore, there is no significant difference between the annotations of the OF Real dataset and those of other datasets. During the annotation process of the OF Real dataset, we found that if the collecting method and principles behind the tactile images are clearly outlined in the prompt, GPT-4o can grasp the concept of tactile images to some extent, even if it has never encountered them before. This is because GPT-4o can interpret images, and data from vision-based tactile sensors are essentially high-quality images. We have added the corresponding raw outputs from GPT-4o for all the prompt examples in the appendix in the revised version. Thank you again for your valuable questions!

We want to thank you for the helpful comments and valuable feedback! We appreciate your recognition of the value of our dataset, the novelty of our framework and the comprehensive experiments.

As is shown in A.4, only 3 frames are used for each tactile video clip. This seems to be a surprisingly small number of frames, and my concern is that this might not be sufficient for extracting some of the fine-grained tactile information (e.g., hardness). One possible ablation experiment would be training the model with different video length and evaluate the performance. See Questions section for more details.

Referring to my first concern in Weaknesses section, I'm curious about the computational cost of training with tactile videos. It's common that the frame rate of the video being limited due to data loading/ computation limits, but only using 3 frames still seems too few to me. My guess is that using more frames may lead to better dynamic perception ability, and I'm wondering if an ablation experiment on this parameter can be possible?

Thank you for your insightful comments. This is a valuable question, as in the real world, the complete process of touching an object can take several seconds or even tens of seconds. Ensuring the model can comprehend an entire tactile video presents a significant challenge. Current large-scale video understanding models, such as Video-LLaMA [1], often process tens or even hundreds of frames as input, encoding them into tokens. However, this comes at the cost of generating very long token sequences, which significantly increase computational overhead and inference time. The tactile modality is frequently used in fine-grained manipulation tasks that demand high real-time performance, which imposes strict requirements on the model's inference speed. As a result, models that rely on long frame sequences are challenging to apply in real-time dynamic perception tasks. Moreover, since touch actions are typically performed at high speeds, even a sequence of three continual frames (equivalent to 0.1 seconds for a DIGIT sensor with a frequency of approximately 30Hz) can exhibit noticeable changes. We anticipated these challenges and, as a result, chose to use a sequence of three continual frames as the input format for tactile videos. This approach also enables the understanding of longer videos by selecting multiple 3-frame segments and either concatenating or summing their features, similar to ImageBind [2]. We also agree that using more frames may lead to better perception performance, but this is essentially a trade-off between performance and both computational cost and reference speed. Since the current model which uses three frames as input, requires two days to complete training, and using more frames would take even longer, we are unable to provide results for an ablation study on the number of frames within a short time We have expanded this discussion in the revised version and we look forward to explore the related experiments in future work. Thank you again for your insightful comments!

[1] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 543–553, 2023.

[2] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180–15190, 2023.

Besides TacQuad, some of the other datasets also contain tactile videos, such as TAG and OF Real. I wonder if they are already treated as videos during the training of UltraTouch, if not, maybe it can further improve the model performance if they are used as videos.

Thank you for raising this point. In fact, except for the Feel, OBF 2.0, and SSVTP datasets, which do not contain continuous video frames, all other datasets include videos, and we used these tactile videos during training. The statistics of the training datasets we used are detailed in Table 5 in the appendix. The ablation experiments in Table 6 also show that training the model with tactile videos for dynamic perception can benefit the model's static perception abilities as well. This further supports the motivation behind our proposed method of learning unified multi-sensor representations from both static and dynamic perspectives. We have clarified this point in the revised version. Thank you again for your comment!

We would like to express our sincere appreciation for your insightful comments. We appreciate your recognition of the novelty of our proposed method combining static and dynamic approaches, as well as the experimental setup and the effort put into collecting the tactile datasets.

Although the author's idea of the aligned multi-sensor dataset, the author focuses to optical-based tactile sensors only, which simplifies the problem with similar tactile modalities(tactile image).

Thank you for your insightful comment. We agree that vision-based tactile sensors are just one branch of tactile sensors, and many other types of tactile sensors are also being used. However, we would like to emphasize that vision-based tactile sensors we focus on are a widely studied and utilized category of tactile sensors, and have recently received substantial attention in both academia [1] and industry [2]. Our motivation is that even within vision-based tactile sensors, there is considerable variety and a low level of standardization. As a result, different vision-based tactile sensors may exhibit differences when perceiving the same tactile information. Consequently, different vision-based tactile sensors may exhibit variations when perceiving the same tactile information. To obtain tactile representations adaptable to a wide array of tasks and sensors, we propose to learn unified multi-sensor representations from a fresh perspective that incorporates both static and dynamic perception. We think that this is already a daunting and meaningful task.

We also believe that other similar types of tactile sensors can share a unified multi-sensor representation space, such as the widely used tactile sensor arrays, which can be considered a low-resolution version of vision-based tactile sensors to some extent. We look forward to further exploring other tactile sensors and their representations in future work, contributing to the continued advancement of the tactile community. We will include the related discussion in the revised version. Thank you again for the insightful comment!

[1] Sudharshan Suresh, Haozhi Qi, Tingfan Wu, Taosha Fan, Luis Pineda, Mike Lambeta, Jitendra Malik, Mrinal Kalakrishnan, Roberto Calandra, Michael Kaess, Joseph Ortiz, and Mustafa Mukadam. Neuralfeels with neural fields: Visuotactile perception for in-hand manipulation. Science Robotics, 9(96):eadl0628, 2024. doi:10.1126/scirobotics.adl0628.

[2] Mike Lambeta, Tingfan Wu, Ali Sengul, Victoria Rose Most, Nolan Black, Kevin Sawyer, Romeo Mercado, Haozhi Qi, Alexander Sohn, Byron Taylor, et al. Digitizing touch with an artificial multimodal fingertip. arXiv preprint arXiv:2411.02479, 2024.

No limitations and future work are described in the main paper.

Thank you for your constructive suggestions! Due to space constraints, we could not fully discuss the limitations of our method earlier. Here, we discuss some potential limitations of our work and propose corresponding solutions for future work:

1. Compared to all the training data, the scale of the TacQuad dataset we have currently collected is still somewhat limited. Capturing the immense variety of object types within a single dataset is challenging in a limited amount of time. Fortunately, the coarse-grained spatial alignment data collection method we propose has the potential to scale up, as data collection can be performed manually without the need for precise alignment. Fine-grained data collection can also be expanded by replicating the calibration platform and increasing manpower. We plan to grow our team to scale up the dataset and enhance object diversity in future work.
2. The types of sensors considered are relatively limited. We have made every effort to collect all available vision-based tactile sensors around us, yet we were only able to include four different types. Moreover, we did not explore the differences between individual sensors of the same type or address issues such as gel damage. Moving forward, we aim to expand our dataset and increase the diversity of sensors through collaborative data collection across multiple laboratories.
3. The scope of tasks for dynamic tactile perception is currently limited. In this work, we validated the dynamic perception capabilities of our model on a single real-world manipulation task: pouring. We hope to explore more challenging and interesting dynamic perception tasks in future work. Additionally, beyond real-world manipulation tasks, studying tactile video understanding—particularly fine-grained dynamic tactile understanding that includes direction and action descriptions—is also an interesting direction to explore.

We have incorporated these discussions in the revised version. Thank you again for your valuable feedback!