Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation

Thanks for your time and valuable comments. We address the questions below:

Using texture reterival from a normal map database

We agree with the reviewer that our work can be potentially extended to other high-resolution geometrical texture data such as the one from a normal map database. In this work, we focus on the data acquired from tactile sensors, due to several reasons. First, the normal map database could be limited in terms of texture variation, e.g., different avocados, as it often contains a single or very few normal maps per object. Second, the normal map data is sometimes hand-crafted rather than captured from real data, making it less realistic. Third, when the user has a particular texture in mind to synthesize, e.g., customizing the plush toy with the fabric of their favorite sweater. In these regards, we believe high-resolution tactile sensors provide a quick and scalable way to capture the precise textures of real objects and, therefore, will be a common way to capture surface textures, in the near future.

Moreover, in the future, we can extend the work of using tactile feedback to model the physical properties, such as the hardness of the material, as part of the object rendering. We thank the reviewer for pointing this out and will add this discussion in the revision.

Generation time of the multi-part generation

As mentioned in the supplementary material, we train all models on A6000 GPUs and each experiment takes about 10 mins and 20G VRAM to run. To be specific, single-part experiment takes about 6 mins and multi-part experiment takes about 8 mins to run.

Experiment of using real Image and real corresponding tactile as input

We show one example result of using a real image and corresponding captured tactile signals as input in Figure 3 of the attached PDF. Our method can recover the color and geometric details reasonably well.

Thanks for your questions and comments. We would like to clarify that in this work, our main contribution is leveraging tactile sensing to enhance geometric details for 3D generation tasks. (1) We are the first to leverage tactile sensing to synthesize high-fidelity geometric details for 3D generation. (2) We present a new technique to learn high-quality 3D texture fields with aligned visual and geometry textures, with a multi-part extension that allows us to synthesize multiple textures across various regions. (3) Our experiments show that our method outperforms existing text-to-3D and image-to-3D methods.

Comparison with the different setup of reconstruction pipeline

We agree that methods like Neuralangelo (Li, et al., 2023) and PermutoSDF (Rosu, et al., 2023) can recover highly detailed geometric information. However, these methods focus on 3D reconstruction tasks and require sufficient data to achieve high-resolution reconstruction, e.g., “50 images” for an object and “~200-1000 images” for a larger scene. In our setup, we focus on a different task of text-driven 3D generation. We make use of a similar 3D representation, i.e., a multi-resolution 3D hash grid, as a texture field and strive to generate highly detailed 3D assets with minimal input, a text prompt / single-view image and a single tactile patch.

Local geometric intricacies captured by Tactile data

The GelSight mini sensor we use to collect the TouchTexture dataset has a sensing area of 21mm x 25mm. The normal maps and height maps in Figure 3 of the main paper and the supporting materials show the tactile data for a single touch, cropped to 18mm x 24mm in physical scale and 240 x 320 in pixels, which corresponds to a tiny patch as shown in Figure 2 of the main paper. We refer to the “submillimeter scale” geometry captured by the sensor as “local geometric intricacies”.

More Diverse Results

We show more complex and diverse results in Figure 4 and Figure 5 of the attached PDF. Our method is compatible with most pipelines that generate colored meshes, and can enhance the geometric details of their output. With different backbones such as RichDreamer (Qiu et al., 2024) and InstantMesh (Xu et al., 2024), our method is able to generate diverse, in-the-wild objects. These results showcase the generalizability of our method. We will include more examples in the revised draft.

Ablation studies

In the main paper, Figures 9,10, and 11 illustrate the ablation studies and the analysis is provided in Line 238-250. Directly applying a normal texture map to a base mesh without joint optimization may introduce unnatural appearance, as the additional normal texture could be conflicting with the original albedo map, as shown in the misaligned strawberry seeds in Figure 9. Figure 10 shows that using the raw tactile data without proposed preprocessing produces much more flattened textures. This is because low-frequency deformation of the gel pad would dominate the tactile signal, reduce the signal-to-noise ratio, and degrade the synthesized geometric details. We also ablate the tactile input, and the example results are shown in Figure 11. Removing tactile input produces overly smooth meshes, as a text prompt could not provide sufficient guidance to geometric details at the millimeter scale. Figure 2 in the attached PDF also shows the ablation of tactile loss on a part of the object, demonstrating that the diffusion prior can infer some geometric variation to a certain level but is not capable of generating high-fidelity regular and consistent textures. We are happy to conduct additional ablation studies if there are specific aspects the reviewers would like us to investigate.

Thank you for your encouraging comments and feeback. We answer each question as below:

Is the proposed method compatible with a purely text-to-3D pipeline?

Yes, our method is compatible with purely text-to-3D backbones, such as RichDreamer (Qiu, et al., 2024). In our method, we generate our base mesh by first generating an image from the input text prompt using SDXL (Podell, et al., 2023) and then running Wonder3D (Long, et al., 2023), which is a ‘text-to-image-to-3D’ pipeline. However, our method is also compatible with most pipelines that output colored meshes. As suggested, we used a state-of-the-art text-to-3D method RichDreamer, and integrated it into our method. Specifically, we generate the base mesh using RichDreamer, and finetune the albedo and normal UV map using our proposed tactile matching loss, diffusion-based refinement loss, and regularization loss. Example results of both RichDreamer and our full method are shown in Figure 4 in the attached PDF. As shown in Figure 4, our method is able to generate 3D objects corresponding to the text prompt while adding high-fidelity geometric details from tactile inputs.

More advanced image-to-3D baseline and more complex objects

Similarly, we can integrate InstantMesh (Xu, et al., 2024), a more recent image-to-3D method, into our method. Figure 4 and Figure 5 in the attached PDF demonstrate the results for more complex objects using the new backbones of RichDreamer and InstantMesh. Our method enhances the geometric details since the tactile information provides guidance of finer scale than the resolution of the base mesh generated by the backbones. These results highlight the compatibility of our method with different backbones, and its generalizability to complex objects. We will include them in the revision.

How to select tactile input and how to ensure the tactile details are compatible with the object?

The tactile input can be selected either for realism or creativity. For generating realistic outputs, we would choose the tactile input similar to the object, e.g., using a tactile patch collected from real strawberry to generate a strawberry mesh. Otherwise, if we aim for creativity, we can choose any tactile texture we want, as the various coffee cups show in Figure 6 of the main paper. Since the tactile details are of millimeter scale, they are added upon the coarse geometry as a normal UV map, which is compatible with the base mesh. To further ensure alignment between the color and tactile details, we jointly optimize the albedo and normal maps using the refinement loss.

What are the efficiency and success rate of the attention-based segmentation?

Since we leverage the same diffusion model to compute the attention maps for segmentation, it is more memory efficient than using additional components such as off-the-shelf segmentors like SAM. In terms of running time, a single-part experiment takes about 6 mins and a multi-part experiment takes about 8 mins.

To quantitatively evaluate our segmentor, we manually segment 3 meshes to obtain ground-truth segmentation masks. We then run our segmentor on 100 renderings for each mesh. In terms of the metric, we calculate the IoU between our predicted masks and the labeled masks and also compute the accuracy of predicted labels. Our diffusion-based segmentor reaches an average IoU of 0.588 and an accuracy of 83.7%, which provides sufficient segmentation for multi-part optimization. Note that our multi-part optimization can partially resolve inaccurate and inconsistent segmentation by aggregating the gradients from different views, so our 2D segmentation does not need to be perfect. We are happy to expand this evaluation in the revision.

Clarification about the color and geometry optimization

The coarse geometry is fixed after the base mesh generation while the local geometric details, defined by normal UV maps, are further refined using tactile information, and thus not derived from the original colors. The color is also optimized simultaneously with the normals using a diffusion-based refinement loss. The optimization could introduce changes of color to align visual and tactile modality, while we add the regularization term so that the refined color maintains consistent with reference color on a larger scale. We balance the loss weight and a single set of parameters works for all our experiments.

Comparison with real objects

We show one example result of using a real image and corresponding captured tactile signals as input in Figure 3 of the attached PDF. Although our method can recover the color and geometric details reasonably well, we think humans can still distinguish between the real and generated objects by comparing with the input real image.

Thank you for your encouraging and insightful comments. We’re pleased that you recognize our setup as “a novel and unexplored task.” Below, we address the individual comments.

Add albedo rendering results

Due to space limit, we omitted the albedo renderings in the main paper. Thanks for pointing it out and we agree that adding albedo renderings helps to demonstrate the quality more clearly. In the Figure 1 of the attached pdf, we show two examples and will add full results in the revision. Notably, our albedo rendering looks natural and contains minimal baked-in lighting effects or geometric details.

Show experiments of textures learned from the diffusion priors, i.e., optimizing $L_{tactile}$ only for the pot and let the diffusion prior decide what the normals for the cactus must look like (or vice versa)

Thanks for your insightful suggestions. We add our experiment results in attached pdf’s Figure 2 for the example “cactus in the pot”, where we keep the text prompt and tactile input unchanged Figure 2(a) has no $L_{tactile}$ for either part; Figure 2(b) and (c) have $L_{tactile}$ for one part, “pot” and “cactus” respectively; Figure 2(d) has $L_{tactile}$ for both parts.

As shown in Figure 2, (b) and (c) contain clear texture for the part with $L_{tactile}$ and synthesize more details for the other parts compared to (a). However, without clear reference, the inferred texture lacks detailed patterns compared to the full results in (d). In short, the texture generated using only diffusion priors appears plausible but tends to look flatter and less detailed.