You Always Recognize Me (YARM): Robust Texture Synthesis Against Multi-View Corruption

We greatly appreciate the reviewers’ thoughtful feedback and insightful suggestions. We have carefully reviewed all the comments and provide our detailed responses below.

A1.Regarding writing issues: Thank you for pointing out these issues! We will correct these errors in the revised version of the paper.

A2.Regarding real-world experiments: Thank you for your question! We fully understand the importance of real-world experiments. However, compared to previous methods (such as Unadv), we have already conducted extensive evaluations of our proposed method on a large dataset containing 40 categories and 400 object instances (whereas Unadv evaluated only 5 objects). We believe this is sufficient to demonstrate the generalization capability of our method. Nevertheless, to further address your concern, we have additionally conducted an experiment on a real-world scene. Specifically, we selected a real scene used in [1]. Since our method requires separating the foreground object, we manually segmented the foreground object from the multi-view images and performed 3D reconstruction and texture optimization accordingly. The experimental results based on ResNet-18 are presented in the table below. The visualization results can be found here https://p.sda1.dev/23/b6df0ef9d63f80fba42e36ce11994fed/real_scene.png

A3.Regarding scalability: Thank you for your question! We believe that large-size images can be downscaled to fit our 3D reconstruction pipeline, and therefore, scalability to larger images is not a major concern in our method. In contrast, images with excessively low resolution may significantly degrade the quality of 3D reconstruction, which could in turn hinder the effectiveness of the optimized textures.

A4.Regarding universal robust textures: Thank you for your question! Indeed, a single object-specific robust texture can only be transferred across different classifiers but cannot be directly transferred to other objects with different shapes, even within the same category. We proposed the concept of universal robust textures because object-specific texture synthesis is an inefficient process. Therefore, our goal is to generate a universal texture for each category, independent of the specific appearance and shape of individual instances. We have validated the feasibility of this idea through preliminary experiments. It is worth noting that the performance of universal robust textures reported in Tables 1 and 2 was evaluated on objects that were not used during training. This demonstrates that these textures indeed provide protection for previously unseen objects within the same category. However, the primary focus of this paper is on object-specific robust textures. As such, we did not devote significant effort to further improve the performance of category-level robust textures, which may explain why their current performance appears unsatisfactory. Nevertheless, we believe that the direction of category-level robust textures is worthy of further exploration. We have also discussed possible ways to improve their performance (see A6 to Reviewer rM6D).

A5.Regarding selection of proxy model: Please refer to A5 to Reviewer rM6D.

A6.Regarding the question about Figure 4: Thank you for your question! We apologize for the lack of clarity in the presentation of Figure 4. Specifically, the first row of Figure 4 shows the original appearance of the objects; the second row shows the appearance after applying object-specific robust texture optimization for each object; and the third row shows the appearance when applying the universal robust texture, generated for the "airplane" category, to objects of different shapes within the same category. As illustrated in Figure 1, we selected 8 objects from the "airplane" category as the training set. We reconstructed voxel grid representations for each of these 8 objects based on their multi-view images. Then, we initialized a random perturbation $\delta$ and, during each iteration of the optimization process, randomly selected one voxel grid to combine with $\delta$ and rendered the corresponding view images. In this way, the resulting category-level robust texture can be directly applied to unseen voxel grid representations of other objects belonging to the "airplane" category, enabling the rendering of corruption-robust views. We hope this explanation helps clarify your concerns regarding the universal robust texture and the content of Figure 4.

We appreciate the reviewers’ thoughtful feedback and provide our detailed responses below.

A1.Regarding the specific differences from prior methods: Thanks for your question! We acknowledge that our work is inspired by Unadv; however, there are clear differences that distinguish our approach from Unadv: 1.Optimization space: Unadv directly optimizes the pixel values of texture images in the 2D image space, while our method performs optimization in the voxel grid space. We believe the high dimensionality of the voxel grid allows for better optimization outcomes. 2. Visual appearance preservation: Directly optimizing texture images in pixel space often results in highly discrete pixel values, which can manifest as noise-like visual artifacts on the object surface. In contrast, the textures generated by our method exhibit a certain degree of local continuity. This advantage is particularly evident when applied to objects without initial textures (see A3). 3.Efficiency: Our method is approximately 100 times faster than Unadv. Unadv's optimization is based on 3D mesh representations combined with a differentiable rendering framework, which typically requires 6 to 7 hours to optimize a single object. In comparison, our approach takes only 4 to 5 minutes to complete the optimization.

A2.Regarding Eq.3: Thanks for your question! We acknowledge that such a situation is theoretically possible; however, the probability is very low. Specifically, for this to happen, the gradient of the $\delta$ term obtained through backpropagation during training would need to remain 0 throughout the entire process. Additionally, it would require that the multi-view images of the original object exhibit strong robustness against all types of corruptions. Furthermore, commonly used optimizers incorporate momentum terms, which help prevent the optimization from getting stuck in such a local optimum. Therefore, we believe this situation is unlikely to occur in practice.

A3.Regarding real-world data and other datasets: The reason why we only used the IM3D dataset in our experiments is that our method requires datasets with specific characteristics: (1) multi-view images along with camera poses to enable 3D reconstruction; and (2) category labels aligned with ImageNet classes to support the classification task. Therefore, datasets commonly used in NeRF-related tasks are not directly applicable to our pipeline. Nevertheless, to demonstrate the generalization ability of our method, we selected 4 objects from the ModelNet40 dataset—originally designed for point cloud classification—and rendered multi-view images of them using Blender for evaluation. Experimental results based on VGG16 is shown below. Visualization results can be found here https://p.sda1.dev/23/b660fbea4c3a6d471effde89e4b4110f/modelnet.png. For real scene results please see A2 to Reviewer mDnf.

A4.Regarding the fairness of the experiments: Thanks for your question! For the baseline methods, we first evaluated their publicly released weights (most of which were pretrained on ImageNet) on the IM3D dataset. However, due to distribution differences, most of these methods exhibited suboptimal performance on IM3D. Therefore, we additionally fine-tuned URIE, VQSA, and DCP on the IM3D dataset to ensure a fair comparison. After fine-tuning, VQSA showed significant performance improvement, URIE exhibited a slight improvement, while DCP showed no noticeable improvement.

A5.Regarding Figure 4: Thanks for your question! In Figure 4, we illustrate the appearance of universal robust texture when applied to multiple instances with different shapes within the same category (airplane). We will add further clarification to the caption of Figure 4 in the revised version and ensure that Figures 1, 2, and 4 are explicitly referenced in the main text.

A6.Regarding the use of ReLU: This design choice was proposed by [1], where the authors claimed that replacing SoftPlus with ReLU better preserves the discontinuities present in real-world signals.

A7.Regarding the IM3D dataset: The IM3D dataset is sourced from [2]. We apologize for the oversight of not citing it in the original submission. We will add the appropriate citation in the revised version.

A8.Regarding the visualization results: Please refer to A3.

[1].ReLU Fields: The Little Non-linearity That Could. [2].Towards viewpoint-invariant visual recognition via adversarial training.

We greatly appreciate the reviewers’ thoughtful feedback and insightful suggestions. We have carefully reviewed all the comments and provide our detailed responses below.

A1.Regarding experiments on ModelNet40 datasets: Thanks for your question! Please refer to A3 to Reviewer eS8T.

A2.Regarding visualization results: Thanks for your question! Since VQSA and DCP are finetuning-based methods, we provided visualization results of URIE, Unadv and our method. Please refer to A3 to Reviewer eS8T for results on ModelNet40 and A2 to Reviewer mDnf for real scene.

A3.Regarding ablation study: Thanks for your question! In fact, our method does not involve many detachable modules that allow for extensive ablation studies. Therefore, in the ablation study section of the paper, we primarily focused on analyzing the impact of voxel grid resolution and the boundary of texture perturbation on the final results. To further illustrate the superiority of voxel grids as a 3D representation in our framework, we additionally compared the performance of robust textures when using NeRF's MLP as the 3D representation. In this experiment, we selected one object from each category for evaluation. The results on Resnet18 is shown below:

A4.Regarding reference: Thank you for your question! However, we would like to clarify that our work is not closely related to transfer learning or style transfer. The reason why the robust textures can be transferred across different classifiers is similar to the phenomenon observed in adversarial examples: different classifiers tend to learn feature spaces with certain similarities. Additionally, image restoration works have a different focus compared to ours. Traditional image restoration aims to recover as much visual detail of the original image as possible, whereas our objective is to enhance the performance of downstreaming models when encountering corrupted images. Image restoration is merely one possible approach that can be incorporated into our framework, but it is not the primary focus of our work.

A5.Regarding selection of proxy model: Thanks for your question! As we explained in section 3.5, employing a proxy model with relatively weaker generalization ability can lead to better transferability to other model architectures. To further clarify this design choice, we consider our task as the reverse of an adversarial attack task. In prior work on adversarial attacks, it has been observed that when the proxy model itself has stronger generalization capability, the generated adversarial examples tend to exhibit higher transferability. Intuitively, this can be likened to a teacher-student scenario: if a question can challenge a very capable student, it is more likely to challenge a less capable one as well. In contrast, our objective is to enable the classifier to correctly recognize the input, which can be viewed as the opposite of adversarial attacks. In this analogy, it is akin to designing questions that students can answer correctly. If a question can be easily answered by a less capable student, it is naturally more likely to be correctly answered by a more capable one as well.

A6.Regarding universal robust textures: Thanks for your question! We proposed universal robust textures because synthesizing robust textures for each individual instance is an inefficient process. However, since the main focus of this paper is on object-specific robust texture synthesis, we only conducted preliminary experiments to validate the feasibility of universal robust texture. We believe that the performance of universal robust textures can be further improved in the following ways: 1. A larger, category-organized multi-view 3D reconstruction dataset. In our current IM3D dataset, each category contains only 10 different objects, and we used only 8 of them to train the robust textures. This clearly limits their generalization capability. 2. More efficient 3D representation. In our current setting, we adopt voxel grids as the 3D representation. However, training universal robust textures on a larger set of objects inevitably requires increasing the capacity of voxel grids, which would significantly raise the GPU cost. NeRF-based representation may be better for scaling up category-level robust texture training.

A7.Regarding end-to-end training: Thanks for your question! Our proposed data-centric approach for enhancing classifier performance is based on the assumption that, in industrial scenarios, deployed classifiers cannot be easily modified or retrained. But we think it's possible to jointly optimize both the textures and the classifier to get better performance.