Improving Robustness of 3D Point Cloud Recognition from a Fourier Perspective

Thank you for appreciating our new contributions as well as providing the valuable feedback. Below we address the detailed comments, and hope that you can find our response satisfactory.

Question 1: There is confusion in the caption of Figure 3, which suggests the Jacobian matrix for an input point cloud. According to the context, it should be the Jacobian matrix of the model regarding the input point cloud.

Thanks for pointing out this issue. We will clarify this in the revision.

Question 2: As FAT is based on adversarial training, it is expected to have comparisons with other adversarial training techniques rather than only mixing- and deformation-based approaches.

We have already compared our proposed FAT with adversarial training technique on the ModelNet-C test set in Table 1 in Sec. 4.2. Our FAT outperforms all other methods in terms of mCE (mean corruption error).

Question 3: Minors: The figures are hard to read as the font size is too small. Missing space before the fourth sentence in the caption of Figure 3.

Thanks for pointing out these issues. We will correct them in the revision.

Question 4: How much computational resources does FAT require regarding computational time, memory usage, etc., and how does it compare with other augmentation approaches.

The computational cost of our FAT implementation primarily involves generating high-frequency and low-frequency adversarial examples. Compared to standard adversarial training and other data augmentations, FAT incurs approximately $1.7 \sim 3.2$ times higher computational costs and about 3 times more memory usage. Despite this limitation, such an increase in computational overhead is deemed acceptable for offline training scenarios. Moreover, our approach would not affect the efficiency of model inference, ensuring unhindered deployment of well-trained models in practical applications. We will add the discussion in the revision.

Thank you for acknowledging the novelty of our paper as well as providing the valuable feedback. Below we address the detailed comments, and hope that you can find our response satisfactory.

Question 1: Complexity of Implementation might hinder the adoption of the method in practical scenarios.

The computational cost of our FAT implementation primarily involves generating high-frequency and low-frequency adversarial examples. Compared to standard adversarial training and other data augmentations, FAT incurs approximately $1.7 \sim 3.2$ times higher computational costs. Despite this limitation, such an increase in computational overhead is deemed acceptable for offline training scenarios. Moreover, our approach would not affect the efficiency of model inference, ensuring unhindered deployment of well-trained models in practical applications. We will add the discussion in the revision.

Question 2: The paper mentions a slight reduction in clean accuracy when using FAT, which might be a concern for applications.

In our limitations section, we have already noted that while FAT slightly reduces clean accuracy, the magnitude of this reduction is minimal and deemed acceptable. This could be attributed to the inherent trade-off between accuracy and robustness [1]. Moreover, as shown in Table 2, existing data augmentation methods decrease clean accuracy by an average of 1.03%, whereas our FAT exhibits a smaller average reduction of 0.55% in clean accuracy. Additionally, Table 2 demonstrates that incorporating FAT results in an average improvement of 0.33% in clean accuracy, indicating that FAT's impact on clean accuracy is not universally negative.

Question 3: The paper would be more convincing by additional experiments on more diverse datasets and real-world scenarios.

Thanks for the valuable suggestion. We further conduct experiments on the KITTI and ScanObjectNN datasets, both collected by LiDAR sensors. Due to space constraints, detailed experimental results and analysis are presented in the Global Rebuttal. (See more results on ScanObjectNN-C in response to Question 5). Experiments on these two real-world datasets [2][3] further validate the superiority and practicality of our FAT.

Question 4: The paper lacks experiments comparing some updated point cloud baselines, which could provide a more comprehensive evaluation of the proposed method.

Thanks for the valuable suggestion. We further conduct experiments for PointNeXt [4] on ModelNet-C and ScanObjectNN-C below. (See more results on PointNeXt in response to Question 5.)

The results demonstrate that FAT achieves consistent performance on the advanced network architecture PointNeXt, similar to observations on PointNet and more. Our FAT outperforms all other methods in terms of mCE. This indicates that FAT's performance is largely independent of the underlying model architecture, making it applicable to both traditional and modern networks. We will incorporate these additional experiments into the revision.

Question 5: The paper lacks comparison with updated augmentation methods such as AdaptPoint, which could highlight the relative performance of FAT against newer augmentation techniques.

Thanks for the valuable suggestion. We further conduct experiments for comparison with AdaptPoint [2] on ScanObjectNN-C. AdaptPoint follows the official experimental settings [2]. The results are shown below.

It is evident that incorporating FAT achieves a lower mCE, indicating its superiority. We will integrate these results into Table 2 and continue to include more comprehensive experiments as per your recommendations in the revised version.

[1] Theoretically Principled Trade-off between Robustness and Accuracy, ICML 2019

[2] Sample-adaptive Augmentation for Point Cloud Recognition Against Real-world Corruptions, ICCV 2023

[3] Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving, CVPR 2023

[4] PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies, NeurIPS 2022

Thank you for appreciating our new contributions as well as providing the valuable feedback. Below we address the detailed comments, and hope that you can find our response satisfactory.

Question 1: The paper would be more convincing by using real-world data from LiDAR sensors, such as KITTI dataset.

Thanks for the valuable suggestion. We further conduct experiments on the KITTI and ScanObjectNN datasets, both collected by LiDAR sensors. Due to space constraints, detailed experimental results and analysis are presented in the Global Rebuttal. Experiments on these two real-world datasets [1][2] further validate the superiority and practicality of our FAT.

Question 2: Why does the proposed method appear to be significantly more effective when combined with other data augmentation techniques?

FAT operates in the frequency domain and considers the underlying structure of compactly represented 3D point clouds, while traditional data augmentation methods focus on spatial transformations within the 3D data itself. Therefore, attributed to the complementary and compatible information from both the spatial and frequency domains, mixing methods result in much better defense.

Question 3: The paper does not analyze the relationship between the frequency and spatial domains in the context of the 3D point cloud.

Thanks for the suggestion. Similar to 2D images, the rough shape in point clouds’ spatial domain is represented by transformed low-frequency components while the fine details of objects are encoded in transformed high-frequency components. Furthermore, the frequency characteristics of point clouds represent higher level and global information than point-to-point relations in spatial domain. That is, the frequency representation encodes more abstract and essential contexts for recognizing the point cloud. This implies that in the frequency domain, point clouds are compactly represented, facilitating a better understanding of low-level distortions that are free of high-level semantics. We will incorporate this analysis in the revision.

Question 4: The paper only studies the defense effects against general 3D data corruptions without considering specific attack methods.

Thanks for the suggestion. Actually, FAT can enhance the models’ robustness against adversarial attacks to some extent. The enhanced PointNet model on ModelNet40 achieves the adversarial accuracy of 31.2% under PGD-20 at $\epsilon$ = 0.05, compared to 0% for the standard trained PointNet. The enhanced PointPillars model on KITTI achieves a defense accuracy of 65.5% with IoU greater than 0.7 against Tu's attack method [3], and 56.8% against Zhu's attack method [4], compared to 42.5% and 26.0% for the standard trained PointPillars model. We will include more comprehensive experimental results and discussions on defending against adversarial attacks in the revision, including references and analyses of specific attack methods [3][4][5][6].

Question 5: Modifications in the frequency domain can sometimes result in physically unrealizable transformations in the spatial domain. Will this limit the application of your work in real-world scenarios?

No, it does not limit applicability. As a training method, FAT only needs to consider transformations in the digital world, which is always feasible regardless of the scenario, unlike attack methods that must consider physical implementation.

Question 6: Can the authors provide more details on the reasons for using GFT rather than DCT or other frequency transformation methods?

Images are typically transformed in the frequency domain with the 2D discrete Fourier transform (DFT) or discrete cosine transform (DCT). Different from images supported on regular grids, although 3D point clouds are highly structured, they reside on irregular domains without an ordering of points, hindering the deployment of traditional Fourier transforms. Specifically, an image represented as $\mathbb{R}^{n \times n}$ is regularly sampled on a grid, ensuring that pixels $(i, j)$ and $(i+1, j)$ are adjacent. In contrast, a point cloud represented as $\mathbb{R}^{n \times 3}$ lacks such ordering of points, where the Euclidean distance between the $i$-th and $(i+1)$-th points can be substantial. Traditional Fourier transforms cannot be directly applied to point clouds due to their unordered nature and loss of relative positional information. However, graphs provide a natural and accurate representation of irregular point clouds. Each point in a point cloud is treated as a vertex, connected to its $K$ nearest neighbors, with each point's coordinates serving as graph signals. Once a graph is constructed to represent the point cloud, the graph Fourier transform (GFT) can compactly transform it into the frequency domain by leveraging the edges of the graph to encode relative positional information.

Question 7: What would be the outcome if medium frequency bands were modified?

Thanks for the suggestion. We further conduct more ablation studies among FAT, as well as FAT variants: FAT with only medium-frequency modified, FAT with only low-frequency modified, and FAT with only high-frequency modified. Due to space constraints, detailed experimental results and analysis are presented in the subsequent comments. We will incorporate these additional experiments into the revision.

Thank you for appreciating our new contributions as well as providing the valuable feedback. Below we address the detailed comments, and hope that you can find our response satisfactory.

Question 1: The paper would be more convincing by using real-world data from LiDAR sensors, such as KITTI dataset.

Thanks for the valuable suggestion. We further conduct experiments on the KITTI and ScanObjectNN datasets, both collected by LiDAR sensors. Experiments on these two real-world datasets further validate the superiority and practicality of our FAT. The experimental settings and evaluation metrics on ScanObjectNN-C [1] are consistent with those on ModelNet-C. The KITTI-C dataset [2] includes four major types of corruptions: Weather-level (e.g., Strong Sunlight), Sensor-level (e.g., Density Decrease), Motion-level (e.g., Moving Object), and Object-level (e.g., Local Gaussian Noise). Following [2], we use $AP_{cor}$ (corruption average precision) at moderate difficulty as the evaluation metric on KITTI-C, where higher values indicate better performance. We employ the representative 3D object detection model PointPillars. The results are shown below.

It can be seen that our FAT generally leads to lower mCE (mean corruption error) on ScanObjectNN-C and higher $AP_{cor}$ (average precision) on KITTI-C. The experimental results on both KITTI and ScanObjectNN datasets further validate the generalizability and applicability of our FAT under real-world conditions. We will add the results in the revision.

Question 2: The paper does not discuss the complexity and efficiency of FAT.

Thanks for the suggestion. The complexity of FAT implementation primarily involves generating high-frequency and low-frequency adversarial examples. Compared to standard adversarial training and other data augmentations, FAT incurs approximately $1.7 \sim 3.2$ times higher computational costs. Despite this limitation, such an increase in computational overhead is deemed acceptable for offline training scenarios. Moreover, our approach would not affect the efficiency of model inference, ensuring unhindered deployment of well-trained models in practical applications. We will add the discussion in the revision.

[1] Sample-adaptive Augmentation for Point Cloud Recognition Against Real-world Corruptions, ICCV 2023

[2] Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving, CVPR 2023