OpenPatch: a 3D patchwork for Out-Of-Distribution detection

We thank the reviewer for the positive feedback.

W1: Most of the OOD detection literature covered by our sec 2 is tailored to 2D data and considers methods that require a training or at least a finetuning phase on the support set that contains samples of known classes. Our approach is instead learning-free. OpenPatch exploits a pre-trained architecture to extract data representations and perform sample-to-class comparisons to detect novelty in 3D data. Despite the different scenarios, the right part of Fig. 2 shows that OpenPatch is competitive with the state-of-the-art in 3D OOD detection methods (according to [Alliegro et al. (2022)]) that train or finetune on the support set.

W2: Unfortunately the reviewer did not provide pointers to specific advanced backbones. We used both well-known architectures on Objaverse and cutting-edge transformer-based ones on OpenShape (models released with the publication [Liu et al. (2023)]).

W3: In Figure 4, we present a depiction of patch assignments using a backbone based on Farthest Point Sampling (FPS). For this instance, we extracted the coordinates of the patch features and subsequently assigned each point in the original point cloud to its nearest patch feature. In the initial column, we color-coded based on the class assignment of the underlying patch embedding, while in the second column, each point was colored using heatmap colors corresponding to the relative embedding distance from the memory bank of the underlying patch embedding. We recognize that discerning the different patches could be difficult due to the color similarity, to counteract this, we added an additional figure in the supplementary material where we color each patch a different color and compare the patch dimension when changing the layer of extraction.

We thank the reviewer for the feedback. The questions highlight a potential gap in familiarity with the existing literature, which we are happy to address through our responses.

W1 + Q1: Out of Distribution Detection aims at labeling a new sample as belonging or not to the training distribution. This is a binary problem. Open Set Recognition refers instead to a class set, so its goal is identifying if a new sample belongs to one of the known classes or to an unknown one. This is a multiclass problem and more precisely, it combines the multiclass recognition of the known classes with the binary task of separating known from unknown data. Open World builds over Open Set, so that once a new class is identified it is incrementally added to the original model. Our focus is on Out of Distribution Detection since we are only interested in the binary task of separating known from unknown data. As in our specific case the unknown data is novel because it belongs to a new semantic class, we also used the expression ‘semantic novelty detection’. We are aware that the names might be confusing at first sight, but they have been explained in the first paragraph of our sec. 2. Having clarified the frameworks it also becomes clear that the papers cited by the reviewers do not match our setting.

W2: We use memory banks but we did not claim novelty on it. Our technical contribution is in the 3D-tailored patch extraction and OOD scoring function. We are the first to propose a metric for 3D OOD detection that combines semantic consistency and patch anomalousness. Moreover, our approach does not need training on the support dataset of known classes, while most of the OOD detection literature relies on this stage. This makes a direct comparison unfair. The approaches that we considered as baseline are the only methods that afford OOD detection without learning/finetuning on the support set. See also the answer to W1 from the reviewer 3nE.

W3 + Q2: The work [Alliegro et al. (2022)] extensively discussed the different challenges of 3D OOD detection and Open Set Learning with respect to those of the corresponding 2D domain.

W4: The component analysis presented in section 4.3 explains in detail the role of each component of our approach. We would be happy to provide further details if the reviewer could specify the meaning of ‘proper ablation’ for our model.

W5: Out of Distribution Detection is a standard name to indicate the task of predicting whether a test sample belongs or not to the nominal data distribution. Unfortunately there is a clear misunderstanding, we are not performing neither object detection nor semantic segmentation.

We thank the reviewer for the comments. Unfortunately, a few observations might be based on misunderstandings. We offer some clarification in the following.

W1: Regarding the “no prior work” claim, we kindly refer the reviewer to [Alliegro et al (2022)] in our references. This work introduced the task of 3D OOD detection and ran an extensive benchmark of methods originally designed for 2D data, showing that their performance does not necessarily transfer to 3D. Regarding the “no possible application” claim, we highlight the essential need for methods able to manage novelty in real-world applications involving 3D information. We refer the reviewer to the following works that appear among our references [Masuda et al. (2021); Bhardwaj et al. (2021); Cen et al. (2021); Wong et al. (2019); Riz et al. (2023)]. Besides autonomous driving where unexpected objects might create critical safety issues, other relevant scenarios are related to industrial robotics applications where agents are supplied with only a few clean 3D object templates to use as reference and they should be able to mitigate potential operational hazards due to novel objects. Both prior works and applications are mentioned in the paper's introduction and related work section.

W2: Regarding “failed to mention how are those patches obtained” we point the reviewer to sec 3 (in particular 3.1) of our submission. There, we explained that representations extracted from internal layers of networks that elaborate on point clouds describe different sized 3D shape portions that we indicate as patches. There is no manual segmentation involved here, the patches are a byproduct of the chosen architecture and the only choice to operate is deciding from which layer to extract them: deeper layers provide larger patches.

W3 + Q: Both BOW and VLAD define global representations from local descriptors to compare samples among each other. Specifically, NetVLAD is an architecture designed and trained (with specific loss) to learn such a representation. The goal is then to use it to search for the most similar samples of a certain query instance in a retrieval scenario. Differently, our work proposes a strategy to extract from large-scale pre-trained models (regardless of the specific backbone and loss) a representation that can be efficiently used to evaluate whether a sample belongs or not to a reference class via a newly introduced scoring function and no further training. Incidentally, collecting features from different layers of a point-cloud network means collecting local representations, thus we designed the metric to evaluate the assignment to known or unknown class to exploit this logic.
We remark that we care about sample-to-class distance rather than sample-to-sample distance. It is well known that BOW-related strategies are suited for the latter but not for the former scenario [*]

[*] “In Defense of Nearest-Neighbor Based Image Classification”, CVPR 2008

We thank the reviewer for the detailed feedback.

W1: Regarding the similarities with Patchcore, we inherit its memory bank creation and downsampling mechanism. We did not claim novelty on either of them. Our technical contribution is in the 3D-tailored patch extraction and OOD scoring function. We are the first to propose a metric for 3D OOD detection that combines semantic consistency and patch anomalousness.

Q1 A: Concerning the mentioned references we can state that: [1] leverages ad hoc pre-training and a very specific architecture designed to be used for semantic novelty detection on 2D data. Instead, our work targets efficiency in design and learning effort: OpenPatch builds on large-scale 3D models trained with standard classification or contrastive objectives and well-known backbones. We remark that, despite 3D transformer models exist, adapting the relational module and the learning pipeline of [1] to the 3D setting is not straightforward. The method proposed in [2] was shown in [1] to be worse than the Mahalanobis baseline that we adopted. Therefore, although adding it to our analysis might improve completeness, our conclusions would remain unchanged. The approach in [3] requires the availability of a textual description for the known data object category. This cannot be given for granted in industrial settings where in-distribution objects are custom CAD models created by users and are possibly identified by numerical codes without any semantic meaning.

Q1 B: [4] and [5] propose scaled-up architectures and improved multi-modality training over OpenShape but they maintain the contrastive loss paradigm. In other words, they learn better 3D representation from which our OpenPatch would benefit: we expect linear improvements over the reported results without any significant change in the general trend. Additionally, [4] is a concurrent ICLR submission we were not aware of while working on OpenPatch, while [5] is an arXiv paper.

Q2: The main bottleneck of OpenPath lies in the nearest neighbor search which depends on the number of patches stored in the memory banks. This total count of patches is determined by the number of samples in the support set and the layer used for feature extraction in the pretrained model encoder. In our specific experiments, we observed negligible performance gains from implementing a more aggressive coreset reduction. We attribute this to the low number of samples in the support set, as defined by the 3DOS benchmark, and our decision to use larger patches rather than numerous smaller local patches. Our main concern for implementing the coreset selection is mainly the limited memory of devices in industrial scenarios.