MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities

Thanks for your insightful reviews, and we appreciate your valuable suggestions! We address your concerns and questions as follows:

Q1: Discuss with other multi-modality works in related works.

A1: Thanks for your insightful suggestion! As shown in Table 10 in our main paper, we indeed compared A2D and NP-Mix with other multimodal self-supervised training tasks, including Contrastive Loss, Relative Norm Alignment, Cross-modal Distillation, and Cross-modal Translation. A2D and NP-Mix show substantial superiority over other multimodal tasks. We will add a subsection on Multimodal Learning in related work to discuss these works.

---

Q2: Evaluate methods on different ID/OOD category splits.

A2: Thanks for your valuable comment! As shown in Figure 13 in our main paper, we indeed evaluated our method on five different ID/OOD category splits. Training with our A2D and NP-Mix is statistically stable and surpasses the baselines significantly under different dataset splits. We also evaluated our method on three different random seeds, as shown in Figure 12 in our main paper. Our method significantly surpasses the baselines under different random seeds.

---

Q3: W.r.t. A2D, the orders of non-ground-truth classes matter. what if the discrepancy is maximized in an order-preserved manner.

A3: Thanks for your interesting idea! There are two small issues with the idea. One issue is the conflict object with A2D training. A2D aims to enlarge the prediction distance between different modalities for all non-ground-truth classes. If we want to preserve the order of predictions for different modalities, for example, we make the most similar non-ground-truth class have the second-largest prediction for both video and optical flow, the prediction distance between them will decrease as a result. The second issue is on the implementation. For each class, we need to manually select its most similar classes to define the order, which is complicated for datasets with a large number of classes.

We implement this idea on EPIC-Kitchens 4/4 using video and optical flow and use Energy as OOD score. EPIC-Kitchens 4/4 has four ID classes: ‘put’, ‘take’, ‘open’, ‘close’. We define 'put' and 'take' as paired classes (most similar) and ‘open’ and ‘close’ as another paired classes. Given an input sample, we preserve its prediction order for both modalities by making the paired classes have the second-largest prediction. As shown below, A2D with order-preserved prediction is better than w/o A2D, but with a large gap compared with the original A2D. Besides, because of the conflict object with A2D, the prediction discrepancy $l_{OOD}-l_{ID}$ for order-preserved A2D is significantly reduced compared with the original A2D.

---

Q4: Detailed comparison between the proposed method and the method in [16]. it seems to also worth comparing with Learning Placeholders for Open-Set Recognition CVPR 2021.

A4: Thanks for your useful suggestion! The major difference between [16] and our NP-Mix is the availability of real outliers during training. In [16], few labeled outliers are available and they generate more synthesized outliers based on the labeled outliers and unlabeled data. However, in our case, we assume only ID data is available during training and we want to generate synthesized outliers using only ID data. Therefore, [16] can't be used in our case directly. We make a small modification to it and implement it as a baseline. Instead of randomly selecting one sample from real outliers and another sample from its nearest neighbor in unlabeled data for Mixup, we randomly select a sample from one ID class and another sample from its nearest neighbor in other ID classes for Mixup. We also include PROSER [a] as a baseline based on your suggestion. PROSER [a] conducts manifold mixup with the pairs from different classes without considering neighborhood information and will inject manifold intrusion problem, as shown in Figure 2 in the attached PDF in global response. Instead, our NP-Mix explores broader feature spaces by leveraging the information from nearest neighbor classes without noisy synthesized outliers injection and achieves the best performances.

[a] Zhou, et al. Learning placeholders for open-set recognition. In: CVPR, 2021

---

Q5: create baselines that combine uni-modal OOD methods and existing typical multi-modality methods outside the OOD area.

A5: Thanks for your suggestions! We added evaluations on HMDB51 25/26 for the ensemble of multiple unimodal OOD methods for each modality to demonstrate the importance of studying MultiOOD. Due to space limits, we put the detailed results in the global response at the top of this page. The ensemble of multiple unimodal OOD methods always brings performance improvements, but still has a large gap compared with our multimodal solution (A2D+NP-Mix), further demonstrating the importance of studying MultiOOD.

For baselines on typical multimodal methods outside the OOD area, we already compared A2D and NP-Mix with four multimodal self-supervised training tasks in Table 10 in our main paper. Here, we further include two multimodal baselines Gradient Blending [b] and SimMMDG [c]. A2D and NP-Mix show substantial superiority over other multimodal methods.

[b] Wang, et al. What makes training multi-modal classification networks hard? In: CVPR, 2020

[c] Dong, et al. SimMMDG: A simple and effective framework for multi-modal domain generalization. In: NeurIPS, 2023

Thanks for your insightful reviews, and we appreciate your valuable suggestions! Please find the responses to your questions below:

Q1: Lacks important baselines (e.g. ensemble of multiple singleOOD methods for each modality). This also relates to the question of "why we should study MultiOOD?"

A1: Thanks for your suggestions! We added evaluations on HMDB51 25/26 for the ensemble of multiple unimodal OOD methods for each modality to demonstrate the importance of studying MultiOOD. Due to space limits, we put the detailed results in the global response at the top of this page. The ensemble of multiple unimodal OOD methods always brings performance improvement, but still has a large gap compared with our multimodal solution (A2D+NP-Mix), further demonstrating the importance of studying MultiOOD.

---

Q2: Limited theoretical analysis of proposed methods. A2D and NP-Mix need more theoretical analysis.

A2: Thanks for bringing up this point. Our A2D is based on the empirical observation of Modality Prediction Discrepancy on our MultiOOD benchmark. This discrepancy can be attributed to the unavailability of semantic information on OOD data during model training, stimulating each modality to generate conjectures based on its unique characteristics upon encountering OOD data during testing. We demonstrate that such discrepancy is highly correlated to the ultimate OOD performance. We also show through extensive experiments that A2D can amplify such discrepancy and enhance the efficacy of OOD detection. For NP-Mix, we give an additional analysis on decision boundaries, as shown in Figure 2 in attached PDF in global response. Vanilla Mixup causes manifold intrusion due to the injection of noisy synthesized outliers. Instead, our NP-Mix only generates synthesized outliers in the intermediary space between two classes, thus avoiding the manifold intrusion and helping the network learn better decision boundaries.

In summary, Multimodal OOD Detection is in its very early era. Our MultiOOD benchmark aims to fill this gap and our proposed A2D+NP-Mix solution is mostly based on empirical observation from extensive experiments. More interesting solutions and theoretical analyses are expected in future work.

---

Q3: The experiments are only conducted on the task of action recognition. the title "MultiOOD" seems over-claiming. In addition, it only focuses on specific modalities (video, optical flow, audio). This makes the scope of the paper somewhat limited.

A3: Thanks for your insightful comments! To further increase the scope of our paper and demonstrate the versatility of the proposed A2D training, we add another task of 3D Semantic Segmentation using LiDAR point cloud and RGB images. We evaluate on SemanticKITTI [a] dataset and set all vehicle classes as OOD classes. During training, we set the labels of OOD classes to void and ignore them. During inference, we aim to segment the known classes with high Intersection over Union (IoU) score, and detect OOD classes as unknown. We adopt three metrics for evaluation, including FPR95, AUROC, and $mIOU_c$ (mean Intersection over Union for known classes). We use ResNet-34 and SalsaNext [b] as the backbones of the camera stream and LiDAR stream. We compare our A2D with basic LiDAR-only and Late Fusion, as well as two multimodal 3D semantic segmentation baselines PMF [c] and XMUDA [d]. Our A2D also shows strong performance under this new task (3D Semantic Segmentation) with different combinations of modalities (LiDAR point cloud and RGB images). We will integrate these new benchmark results in our paper to further increase its scope.

[a] Behley, et al. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In ICCV, 2021

[b] Cortinhal, et al. Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. In ISVC, 2020

[c] Zhuang, et al. Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In ICCV, 2021

[d] Jaritz, et al. xmuda: Cross-modal unsupervised domain adaptation for 3d semantic segmentation. In CVPR, 2020

---

Q4: Some figures (e.g. Fig. 4) could be improved for clarity, with consistent color-coding and same x-axis scales.

A4: Thanks for your suggestion! We have improved the figures and will update them in the final version of the paper.

---

Q5: Why "$\alpha$ > 1" can ensure the synthesized outliers reside in the intermediary space between two prototypes, rather than near the ID data?

A5: Given a $\lambda$ sampled from distribution Beta($\alpha$, $\alpha$), as shown in Figure 1 in the attached PDF in global response, when $\alpha$ < 1, $\lambda$ has a very high probability of being close to 0 or 1. As a result, the synthesized data Z = $\lambda$ * Z1 + ($1-\lambda$) * Z2 will be close to Z1 or Z2. Instead, when $\alpha$ > 1, $\lambda$ has the highest probability near 0.5. As a result, the synthesized data Z = $\lambda$ * Z1 + ($1-\lambda$) * Z2 will be in the intermediary space between Z1 and Z2.

---

Q6: How sensitive are the A2D and NP-Mix methods to hyperparameter choices?

A6: For A2D, we analyzed the choices of different distance functions, as shown in Table 5 in our main paper. A2D training exhibits robustness across various distance functions. Regardless of the specific distance metric employed, substantial improvements are consistently observed compared to the baseline approach without A2D training.

We investigated the parameter sensitivity of NP-Mix on Nearest Neighbor parameter N and Mixup parameter $\alpha$, as shown in Figure 10 and Figure 11 in our main paper. NP-Mix demonstrates robustness across different parameter settings and yields substantial enhancements in OOD performance for all cases.

Thanks for your insightful reviews and great support of our paper! We provide the responses to your questions as follows:

Q1: In the video modality, what features do you extract to characterize the stream? Do you extract per-frame features, or per-video features.

A1: Thanks for your valuable question! We use the SlowFast [a] network as the backbone to extract per-video features. The SlowFast model involves a Slow pathway operating at a low frame rate to capture spatial semantics and a Fast pathway operating at a high frame rate to capture motion at fine temporal resolution.

[a] Feichtenhofer, et al. Slowfast networks for video recognition. In: ICCV, 2019

---

Q2: Same question for the audio modality

A2: We also extract per-video features for the audio modality. Each 10-second audio waveform is converted into one spectrogram and then inputted to the ResNet-18 audio encoder. Audios that are less than 10 seconds are padded to 10 seconds.

---

Q3: You use 5 datasets, all of them having the video and optical flow modalities, but audio modality is missing from 2 of them. In this case, what protocol do you adopt? Do you ignore them and perform the audio analysis only on the remaining three?

A3: All datasets have video and optical flow modalities. Therefore, we create four Multimodal Near-OOD benchmarks and two Multimodal Far-OOD benchmarks using video and optical flow, as shown in Figure 2 in the paper. Only EPIC-Kitchens, HAC, and Kinetics have audio modality. We create two challenging Multimodal Near-OOD benchmarks using EPIC-Kitchens and Kinetics with different combination of modalities (video-audio, flow-audio, video-flow-audio), as shown in Table 9 in the paper.

---

Q4: Is your approach robust when one modality is missing?

A4: Thanks for your interesting comment! In our framework, we also train a classifier for each modality to get predictions from each modality separately. By default, we use the predictions obtained from the combined embeddings of all modalities to calculate the OOD score. However, when one modality is missing, we can use the predictions from the remaining modality to calculate the OOD score. We added the evaluations on HMDB51 25/26 under this challenging condition as below. We use Energy as the OOD score.

When one modality is missing, the performance drops a little, especially when the video is missing (A2D+NP-Mix (Flow)). However, compared with training on one modality alone (Video-only and Flow-only), training with A2D and NP-Mix can also bring significant improvements for each modality when another modality is missing. For example, A2D+NP-Mix (Video), the case when optical flow is missing, yields a 16.56% relative improvement on FPR95 compared with Video-only. This underscores the importance of multimodal training for Multimodal OOD Detection.