Extremely Simple Multimodal Outlier Synthesis for Out-of-Distribution Detection and Segmentation

Thanks for your insightful reviews and great support of our paper! We provide the responses to your questions as follows:

Q1: What if there are more modalities? How are the features going to be swapped?

A1: Thanks for your suggestion! To further demonstrate the generality of our approach, we conducted an additional experiment on a three-modality setting involving video, optical flow, and audio for OOD detection on the EPIC-Kitchens dataset. We extend Feature Mixing to this setting by randomly selecting a subset of N feature dimensions from each modality and swapping them cyclically: video->audio, audio->optical flow, and optical flow->video, thus creating outlier features for all three modalities. As shown in the table below, Feature Mixing consistently outperforms NP-Mix across all metrics in this three-modality setting (evaluated using the Energy score). These results further support the robustness and versatility of our method.

Importantly, FM is inherently modality-agnostic: it operates in the feature space without relying on assumptions about the input structure, enabling straightforward application to diverse and heterogeneous modality combinations. We further extend FM to the single-modality setting by randomly selecting N dimensions from the first and second halves of the feature embedding and swapping them within the same modality. This enables outlier synthesis without relying on a second modality. We evaluate this unimodal variant on OOD detection using HMDB51 (video modality). As shown below, our unimodal FM achieves strong performance, outperforming the best NP-Mix baseline, which validates the applicability of FM beyond multimodal settings.

Q2: Clarification of the "two key properties" and their benefit to OOD detection

A2: Thank you for the thoughtful suggestion. Theoretical results in Theorems 1 and 2 establish two key properties of Feature Mixing (FM): (1) the synthesized outliers have low likelihood under the ID distribution, and (2) their deviation from the ID distribution is bounded. These properties are important for OOD detection and segmentation, as they ensure that FM produces challenging but plausible outliers that populate the low-density regions near the boundary of the ID distribution. This supports effective decision boundary regularization and reduces model overconfidence on unseen data.

As shown in Figure 4, the FM-generated outliers are well-separated from the ID features in the embedding space. This separation helps the model to better distinguish between in-distribution and out-of-distribution regions. These visualizations provide intuitive, empirical evidence that our method generates meaningful and bounded outliers, as predicted by our theoretical analysis.

In response to your suggestion, we have additionally visualized real outliers alongside FM-synthesized outliers, and we observe a significant degree of overlap between them. This provides empirical evidence that the synthesized outliers are representative of real OOD samples, which supports their effectiveness in regularizing the model. Interestingly, FM-synthesized outliers also extend into other low-likelihood regions not covered by the real OOD samples, further increasing the diversity and coverage of the synthesized data.

Q3: the mask is not consistent

A3: Thank you for pointing this out, and we apologize for the confusion. In Appendix, we used the same mask notation for both modalities in Equations (1) and (2) for notational simplicity in the theoretical analysis. However, as correctly illustrated in Figure 3 and Algorithm 1, we do use two distinct masks—one for each modality—in the actual implementation.

This simplification in notation does not affect the validity of the theoretical results. For example, the expectations computed in Equations (3) and (4) in appendix remain the same regardless of whether the two modalities share the same mask or not, since the proofs rely on the marginal distributions rather than the specific mask values.

We will correct the notation in the final version of the paper to accurately reflect the use of distinct masks and to ensure consistency throughout the manuscript.

Q4: discussion on limitations

A4: Sorry for the confusion. We discuss the limitations and future work here and will include them in the final version of the paper.

Feature Mixing uses a random selection of feature dimensions to swap between modalities. While this is highly efficient and theoretically grounded, it may not always target the most informative feature regions for outlier synthesis. Future work could explore adaptive or learnable selection mechanisms that dynamically identify features that maximize OOD separability.

We sincerely thank the reviewer again for the constructive feedback. We will incorporate all clarifications and updates from this rebuttal into the final version to make the paper clearer. If there are any additional questions or concerns, we would be more than happy to further address them.

Thanks for your insightful reviews and great support of our paper! We provide the responses to your questions as follows:

Q1: whether the model learns to implicitly detect these classes due to this decision

A1: Thank you for your insightful comment. We follow the standard protocol in prior OOD segmentation works (e.g., [1]), where certain classes are assigned the void label and are excluded from the loss computation during training. This setup is commonly used due to the absence of publicly available 3D OOD segmentation datasets.

However, as you correctly pointed out, this approach introduces the risk that the model may implicitly learn to detect these void-labeled classes, since their input features are still processed by the network. To address this limitation and eliminate the possibility of such implicit supervision, we introduce the CARLA-OOD dataset, a new benchmark for multimodal OOD segmentation. In CARLA-OOD, OOD classes are strictly held out from the training data and only appear during testing, ensuring a clean evaluation protocol without any data leakage.

We believe this new dataset provides a more rigorous and realistic testbed for multimodal OOD segmentation in autonomous driving.

[1] Cen et al., Open-world Semantic Segmentation for LIDAR Point Clouds, ECCV 2022

Q2: How the method works for segmentation? the ablation study of the parameter N for segmentation?

A2: Thank you for the helpful questions, and we apologize for the lack of clarity. For segmentation, we first project the LiDAR point cloud onto the image plane, following the approach in [1], to obtain spatially aligned pairs of LiDAR points and image pixels. Feature mixing is then performed at each valid pixel location—i.e., only where a corresponding LiDAR point exists—by mixing the aligned image and point features. Pixels without projected points are excluded from the mixing.

We have now also included the ablation study of the parameter N for segmentation on SemanticKITTI, as shown in the table below. We use N=10 by default, and the results show that the performance is robust across different values of N.

[1] Zhuang et al., Perception-aware Multi-sensor Fusion for 3D LiDAR Semantic Segmentation, ICCV 2021

Q3: Feature Mixing on a single modality

A3: Thank you for the insightful suggestion. We explored extending Feature Mixing to the single-modality setting by randomly selecting N dimensions from the first and second halves of the feature embedding and swapping them within the same modality. This approach enables outlier synthesis without relying on a second modality. We first evaluate this unimodal variant on the OOD detection task using the HMDB51 dataset (video modality). As shown in the table below, our unimodal FM achieves strong performance, outperforming the best NP-Mix baseline, which validates the applicability of our method beyond multimodal settings.

We then apply the same unimodal FM strategy to the OOD segmentation task on SemanticKITTI using only LiDAR data. The results, also provided in the table below, demonstrate that our method scales effectively to segmentation and maintains competitive performance in the single-modality case. These results suggest that FM is flexible and generalizable, and can be extended to unimodal settings with minimal modifications.

To further demonstrate the generality of our approach, we conducted an additional experiment on a three-modality setting involving video, optical flow, and audio for OOD detection on the EPIC-Kitchens dataset. We extend Feature Mixing to this setting by randomly selecting a subset of N feature dimensions from each modality and swapping them cyclically: video->audio, audio->optical flow, and optical flow->video, thus creating outlier features for all three modalities. As shown in the table below, Feature Mixing consistently outperforms NP-Mix across all metrics in this three-modality setting (evaluated using the Energy score). These results further support the robustness and versatility of our method.

Q4: standard deviations

A4: Thank you for pointing this out, and we apologize for the confusion. To maintain clarity and readability in the main paper, especially in longer tables such as Table 1 and Table 2, we chose to report only the mean values, as including standard deviations would have made the tables overly cluttered. That said, we agree that reporting variance is important for assessing the reliability of the results. We confirm that each experiment was run with three different random seeds, and we will include the standard deviations in the appendix in the final version of the paper for completeness.

Q5: Societal impact

A5: Thank you for your comment. We agree that OOD detection for self-driving cars has important societal implications. In our work, Feature Mixing advances multimodal OOD detection and segmentation by enabling more effective and efficient outlier exposure during training. This contributes directly to the reliability and safety of autonomous systems, including self-driving cars, by helping models handle unfamiliar or unexpected scenarios in open-world environments.

Beyond autonomous driving, the method also has broader societal impact in other safety-critical domains such as healthcare and security, where robustness to unknown or anomalous data is crucial. By improving model reliability in the presence of distribution shifts, our approach supports the deployment of AI systems in dynamic, high-stakes environments where failure could have significant consequences.

We will include a discussion of these points in the appendix as suggested.

We sincerely thank the reviewer again for the constructive feedback. We will incorporate all clarifications and updates from this rebuttal into the final version to make the paper clearer. If there are any additional questions or concerns, we would be more than happy to further address them.

Thanks for your insightful reviews and valuable comments on our paper! We provide the responses to your questions as follows:

Q1: how the method behaves with varying modality combinations

A1: Thank you for raising this important point. We agree that analyzing the method's behavior across diverse modality combinations is crucial for assessing its generalizability. In our current experiments, we already evaluated Feature Mixing (FM) on substantially different modality pairs, such as point cloud + image for multimodal OOD segmentation (Table 1), as well as video + optical flow for multimodal OOD detection (Table 2). To further demonstrate the generality, we conducted an additional experiment on a three-modality setting involving video, optical flow, and audio for OOD detection on EPIC-Kitchens dataset. We extend FM to this setting by randomly selecting a subset of N feature dimensions from each modality and swapping them cyclically: video->audio, audio->optical flow, and optical flow->video, thus creating outlier features for all three modalities. As shown below, FM consistently outperforms NP-Mix across all metrics in this three-modality setting. These results further support the robustness and versatility of our method.

Importantly, FM is inherently modality-agnostic: it operates in the feature space without relying on assumptions about the input structure, enabling straightforward application to diverse and heterogeneous modality combinations. We further extend FM to the single-modality setting by randomly selecting N dimensions from the first and second halves of the feature embedding and swapping them within the same modality. This enables outlier synthesis without relying on a second modality. We evaluate this unimodal variant on OOD detection using HMDB51 (with video modality). As shown below, our unimodal FM achieves strong performance, outperforming the best NP-Mix baseline, which validates the applicability of FM beyond multimodal settings.

Q2: positioning relative to prior work on entropy maximization

A2: Thank you for the helpful comment. We would like to clarify that entropy maximization is not the main contribution of our work—it serves as a lightweight auxiliary component for outlier training. The core novelty of our approach lies in multimodal outlier synthesis via Feature Mixing, which addresses key limitations of prior methods, particularly in terms of scalability and computational efficiency in multimodal settings. While entropy-based techniques (e.g., ICCV 2021 work) have been explored in the unimodal segmentation context, they typically rely on external OOD datasets for training and do not generalize well to multimodal scenarios. In contrast, our method performs online outlier synthesis using only in-distribution data, eliminating the need for external data, and integrates entropy regularization as a self-contained, plug-in loss. Additionally, our method is significantly more efficient—achieving up to 370× speed-up over prior approaches. We will revise the paper to clarify this positioning and more clearly distinguish our contributions from prior work.

Q3: challenges of extending existing solutions to multimodal setting

A3: We agree that clarifying the challenges of extending unimodal methods to the multimodal setting is important for highlighting our motivation. While methods such as Mixup and VOS have shown success in unimodal tasks, directly applying them to multimodal data is non-trivial due to the following key challenges:

(1) Modality heterogeneity: Multimodal data often involves inputs with fundamentally different structures and distributions—e.g., dense 2D images, sparse 3D point clouds, temporal audio signals, etc. Pixel-level interpolation techniques like Mixup are inherently incompatible across heterogeneous modalities, making it difficult to define meaningful cross-modal augmentations. Moreover, Mixup's random linear blending can produce ambiguous or noisy samples that may lie near or within the in-distribution (ID) manifold, reducing its effectiveness for OOD detection.

(2) Computational inefficiency: Methods like VOS and NP-Mix rely on distribution estimation, sampling, or searching in high-dimensional feature spaces, which becomes computationally expensive when extended to multimodal settings—especially for dense prediction tasks such as segmentation, where per-pixel or per-point processing is required.

Our Feature Mixing addresses these challenges by (i) operating entirely in the feature space, where modality-specific structures have already been abstracted, (ii) introducing a lightweight, swap-based perturbation strategy to synthesize outliers efficiently, avoiding costly sampling or searching, (iii) being theoretically grounded (Theorems 1 and 2) and empirically validated to maintain ID-OOD separation while improving efficiency and scalability.

We will revise the paper to include this discussion and better highlight the motivation and advantages of our approach.

Q4: intuitive or empirical analysis on Feature Mixing

A4: Thank you for the helpful comment. We provide both theoretical and empirical evidence to support the claim that our proposed method generates true outliers without injecting noise.

Theoretically, Theorems 1 and 2 in the paper prove that samples generated via Feature Mixing have a low likelihood of residing within the in-distribution (ID) manifold, satisfying the criteria for true outliers. In contrast to interpolation-based methods like Mixup, our approach perturbs features in a structured manner, which helps avoid overlapping with the ID region.

Empirically, we visualize the synthesized outliers using t-SNE on HMDB51 dataset in Figure 4. As shown, Mixup tends to generate noisy samples that remain close to or within the ID feature cluster. In contrast, Feature Mixing produces outliers that are more clearly separated from the ID distribution, effectively expanding the boundary of the embedding space.

Q5: comparison with mix-and-match

A5: While our Feature Mixing and the Mix-and-Match strategy both involve perturbations in feature space, they differ fundamentally in objective, mechanism, and application domain:

1. Different Objectives: The primary goal of mix-and-match is to encourage diversity within the data distribution for stochastic human motion prediction. In contrast, our Feature Mixing is designed to synthesize OOD samples for the explicit purpose of improving OOD detection and segmentation. We aim to create features that lie in low-likelihood regions of the in-distribution data, rather than generating plausible variations of it.

2. Different Mechanisms: Mix-and-match perturbs a unimodal hidden state with random noise. Our Feature Mixing, however, operates in a multimodal setting by swapping feature dimensions between two different modalities (e.g., images and point clouds). This cross-modal exchange is a core element of our approach and is fundamentally different from adding noise to a single modality's representation.

3. Different Applications: Mix-and-match is applied to generative modeling for sequence prediction. Our method is designed for discriminative models in the context of OOD detection.

In summary, while both methods involve a form of feature perturbation, our Feature Mixing is a novel approach for multimodal OOD synthesis, distinct from the unimodal, diversity-focused mix-and-match perturbation.

Q6: include a decoder or reconstruction pipeline

A6: Thank you for the thoughtful suggestion. We agree that incorporating a decoder or reconstruction pipeline from feature space back to image space could enhance interpretability by providing visual intuition about the synthesized outliers. In this work, our primary focus is on efficient multimodal outlier generation in the feature space, which avoids the complexity and overhead of generative modeling. As shown in Figure 4, we already provide t-SNE visualizations to empirically demonstrate that the outliers generated by Feature Mixing are well-separated from the in-distribution data, supporting their OOD nature.

We explored the feasibility of feature-to-image inversion in a controlled setting using a text-to-image diffusion model (Stable Diffusion 3 medium). Specifically, we used the text prompt "a photo of a dog" and extracted embeddings for the text. We then applied Feature Mixing by randomly selecting N dimensions from the first and second halves of the text feature embeddings and swapping them. Finally, we used the resulting text embedding as input to Stable Diffusion 3 to generate an image. We found that the generated images shifted away from the dog concept, indicating that the perturbed embedding moved outside the in-distribution manifold. This serves as initial evidence that Feature Mixing does in fact produce semantically meaningful outliers in embedding space. For multimodal scenarios, however, such as video or 3D point clouds, constructing reliable decoders is substantially more difficult due to data scarcity and the complexity of generative modeling across heterogeneous modalities. We therefore consider feature-to-modality inversion for complex modalities a valuable and promising direction for future work.

We sincerely thank the reviewer again for the constructive feedback. We will incorporate all clarifications and updates from this rebuttal into the final version to make the paper clearer. If there are any additional questions or concerns, we would be more than happy to further address them.

Dear Reviewer an3N,

We’ve carefully addressed all your insightful comments. Your feedback has been invaluable in improving the quality of our work, and we greatly appreciate your thoughtful review.

As the discussion period is nearing its end, we wanted to ensure you’ve had the opportunity to review our responses. Please let us know if there are any remaining questions or concerns we can address further.

Thank you once again for your time and consideration.

Thanks for your insightful reviews and great support of our paper! We provide the responses to your questions as follows:

Q1: how the mixed branch is optimized? need some reorganization to help readers better comprehend

A1: Thank you for the insightful comment. We agree that a clearer explanation of how the entropy maximization loss optimizes the mixed branch is important. The core idea is that the mixed branch generates pseudo-OOD features through feature mixing. We apply an entropy maximization loss (Eq. 5) on these features to explicitly encourage the model to produce uncertain (i.e., high-entropy) predictions for them. This discourages the model from making overconfident predictions on ambiguous or unfamiliar inputs—behavior that is characteristic of OOD samples.

In contrast, the classification loss encourages low-entropy (i.e., high-confidence) predictions on in-distribution (ID) samples. Together, these complementary objectives train the model to develop a confidence-based decision boundary: ID samples are associated with confident predictions, while pseudo-OOD samples—introduced through feature mixing—are explicitly pushed toward uncertain predictions.

As shown in Figure 2, this training objective increases the separation in predictive confidence between ID and OOD inputs, which is crucial for effective OOD detection. We appreciate your feedback and will revise the narrative between lines 208 and 213 to better highlight this motivation and improve the clarity around the optimization of the mixed branch.

Q2: support around Figure 4 in paper and Figure 1 in appendix

A2: Thank you for the thoughtful question. We apologize for the confusion. The visualizations in Figure 4 (main paper) and Figure 1 (appendix) are based on real-world data from the HMDB51 dataset—not on toy or synthetic examples. Specifically, we apply our feature mixing strategy to generate pseudo-OOD features, and then use t-SNE to project the high-dimensional features into 2D for visualization.

Regarding the ID features (purple dots): they are indeed identical across all methods. The apparent differences in their positions are due to the stochastic nature of t-SNE, which may project the same high-dimensional points differently depending on the surrounding context (i.e., the presence of different OOD samples). Since t-SNE emphasizes preserving local neighborhood structures rather than global geometry, the introduction of different OOD features can cause slight shifts in the 2D projections of the ID features as well.

To further clarify how these visualizations support our theoretical claims: Theorems 1 and 2 establish that feature mixing generates samples that lie outside the ID manifold (i.e., are valid outliers), yet remain bounded in their deviation. This is empirically reflected in Figure 4(d): the pseudo-OOD features are clearly separated from the ID cluster (supporting Theorem 1), while remaining relatively close rather than scattered far away (supporting Theorem 2). For Figure 1 (appendix), we prove in Eq. 6 (appendix) that the shifts of outliers from the ID distribution are proportional to N. Therefore, a large N promotes outliers to lie in low-likelihood regions of the ID feature distribution.

Together, these visualizations provide intuitive, empirical evidence that our method generates meaningful and bounded outliers, as predicted by our theoretical analysis. We will revise the text to clarify how the figures are generated and better connect the visualizations to Theorems 1 and 2.

Q3: Consistent bold text in tables

A3: Thank you for pointing this out. We acknowledge the inconsistency in highlighting the best-performing entries in Tables 1–4. In the final version, we will ensure that all tables consistently use bold text to indicate the best results in each comparison. This will significantly improve readability and make performance differences easier to interpret.

Q4: discussion on limitations

A4: Sorry for the confusion. We discuss the limitations and future work here and will include them in the final version of the paper.

Feature Mixing uses a random selection of feature dimensions to swap between modalities. While this is highly efficient and theoretically grounded, it may not always target the most informative feature regions for outlier synthesis. Future work could explore adaptive or learnable selection mechanisms that dynamically identify features that maximize OOD separability.

We sincerely thank the reviewer again for the constructive feedback. We will incorporate all clarifications and updates from this rebuttal into the final version to make the paper clearer. If there are any additional questions or concerns, we would be more than happy to further address them.