Out-of-Distribution Detection using Synthetic Data Generation

3. I'm not sure if it is a fair comparison if you compare the synthetic OOD setup with other baselines which are not trained on any OOD data. I think the result in Table 5 is a more realistic one where you compare the synthetic data with the real one. This tells you how good the synthetic OOD data is. I see the point of the OOD scarcity issue if you consider everything as training data, then it would be hard to find real-world OOD data for training the detector. But it would be nice to have some more analysis about the quality of the synthetic as the result in Table 5 is contrary to the discussion in line 259-266, stating that synthetic OOD data is nearly as effective as real OOD data.

Thank you for your thoughtful comment. Please note that the baselines (MSP, Energy, ReAct, DICE) we used are standard baselines in the OOD detection literature, both in the text and image domains. Methods that incorporate external OOD data, such as Du et al. (ICLR, 2024), Katz-Samuels et al. (ICML, 2022), and Hendrycks et al. (ICLR, 2019), also compare against these baselines. Additionally, the setup in Table 1 (now Table 3 in the revised version) follows the standard approach used in these works, but we go a step further by evaluating our method on cross-generalization performance in Table 5 (now Table 6 in the revised version).

Note, our method and the baselines have access to the same real InD data, thus making it a fair comparison. In practice, when OOD robustness is lacking, collecting appropriate real data can be time-consuming and resource-intensive. As Yang et al. (2024) highlight, "approaches impose a strong assumption on the availability of OOD training data, which can be infeasible in practice." Nonetheless, we still consider an ideal/oracle baseline (trained directly on real OOD data). In contrast, our synthetic data approach offers an immediate, practical solution that avoids this assumption.

Regarding the quality of synthetic OOD data, we believe that our results in Table 1 (now Table 3 in the revised version), particularly the perfect zero FPR95 on far-OOD tasks and the FPR95 closest to the ideal model on near-OOD tasks, demonstrate the high effectiveness of our approach. Moreover, the results in Table 5 (now Table 6 in the revised version), which present our method's performance in cross-generalization experiments—a less commonly explored setting in the literature—further reinforce the robustness of the synthetic data.

That said, we do acknowledge the performance gap in the CC/BT-MBPP pair in Table 5 (now Table 6 in the revised version). As noted in the paper, improving this performance is part of our future work. We believe that enhancing prompt diversity and creativity will be key to addressing this gap and further improving synthetic data quality in such cases.

In summary, while we recognize the challenges with synthetic data in certain scenarios, the overall results indicate that the synthetic OOD data used in our study is both effective and of high quality.

References:

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. ICLR 2019.

Julian Katz-Samuels, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. ICML 2022.

Xuefeng Du, Zhen Fang, Ilias Diakonikolas, and Yixuan Li. How does unlabeled data provably help out-of-distribution detection? ICLR 2024.

Yang, Jingkang, et al. "Generalized out-of-distribution detection: A survey." International Journal of Computer Vision (2024): 1-28.

Thank you for reviewing our paper. We have dedicated considerable time and effort to thoroughly address your concerns. While our responses may be detailed, we have made every effort to be concise while ensuring clarity and comprehensiveness. We look forward to further discussion.

1. The overall writing and the formatting are poor. The arrangement of the figures all around and the main text always refers to figures in the appendix, which should be avoided. The author first showed the results and comparisons to baselines in section 3 without describing the method and setup first.

Thank you for your feedback. We appreciate your observations and haved addressed them in the revised draft. Regarding the placement of figures to the appendix, we acknowledge that it may seem disruptive. However, due to space constraints, we placed some supplementary details in the appendix to keep the main text focused on the core content. Regarding Section 3, presenting the results on selective classification before describing the method was a deliberate design choice to motivate the study. By showing the importance of synthetic data for selective classification, our aim was to provide context for why our approach is a promising direction for OOD detectoin in the forthcoming sections. However, as per your suggestion, we have now updated the revised draft by explaining our method prior to the selective classification experiemtns.

2. The results in Table 5 show that using real OOD data outperforms the synthetic OOD data in most of the cases. This doesn't support the claim of the author about the quality of their synthetic data, which was discussed in section 4.

Thank you for your comment. We would like to clarify that in Table 5 (Table 6 in the updated draft), real OOD data outperforms synthetic OOD data in only 6 out of 24 cases (3 metrics * 8 experiments = 24 cases), not across the board. It's important to note that the model trained on the CC/BT-GSM8K pair shows exceptional generalization to the CC/BT-MBPP test pair, achieving strong results for both InD datasets for all three metrics. This, along with the positive results presented in Table 1 (now Table 3 in the revised version), supports the quality of the synthetic data used. Furthermore, while a model trained on the CC/BT-MBPP pair does not perform as well on the CC/BT-GSM8K test pair in terms of FPR95, the InD accuracy is on par or even better. We explicitly mention in the paper that addressing this gap in FPR95 performance is part of our future work as we believe that improving prompt diversity will be key to closing this gap and enhancing the performance of synthetic data in such cases.

Moreover, per your feedback, we have revised our statement about the quality of the synthetic data, originally discussed in Section 4 (now Section 3 in the revised draft). We clarify that our synthetic data is comparable to real OOD data and may offer greater diversity, sometimes leading to better generalization than real data.

Thank you for reviewing our paper. We have dedicated considerable time and effort to thoroughly address your concerns. While our responses may be detailed, we have made every effort to be concise while ensuring clarity and comprehensiveness. We look forward to further discussion.

1. even those termed “near-OOD,” seem simplistic and easy to solve. For instance, distinguishing between tasks like CC and GSM8k does not appear to be particularly challenging. It is not clear what is the real-world relevance or difficulty of these OOD tasks. It would be very surprising that no baseline method perform well on these tasks. Why the tasks presented in Table 1 are not solved by simple heuristics (different tasks clearly ask different questions, it should be fairly easy to recognise them)?

It seems like you're referring to 'far-OOD' tasks, not 'near-OOD', as the example you provided—distinguishing between tasks like CC and GSM8k—fits the far-OOD category. The notion of far- and near-OOD is not new and has been prevalent in several previous works including Liu et al. (2023); Yang et al. (2022); Winkens et al. (2020). For example, Liu et al. (2023) consider InD as a Sentiment Analysis task and Far-OOD as a Question Classification task.

Far-OOD detection is crucial, especially in real-world applications like systems that need to detect and handle tasks such as math or coding problems differently. For instance, when a system encounters math or code problems, it should avoid applying certain types of processing, such as a harmful content aligner (e.g. another LLM), which might be useful for general text but would be unnecessary (and costly) for math or code tasks.

Regarding your concern about baseline methods (e.g. MSP, ReAct Energy, DICE), while it may seem surprising that baseline methods perform poorly on far-OOD tasks like CC versus GSM8k, this is actually expected. It's important to note that these techniques, originally developed for image tasks, are widely used as a standard in the text domain. However, these methods often struggle when applied to text, due to the inherent challenges of language data, such as greater variability in input forms, semantics, and structure. Several previous studies have demonstrated that these baselines tend to yield very high False Positive Rates (FPR95) when applied to text datasets. For instance, when tested on the SST-2-IMDB as an InD-OOD pair, these methods produced FPR95 scores of 77.7, 79.1, 79.6, and even 100% on MSP, ReAct Energy, and DICE (as shown in Table 8 of Baran et al.'s 2023 ACL paper; note that on most InD-OOD pairs FPR95 is above 50). In contrast, our method yields surprisingly low FPR95 (e.g. a perfect zero on far-ood tasks and an FPR95 closest to the ideal model on near-OOD tasks, see Table 1 (now Table 3in the revised version) in our paper).

Reference: Mateusz Baran, Joanna Baran, Mateusz Wójcik, Maciej Zieba, and Adam Gonczarek. Classical out-of-distribution detection methods benchmark in text classification tasks. ACL 2023.

2. There seems to be an unusual amount of repetitions of numbers, which seems to point to mistakes in reporting. Why are there some many repeated numbers in Table 1?

There are no mistakes in the results reported in Table 1 (now Table 3 in the revised version). We believe the repetition you're noticing refers to the InD accuracy, which is the same for the baseline methods—MSP, Energy, DICE, and ReAct. This is expected because all these methods use the same underlying model for prediction and only differ in how they perform OOD detection. We included a detailed explanation of how these baselines work in Appendix A, so it's possible that the reviewer inadvertently overlooked this part.

Moreover, we also include the code for reproducing our experiments, including the implementation of these baselines. Therefore, the results in Table 1 (now Table 3 in the revised version) can be easily verified by running the provided code.

3a. Do we need Synthetic OOD Data: Given the definition of OOD used here as “task vs. task” detection, it’s unclear why synthetic data generation is necessary. Instead of synthetic data, one could simply use any existing task/dataset as OOD data for this problem.

We thank the reviewer for the insightful comment. Indeed, it is possible to use existing datasets as OOD data, and this approach has been explored in several previous works, including Hendrycks et al. (NeurIPS 2018, ICLR 2019), Zhang et al. (WACV, 2023), and more recently by Du et al. (ICLR 2024) and Katz-Samuels et al. (ICML 2022). For example, Du et al. (ICLR, 2024), Katz-Samuels et al. (ICML 2022), and Hendrycks et al. (ICLR 2019) all use existing data to improve OOD detection. However, their approach relies on the assumption that such external data is both sufficiently available and representative of real-world OOD scenarios. In practice, real-world OOD inputs are highly diverse and unpredictable, making it difficult to curate datasets that capture all potential distribution shifts; as Yang et al. (2024) highlight, "approaches impose a strong assumption on the availability of OOD training data, which can be infeasible in practice," practical constraints have led to a shift in recent research toward settings where real OOD data is either unavailable or significantly limited.

In contrast, synthetic OOD data generation allows us to create more controlled and flexible test conditions. By creating diverse synthetic data (see Figure 4 - now Figure 3 in the revised draft) that simulates various distribution shifts, we can train a more robust OOD detector, capable of approaching ideal performance (See Table 1 (now Table 3 in the revised version)).

Moreover, to clarify, the notion of "task vs. task" OOD detection is not new in the literature, with several prior works like Du et al. (ICLR 2024), Katz-Samuels et al. (ICML 2022), and Hendrycks et al. (ICLR 2019) all addressing this approach to OOD detection.

3b. This should actually be a baseline to test.

Thank you for the suggestion. In fact, we have already included a more competitive baseline in our experiments, which we refer to as "Original (Ideal)". This baseline assumes the availability of the original OOD data for training the detector, without the need for synthetic data. It serves as a competitive benchmark, as it is directly trained on the original OOD data, rather than relying on outlier data filtered from a pool of OOD+InD data that wrongly identifies some InD samples as OOD as done by Du et al. (ICLR 2024), Katz-Samuels et al. (ICML 2022). As shown in Table 1 (now Table 3 in the revised version), our model performs comparably to this "Original (Ideal)" baseline, matching a perfect zero FPR95 on far-OOD data, and being closest to it on near-OOD data.

4. Generally, I feel that it would be more interesting to focus on within-task OOD detections.

Thank you for your suggestion. The results in our selective classification experiments (section Section 4 of the revised draft) already address within-task OOD detection and show substantially improved performance in this setup.

5. Clarity of the Presentation: confusion about the different types of contribution between modifications to the OOD training pipeline (generating synthetic OOD data) and modifications to the OOD detector itself (adding a third class instead of using a binary setup).

Thank you for your feedback. We'll make sure to clarify this in the final version. Specifically, we will include results using existing methods with our synthetic OOD data to provide a clearer comparison. Additionally, we want to emphasize that the 3-way design is simply a matter of convenience and not a design choice for the success of our method.

References:

Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. NeurIPS, 2018.

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. ICLR 2019.

Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments. WACV 2023.

Julian Katz-Samuels, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. ICML 2022.

Xuefeng Du, Zhen Fang, Ilias Diakonikolas, and Yixuan Li. How does unlabeled data provably help out-of-distribution detection? ICLR 2024.

Yang, Jingkang, et al. "Generalized out-of-distribution detection: A survey." International Journal of Computer Vision (2024): 1-28.

We were able to conduct new experiments to clarify that the three-way design is primarily a matter of convenience rather than a critical design choice for the success of our approach. We conducted experiments on several InD-OOD pairs, including CC-GSM8k, CC-SST-2, and CC-ToxiGen, where we trained a binary model alongside our three-class model, ensuring both models were trained on an equal number of samples for consistency.

The results in the Tables below indicate that the two models perform comparably across all key metrics, suggesting that the primary performance improvement stems from our synthetic data generation pipeline rather than the choice of a three-way model design. Lastly, note that the InD accuracy is the same as other baselines since we use the same classifier and only differ in how OOD is detected, wherein the baselines use the scoring method detailed in Appendix A while our method uses synthetic data.

Table for GSM8K:

Table for SST-2:

Table for ToxiGen:

2. In prompting the model to generate synthetic data, did you use a fixed prompt template? Have you tried different prompt templates? Is it necessary to test the robustness with different prompts? What kind of decoding strategy did you use? What kind of decoding strategy did you use?

In our preliminary experiments, we tried several prompt templates for the data synthesis process. We aimed to refine the prompts to achieve higher quality and more diverse results by manually inspecting a few generated samples. Examples of generated synthetic data and the final prompts are detailed in Tables 7-20.

We used a top-k decoding strategy for generating outputs. At each step, the model considers only the top-k most probable tokens from the probability distribution predicted by the model, limiting the candidate set to the most relevant tokens.

Thank you for reviewing our paper. We have dedicated considerable time and effort to thoroughly address your concerns. While our responses may be detailed, we have made every effort to be concise while ensuring clarity and comprehensiveness. We look forward to further discussion.

1. The authors seemed to use different models, e.g., Llama3 70b for generating synthetic data, Llama2 13b for fine-tuning on the datasets tested, and Starling 7B for the RLHF model. Why did you use different settings rather than being consistent?

The Llama3 70B variant was used for data generation because larger models tend to produce better, more coherent generations. However, we also conducted additional experiments using the Llama-3 8B-instruct to generate the synthetic data and evaluated its performance on several InD-OOD pairs, including CC-GSM8k, CC-SST-2, and CC-ToxiGen. The results for these pairs are shown in the tables below, with new results highlighted in bold:

Table for GSM8K:

Table for SST-2:

Table for ToxiGen:

As seen in the tables above, even a smaller model like Llama-3 8B-instruct is able to generate data capable of achieving perfect zero FPR95 on the far-OOD CC-GSM8k InD-OOD pair. Furthermore, on near-OOD datasets, its performance is second only to the Ideal baseline, showing that smaller models can still generate high-quality synthetic data for OOD detection tasks. We have added these results in Table 3 of the updated paper along with explanations in Section 5.3. We plan to continue evaluating the remaining InD-OOD pairs and will update our results in the final version of the paper.

For the detector models, we chose smaller 7B and 13B Llama variants because detector systems are meant to be simpler and computationally efficient. Their primary function is to filter user inputs detected as OOD and avoid predictions on them. Using larger models would complicate the system unnecessarily and increase computational costs.

For the RLHF experiment, we used Starling-RM-7B-alpha because, unlike general Llama models, it is a pre-trained reward model specifically designed for the RLHF pipeline. It is optimized to assign scores to model outputs, reducing the need for continuous human labeling. Moreover, we chose Starling-RM-7B-alpha in particular because, like many reward models, it excels in certain areas, such as achieving a 98.0% win rate in the Chat category on the RewardBench Leaderboard, but its performance drops to just 58.0% in the Reasoning category. Our goal is to redesign the reward model so that it serves two purposes: not only will it evaluate LLM responses with a score, but it will also classify those responses as either high-performing (InD) or low-performing (OOD), based on their win rate. The model will output two things: 1) a score, and 2) a classification label (InD or OOD). This dual-purpose approach enhances the RLHF pipeline by enabling practitioners to filter out responses where this reward model underperforms, ultimately helping to train a stronger and more reliable LLM.

3. I really don't like the way your wrote your related work section (first two paragraphs)

Thank you for your feedback. We have thoroughly revised the related work section to address your concerns and improved its clarity and strcuture in the revised version of the paper (changes highlighted in red).

Thank you for reviewing our paper. We have dedicated considerable time and effort to thoroughly address your concerns. While our responses may be detailed, we have made every effort to be concise while ensuring clarity and comprehensiveness. We look forward to further discussion.

1. Are all the baseline methods (MSP, Energy, DICE) trained with the original real data? And are the synthetic data the same size as the original OOD data?

The MSP, Energy, and DICE baselines are trained only on the in-distribution (InD) data and do not incorporate any out-of-distribution (OOD) data, neither original nor synthetic, during training. These methods are well-established in the OOD detection literature and follow a standard, widely accepted approach. As detailed in Appendix A, these baselines utilize a $K$-class model trained solely on InD data to produce binary (i.e. OOD vs. InD) predictions using a scoring function and threshold. Due to space limitations, we provided a detailed explanation in Appendix A, and it is possible the reviewer may have inadvertently overlooked these details.

For the methods that use OOD data (i.e., Original (Ideal) and Synthetic (Ours)), the size of the synthetic and original data is kept similar in our experiments. However, it's important to note that synthetic data can be generated in large amounts, and our approach isn't limited by the volume of data. This flexibility allows us to improve the model’s performance with more data, if needed. We chose to keep the sizes of the synthetic and original data similar for consistency. Moreover, note that using the real OOD data is an idealized baseline, which isn’t commonly used in OOD research. Real-world OOD data can vary widely, and we often don’t have enough of it. Still, we think it's valuable to compare against this ideal baseline, as our results are closest to ideal, demonstrating the effectiveness of our approach. We apologize for not mentioning this earlier and have included it in the revised version of the paper.

2. You are using a 70B model to generate the synthetic data but using 13B or 7B data for the OOD detection task. In a way this is distillation? Have you analyzed the impact of the size of the synthetic data generation model? Would a 7B data be able to generate high-quality OOD data?

We conducted additional experiments to address your question regarding the use of smaller models for generating synthetic data. Specifically, we used Llama-3 8B-instruct to generate the data and evaluated its performance on several InD-OOD pairs, including CC-GSM8k, CC-SST-2, and CC-ToxiGen. The results are shown in the tables below, with new results highlighted in bold:

Table for GSM8K:

Table for SST-2:

Table for ToxiGen:

As seen in the tables above, even a smaller 8B model is able to generate data capable of achieving perfect zero FPR95 on the far-OOD CC-GSM8k InD-OOD pair. Furthermore, on near-OOD datasets, its performance is second only to the Ideal baseline, showing that smaller models can still generate high-quality synthetic data for OOD detection tasks.

We added these results in Table 3 of the updated paper along with explanations in Section 5.3. We plan to continue evaluating the remaining InD-OOD pairs and will update our results in the final version of the paper.