Breaking the Detection-Generalization Paradox on Out-Of-Distribution Data

Q4: About the access of semantic-shifted data.

Similar to other training-required OOD detection methods, the proposed method also needs to access semantic-shifted data during training. This limitation widely exists in many previous OOD detection methods, but still worth noting here.

Reply: Thanks for the feedback. Follwoing this suggestion, we have add this to the discussion of the direction of the training-based OOD-D in Appendix F. However, we should note that this submission mainly focus on handling the detection-generalization paradox instead of the requirement for semantic-shifted data access. Accessing semantic-shifted data is commonly adopted in OE-related studies, including [1,2,3,4,5,6]. In future work, we would try to involve data synthetic methods in our framework to alleviate the dependence on auxiliary outliers.

[1] Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection. ICML'23

[2] Unified out-of-distribution detection: A model-specific perspective. CVPR'23

[3] The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection. NeurIPS'24

[4] Training ood detectors in their natural habitats. ICML'22

[5] Poem: Out-of-distribution detection with posterior sampling. ICML'22

[6] Atom: Robustifying out-of-distribution detection using outlier mining. ECML PKDD'21

Q5: About the illustration in Fig.3(d).

In Figure 3(d), why there is no blue area (ID data) in the figure?

Reply: Thanks for this question. Please note that the model fine-tuned with the OOD-G method would reduce the gap between $D_{\text{ID}}$ and $D_{\text{CS}}$ on the representation space, as we discovered in Sec.3.1. As a result, the learned representation are almost overlapped with each other in Fig.3(d).

Q6: About the formulation of OOD-G in Eq.3.

I am unsure whether the formulation of OOD generalization in Eq. 3 is correct. $D_{CS}$ is mentioned by the text above Eq.3. However, there is no such notation in the following equation. The formulation should be revised carefully and a proper citation should be provided here.

Reply: Thanks for this valuable feedback. Following this suggestion, we follow the representative work IRM[1] and DomainBed[2] to refine the formulation. Specifically, we consider dataset $D_{e}=${$(x_i^e, y_i^e)$}, where $i =${$1... n_e$} collected under multiple environment $e \in \mathcal{E}$. $D_{CS}$ = {$D_{e}$:$e \in \mathcal{E}$} denotes the collection of datasets from multiple environments with potential covariant shifts. We have added the above content in our updated submission in lines 146 to 148.

[1] Invariant Risk Minimization. In arXiv, 2020.

[2] In Search of Lost Domain Generalization. In ICLR, 2021.

Q7: About the visualization.

The authors post a visualization of the loss landscape in Figure 10. However, such a visualization contains limited information. The reviewer suggests comparing with SAM, OE, and the original ERM.

Reply: Thanks for this constructive feedback. We have updated the Fig.10 and added a detailed discussion on the new visualization.

In Fig.10, we demonstrate the loss of landscapes for different methods on $D_{\text{CS}}^{\text{test}}$ of CIFAR-10. OE leads the model to minima with high sharpness, making it sensitive to distribution shift, thus enhancing the model's detection ability while sacrificing the generalization performance. ERM achieves higher sharpness than SAM and DR-SAM, but it does not explicitly optimize the loss's sharpness. DR-SAM, on the other hand, achieves lower sharpness within the specific radii but high at the outer region. This allows the model to be robust to the covariate-shifted samples while sensitive to the semantic-shifted ones.

# ref: https://github.com/hendrycks/robustness/blob/master/ImageNet-C/create_c/make_cifar_c.py
from skimage.filters import gaussian
import numpy as np 

def gaussian_noise(x, severity=1):
    c = [0.04, 0.06, .08, .09, .10][severity - 1]

    x = np.array(x) / 255.
    return np.clip(x + np.random.normal(size=x.shape, scale=c), 0, 1) * 255

def gaussian_blur(x, severity=1):
    c = [.4, .6, 0.7, .8, 1][severity - 1]

    x = gaussian(np.array(x) / 255., sigma=c, channel_axis=-1)
    return np.clip(x, 0, 1) * 255

# a test demo for the functions
dummy_x = np.random.rand(1, 3, 224, 224) * 255 # Batch, Channels, Height, Width
dummy_x_gaussian_noise = gaussian_noise(dummy_x, severity=1)
dummy_x_gaussian_blur = gaussian_blur(dummy_x, severity=2)

• An ideal model should have low sharpness on both in-distribution and covariate-shift data. We observes that OOD-G fine-tuned model have higher sharpness in $D_{\text{ID}}^{\text{train}}$ and $D_{\text{ID}}^{\text{test}}$ but lower in $D_{\text{CS}}^{\text{test}}$ compare to pre-trained model. Subsequently, the OOD-G fine-tuned model generalize well in $D_{\text{CS}}^{\text{test}}$ but fail to achieve competitive result in $D_{\text{ID}}^{\text{train}}$ and $D_{\text{ID}}^{\text{test}}$, as shown in Tab. 1 and 2. Since a lower sharpness is desired for the model to generalize well on the specific distribution (lines 256-257 in the original submission), such an observation leads to the claim that: "an ideal model should have low sharpness on both in-distribution and covariate-shift data."

In general, training with OOD-G can not ensure low sharpness for $D_{\text{ID}}$ and $D_{\text{test}}$ based on our observation and discussion in [1].

[1] Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts. In ICLR, 2024.

Q3: Comparing additional methods.

Lock of essential comparison. Although the experiments in Section 5 encompass a few representative works in OOD detection and generalization, some of the most related works are missed. Several recent methods also jointly consider OOD detection and generalization [1] [3] [4]. Thus, the comparison in this current version is biased and incomplete. Besides, I also feel that some SOTA OOD detection methods are also missed. For example, POEM [5], NPOS[6], and NTOM[7]. As far as I can tell, POEM substantially surpasses OE, MCD, and MixOE in terms of OOD detection on CIFAR benchmarks. SCONE, WOODS, and DUL which jointly pursue OOD detection and generalization can achieve much better overall performance compared to the baselines in Table 1 2 and 3. The reviewer suggests comparing the proposed method with these methods.

Reply: Thanks for this feedback. Following this suggestion, we provide experiments on the mentioned baselines. DR-SAM achieves leading performance in OOD-G and OOD-D performance.

We provide a comparison of additional baselines, including POEM, NOPS, and NTOM. We additionally conduct experiments with SCONE and WOODS. We conduct experiments with settings aligned with the original submission. As NPOS trains the CNN backbone without the final linear classifier [1], we can only provide its OOD-D performance and leave the accuracy as N/A. In addition, DUL has not yet released its source code; we will leave the comparison with it for future work.

We notice that POEM would degenerate the model's classification ability in both in-distribution and covariate-shifted samples, which leads to sub-optimal performance compared to the OE. POEM cannot exceed the OE when using the TIN-597 as auxiliary outliers when training on the CIFAR benchmark, as DUL also has the same report. The WOODS and SCONE, on the other hand, cannot perform well in CIFAR-100 under the traditional OOD-D setting, and also scraify the OOD-G performance.

Overall, DR-SAM achieves leading performance in OOD-G and OOD-D performance on the CIFAR benchmark under the traditional OOD-D setting. We have updated the results in Appendix B.3.

[1] OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. In arXiv, 2023.

We thank the reviewer 5Bu7 for the valuable feedback. We addressed all the comments. Please kindly find the detailed responses below. Any further comments and discussions are welcomed!

Q1: About the contribution.

Vague contributions. The authors claim that the main contribution of this paper is to identify the detection-generalization paradox. However, as far as I could tell, the trade-off/conflict relationship has been pointed out by several previous works [1] [2] [3]. Thus such a claim should be either toned down or a clear explanation about the difference from previous work should be provided.

Reply: Thanks for this valuable feedback. Please note that we identify the paradox with $D_{\text{ID}}^{\text{train}}$ and $D_{\text{SS}}^{\text{train}}$ instead of the wild data in [1,2]. In addition, we develop the model under the framework of SAM, which is orthogonal to the Bayesian framework in [3].

We provide a comprehensive comparison of the submission and the previous works.

We have updated the above discussion in the main text in our latest submission.

Q2: About the motivation.

Unclear motivation. In lines 266-269, the authors claim that the ideal model should yield low sharpness on both ID and covariate-shifted data. They claim that this cannot be adopted OOD-G method. The logic here is hard for me to follow. Why solely using OOD-G method can not ensure low sharpness for ID and covariate-shifted data? I guess this is a typo. Is the covariate-shifted data here should be replaced with semantic-shifted data?

Reply: Thanks for this valuable feedback. Basically, the OOD-G methods cannot ensure lower sharpness as they focus on loss consistency across domains.

We would like to (1) clarify the reason for OOD-G method could not ensure low sharpness and (2) explain our motivation further.

• OOD-G methods, such as GroupDRO and VRE-x, focus on loss consistency across domains rather than lowering the sharpness. In addition, the differences in data amounts and task difficulty of different domains in OOD-G would result in biased sharpness estimation towards specific domains, which may lead to a suboptimal sharp minima with a similar loss scale [1]. As OOD-G methods do not explicitly consider the sharpness during optimization, training with OOD-G cannot guarantee low sharpness.

We thank the reviewer 5Bu7 for the further comments. We addressed all these comments. Please find the detailed responses below.

Q1: About the sharpness of OOD-G methods.

The authors claim that OOD-G methods cannot ensure lower sharpness as they focus on loss consistency across domains. I am not sure whether such claim is too strong to be correct. I am still confused why OOD-G methods like SAM or mixup can not achieve low sharpness on covariate-shifted data. The mentioned two OOD-G methods, i.e., GroupDRO and VRE-x may not be represnentative enough for all OOD-G methods.

Reply: Thanks for your follow-up feedback. Here, we provide a further explanation as follows.

• Generally, SAM can achieve lower sharpness on covariate-shifted data as it explicitly optimizes the sharpness of the loss. Mixup that interpolates the sample space to maintain the region consistency in prediction, to some extent, can also minimize the sharpness in an implicit manner. In the submission, we mean not to make a strong claim towards these methods.
• On the other hand, the OOD-G methods that optimize the loss via typical optimization methods, like SGD or Adam, would generally lead to sub-optimal performance when converging the sharp and narrow minima [1,2,3,4,5,6,7]. This is only related to the optimizer adopted by different OOD-G methods regardless of their proposed losses.

We apologize for the previous inexact claim and agree with the reviewer's point, once we expand the taxonomy of the OOD-G methods to include SAM and mixup. We have refined the corresponding sentences for better clarification to avoid any controversy in the taxonomy.

[1] On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017.

[2] Loss surfaces, mode connectivity, and fast ensembling of dnns. In NeurIPS, 2018.

[3] Averaging Weights Leads to Wider Optima and Better Generalization. In arXiv, 2019.

[4] Sharpness-aware minimization for efficiently improving generalization. In ICLR, 2021.

[5] Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. In arXiv, 2017.

[6] Fantastic Generalization Measures and Where to Find Them. In arXiv, 2019.

[7] SWAD: Domain Generalization by Seeking Flat Minima. In NeurIPS, 2021.

Q2: About the experimental settings.

Experimental settings. The authors claim that they deploy brightness as data augmentation (Appendix C). I have concerns about whether using brightness augmentation during training can result in information leakage from the test covariate-shifted distribution. Since all the other corruptions in CIFAR10/100-C or ImageNet-C may also alter the brightness of images. As far as I could tell, in standard OOD generalization settings, the test-time covariate-shifted distribution should be kept unknown during training. Besides, the authors seem to tune the augmentation and make such a choice as they said in lines 466-468. Thus, the dependence on manually selected augmentation is a notable limitation that makes the OOD generalization problem more like domain adaption or even a trivial problem.

One of my major concerns, i.e., the experimental settings and dependency on data augmentation remains unanwsered.

Reply: Sorry for the late response due to the missing copy of the prepared response to this concern. To address the reviewer's concern, we would like to clarify:

• We have no intention of tuning a specific augmentation. During training, we chose brightness as the argumentation with the other 17 augmentations for general usage purposes. We do not tune the augmentation based on the performance of the model on the test set. In lines 494-495 (corresponding to lines 466-468 in the original submission), we aim to show that a proper data augmentation can enhance the model's detection and generalization ability simultaneously, but we do not use the test performance as the guiding principle to tune the augmentation. Instead, we still follow the conventional augmentation setup during the training without any specific tuning.

• Regarding the brightness augmentation, to further address the reviewer's concern about information leakage, we conducted an experiment on the CIFAR-10C/100C dataset without using brightness augmentation. Specifically, we adopted Gaussian blur and Gaussian noise to augment the test set of the CIFAR datasets and reported the accuracy of the mixed dataset as follows.

As can be seen from the table above, DR-SAM exceeds SAM in terms of OOD-G accuracy on CIFAR-10C/100C without brightness augmentation.

We appreciate the reviewer's question, and have updated the above discussion in Appendix B.4 to avoid misunderstanding.

Q9: About the experimental results.

Regarding the experimental results, why the results are not compared with recent approaches that has been studied for both detection and generalization? Also, why the comparison has not been made for the Imagenet-200 benchmarks in terms of OOD-G methods in Table 3?

Reply: Thanks for this feedback. Following this suggestion, we conduct experiments on additional baselines, including WOODS and SCONE, with settings aligned with the original submission. We then provide justification for not adopting OOD-G methods on ImageNet-200 benchmarks.

• Overall, DR-SAM achieved leading performance in OOD-G and OOD-D performance on the CIFAR benchmark under the traditional OOD-D setting.

As can be seen from the table above, WOODS and SCONE cannot perform well in CIFAR-100 under the traditional OOD-D setting, and also scraify the OOD-G performance. We have updated the results in Appendix B.3.

• We follow the common setting adopted in [1,2,3,4,5] to conduct OOD-G experiments. Generally, OOD-G methods focus on small-scale datasets like the CIFAR benchmark but on large-scale ones like ImageNet-200. Since we aim to evaluate the model's detection and generalization performance, we conduct OOD-G experiments on the commonly adopted CIFAR benchmark. The scale and difficulty of the ImageNet-200 are much larger and harder than the CIFAR benchmark, which requires a longer optimization time. We are running the experiment and will update once we have the results. We will update the ImageNet-200 experiments to Tab.3 in the main text.

[1] Towards out-of-distribution generalization: A survey. In arXiv, 2021.

[2] Invariant Risk Minimization. In arXiv, 2020.

[3] Out-of-Distribution Generalization via Risk Extrapolation. In ICML, 2021.

[4] In Search of Lost Domain Generalization. In ICLR, 2021.

[5] Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detection. In ICML, 2023.

We thank the reviewer L2A2 for the valuable feedback. We addressed all the comments. Please kindly find the detailed responses below. Any further comments and discussions are welcomed!

Q1: About the typos.

The paper has severe typos, and a thorough proof-read is required. For instance, covariate is written as covariant in many places.

Reply: We apologize for the typos. Following this suggestion, we have gone through a carefully proofread and updated the paper, highlighting the changed part in blue.

Q2: About the README file for the source code.

The provided source code is very hard to reproduce as it does not have a readme file, and to replicate the exact experiment is difficult.

Reply: Thanks for this constructive feedback. Following this suggestion, we have provided a detailed ReadME file in the anonymous link. We provide detailed guidance from the environment setup to the commands used to reproduce the experiment.

Q3: About the related works in the appendix.

There are irrelevant and lot of details in the appendices of the related works section, which is not necessary at all.

Reply: Thanks for this valuable feedback. Following this suggestion, we have refined the appendix to improve its readability and quality. We organize the methods based on their underlying methodology to provide a comprehensive understanding of different methods. For OOD-G, we discuss representative work like IRM and VRE-x and reduce the potential irrelevant discussions. For OOD-D, we reduce the discussion on the post-hoc method and focus on the OE-based method. We shorten the discussion on robust finetuning to reduce the irrelevant details.

Q4: About the OOD-G experiment.

The experiments and results are promising, but it could have been done better by comparing with OOD-G methods too because there is a lack of proof indicating DR-SAM can beat existing OOD-G methods. The results are majorly focused on semantic shifts based methods only.

Reply: Thanks for this valuable feedback. Please note that the OOD-G method is trained on covariate-shifted samples with explicit regulation, which allows it to be robust to potential distribution shifts. We have provided a comparison in Tab.1 and 2 in our original submission, which shows that DR-SAM can beat OOD-G methods in terms of OOD-D performance.

The OOD-G methods can enhance the model's OOD-G performance but sacrifice its ability to identify semantic-shifted samples. As shown in Tab.1 and Tab.2, the FPR@95 and AUROC of the OOD-G approaches have dropped significantly compared with that of the pre-trained model with the post-hoc OOD-D detection method (e.g., KNN, RMDS). In addition, training with OOD-G solely would also decrease the model's ID accuracy.

This submission aims to enhance the model's detection and generalization ability simultaneously. As we identified the paradox at the beginning, solely training with either OOD-G or OOD-D methods cannot ensure the model excels in both tasks, which is important for real-world applications. We provide experiment results to show that DR-SAM can resolve this paradox effectively.

Q5: About the typo.

In line 082, Analysis on logit space: For better OOD_D method enlarges the gap of prediction confidence between D_ID and D_CS. Is this a typo? Shouldn't it be D_ID and D_SS instead of D_ID and D_CS?

Reply: Thanks for this question. Yes, you are correct. It should be "The OOD-D method enlarges the gap of prediction confidence between $D_{\text{ID}}$ and $D_{\text{SS}}$ (better OOD-D) and decreases the model's prediction on $D_{\text{CS}}$ (worse OOD-G)." We have updated it in the revised submission.

Q6: Explaining the Fig.2.

Fig. 2 is not clear. Atleast not very explaining. Why the FPR is in the range of -ve and why the OOD-G accuracy of DR-SAM around 1.7? A delta term is used for both FPR and ACC. What is this delta? Is it the difference? It needs to be clarified both in the caption and the actual. It is difficult for a normal reader to apprehend whats going in this figure.

Reply: Thanks for this constructive feedback. Please note that we report the $\Delta$ term of the model fine-tuned with different methods ($e.g.$, OOD-D and OOD-G) by calculating the performance difference between the pre-trained and fine-tuned models. For example, the DR-SAM's OOD-G accuracy (around 1.7) in Fig.2(a) is calculated by the result presented in Tab.1 (80.50 (DR-SAM) - 79.24(MSP)). The FPR@95 is calculated in the same manner. We have added the explanation to Fig.2's caption in the revised submission.

Q7: Explaining the Fig.5.

In Fig. 5 the sharpness value is larger and in 6 and 7 the sharpness value is smaller. Is this the preferred characteristics of the curves for DR_SAM? Shouldn't the sharpness value in the Fig.5d remain almost steady across the value of rho? Because this characteristic contradicts with the statement made in the line 255-256.

Reply: Thanks for this feedback. Please note that DR-SAM exhibits lower sharpness within specific regions because we prompt the optimizer to do so. We present ablation studies in Fig. 8 (a) and (b) to examine the impact of perturbation strength ($\rho$) on the model's detection and generalization performance. The hyperparameter $\rho$ determines the neighborhood's region that the optimizer focuses on, with larger values of $\rho$ prompting the optimizer to minimize the sharpness over a broader neighborhood. Notably, selecting an appropriate $\rho$ (e.g., $\rho = 0.5$ for CIFAR-10) can simultaneously enhance detection and generalization performance, as evidenced by the flatness observed in Fig. 5 (d).

Q8: About the performance of DR-SAM and vanilla SAM.

Why is the ID and OOD accuracy of DR-SAM less than that of vanilla SAM? As per the result, the gain can only be seen in the AUROC which is the metric for detecting semantic shifts. What about for the covariate-shift part? Shouldn't the OOD accuracy be at least on par or better than the reference and vanilla SAM as per the claim of breaking the detection-generalization paradox? Please clarify.

Reply: Thanks for this valuable feedback. Please note that the employment of auxiliary outliers would cause the degeneration of OOD-G performance, and DR-SAM can alleviate this issue.

We conducted experiments on CIFAR-100 using the same settings as those adopted in Tab 2 in our original submission.

We have the following two observations from the table above:

1. Model training with auxiliary outliers would sacrifice the model's ID and OOD-G performance. Simply cooperating OE with SAM can improve the model's performance, but cannot compete with vanilla SAM.
2. DR-SAM alleviates the side-effect of training with outliers. This indicates the necessity of the $\text{aug}(\cdot)$ for enhancing both detection and generalization performance.

We further conduct experiments on ImageNet-200 to shows that DR-SAM outperform the vanilla SAM in both OOD-D and OOD-G, shown as table follows.

In general, DR-SAM effectively resolves the detection-generalization paradox.

We thank the reviewer L2A2 for the further comments. We addressed all these comments. Please find the detailed responses below.

Q1: About the sharpness of the model.

Q7) Thank you for the answers. But the question was really about steady performance (desired) of DR-SAM across the range of rho where the values of FPR@95 should remain low and OOD-G acc should remain high. Ideally, we are aiming here to achieve lower sharpness value which should equate to lower FPR@95 and higher OOD-G acc, however, this characteristics is not observed for DR-SAM in Fig 5d. Instead, the pretrained model is showing low and steady sharpness across range of rhos.

Reply: Thanks for this question. We would like to answer this question in threefold. The $\rho$ here not only indicate the neighborhood radii, but also act as the hyper-parameter for training the DR-SAM.

• DR-SAM would have better and steady OOD-D and OOD-G performance when optimized with lower $\rho$. As shown in Fig. 8 (a) and (b), a smaller $\rho$ ($\rho$=0.5) is desired to have better detection performance. We also notice that $\rho$ has a relatively weak impact on the model's OOD-G performance within the specific region ($\rho \in [0,1.5]$). Thus, to enhance the detection and generalization performance of the model simultaneously, we choose $\rho=0.5$.
• The sharpness is only related to the OOD-G performance of the model. Lower sharpness indicates higher OOD-G accuracy [1], as we discussed in our original submission in lines 256 to 257. The sharpness shows less of a relationship with the detection performance of the model.
• A steady low sharpness across range of rhos on $D_{\text{ID}}^{\text{train}}$ might not ensure low sharpness on $D_{\text{ID}}^{\text{test}}$ (Fig. 6 (d)) and $D_{\text{CS}}^{\text{test}}$(Fig. 7 (d)). Compared to DR-SAM, the pretrained and OE fine-tuned model shows steady low sharpness on $D_{\text{ID}}^{\text{train}}$ (Fig. 5 (d)) but higher sharpness on $D_{\text{CS}}^{\text{test}}$ (Fig. 7 (d)). As a result, their generalization performance cannot exceed the OOD-G methods.

In general, DR-SAM would have better performance when $\rho$ is smaller ($\rho=0.5$). A steady low sharpness across range of $\rho$ on $D_{\text{ID}}^{\text{train}}$ might not leads to better generlization on $D_{\text{CS}}^{\text{test}}$.

[1] Sharpness-aware minimization for efficiently improving generalization. In ICLR, 2021.

Q2: About the performance of DR-SAM and vanilla SAM on CIFAR-100.

Q8) Thanks for the reply. I can understand why the SAM+OE can beat vanilla SAM, because of its capability to introduce extra regularization with outliers for dealing with semantic shifted samples. And the degeneration of OOD-G with SAM+OE is understandable. However, my concern here is about DR-SAM (which does not use OE) should atleast have FPR@95 and OOD-G accuracy better than the vanilla SAM. I see that the authors have updated the new results for Imagenet-200, whereas for CIFAR-100 case, according to the table SAM is still better than DR-SAM across all aspects.

Reply: Thanks for this question. We would like to respectfully clarify the misunderstanding of the reviewer regarding the FPR@95 metric and DR-SAM. Actually, our DR-SAM does employ OE. Regarding performance, DR-SAM is better than SAM in terms of OOD-D metric FPR@95 (lower is better), and achieves competitive OOD-G accuracy compared to SAM on CIFAR-100.

• DR-SAM employs OE to enhance the model's detection ability. In Algorithm 1 of the submission, we show that DR-SAM employs auxiliary outliers to enhance the model's detection ability. By conducting experiments on SAM+OE, We show that without $aug(\cdot)$, simply cooperating SAM with OE cannot enhance the detection and generalization ability simultaneously. The above arguments further justify the vadality of the proposed DR-SAM in handling the detection-generalization paradox.

• DR-SAM exceeds the OOD-D performance (FPR@95) of SAM by a large margin. As the table above shows, DR-SAM alleviates the issue of training with OE by improving the model's generalization ability (OOD-G accuracy). We should also note that as our goal is to achieve good detection and generalization performance simultaneously, the results should be considered collectively in both measures.

In general, DR-SAM enhances the model's generalization and detection ability by training with $aug(D^{\text{train}}_{ID})$ and auxiliary outliers (OE).

We thank the reviewer, iCQm, for the valuable feedback. We addressed all the comments. Please kindly find the detailed responses below. Any further comments and discussions are welcomed!

Q1: About the contribution.

The proposed method DR-SAM lacks innovation. It appears to combine OE and SAM as optimization objectives, with an additional data augmentation to calculate perturbation factor $\epsilon$.

Reply: Thanks for this comment. We would kindly note that simply combining OE and SAM without $\text{aug}(\cdot)$ cannot simultaneously enhance detection and generalization performance. Here, we first clarify the difference between DR-SAM and standard SAM. Second, we show that simply combining SAM with OE cannot handle the detection-generalization paradox. Finally, we highlight our contribution.

• A key difference between DR-SAM and the standard SAM is employing $\text{aug}(\cdot)$ only during the perturbation generation phase. Analytically, we compare the difference between DR-SAM and standard SAM in the following table from three: (1) the ultimate optimization target, (2) the data sources for acquiring perturbation, and (3) the parameter optimization stage.

Q3: About the effect of data augmentation.

In Algorithm 1, does using data augmentation to calculate the perturbation factor $\epsilon$ in line 4 affect the model's performance on $D^{\text{test}}_{\text{ID}}$ compared to vanilla SAM?

Reply: Thanks for this valuable feedback. The employment of auxiliary outliers cause the degeneration on $D^{\text{test}}_{\text{ID}}$, and data augmentation can alleviate this issue.

We conducted experiments on CIFAR-100 using the same settings as those adopted in Tab 2 in our original submission.

We have the following two observations from the table above:

1. Model training with auxiliary outliers would sacrifice the model's ID performance. Simply cooperating OE with SAM can improve the model's ID performance, but cannot compete with vanilla SAM. In addition, SAM+OE also fails to achieve competitive ID performance with vanilla SAM.
2. DR-SAM alleviates the side-effect of training with outliers. This indicates the necessity of the $\text{aug}(\cdot)$ for enhancing both detection and generalization performance.

In general, DR-SAM effectively resolves the detection-generalization paradox.

Q4: Ablation study on data augmentation.

Is the capability for OOD-G derived from data augmentation or SAM? The authors should include relevant ablation experiments to clarify this.

Data augmentation seems to be the most significant innovation in DR-SAM, and the authors should include experiments to demonstrate the impact of having or not having data augmentation.

Reply: Thanks for this valuable suggestion. Following this suggestion, we empirically show that the capability for OOD-G of DR-SAM is derived from data augmentation. We conduct ablation experiments on CIFAR-10 and ImageNet-200.

As can be seen from the table above, compared to the MSP baseline, vanilla SAM can enhance the model's OOD-G performance in CIFAR-10 but fails in more fine-grained datasets like ImageNet-200. Whereas DR-SAM (w/o $\text{aug}(\cdot)$) would degenerate the model's OOD-G and OOD-D performance compared to the DR-SAM. The above experiments indicate that the capability for OOD-G is derived from data augmentation rather than SAM.

We also update the results in our revised submission in Appendix B. We kindly refer the reviewer to check our latest version of the submission.

• Simple "SAM+OE" cannot enhance detection and generalization simultaneously. To further justify the validity of $\text{aug}(\cdot)$, we simply cooperate auxiliary outliers to obtain perturbation for SAM optimization, termed as "SAM+OE." We compare the SAM+OE with standard SAM and proposed DR-SAM as follows:

As can be seen from the tables, simply combining SAM+OE cannot enhance both targets simultaneously, indicating the validity of introducing the $\text{aug}(\cdot)$ for creating a challenging perturbation aware of the worst case for generalization under covariate shift.

• The proposed DR-SAM is based on the previous analysis of the detection-generalization paradox we identified. Specifically, we identify the detection-generalization paradox, arising from the prevailing OOD-D and OOD-G methods. We analyze the reason behind the paradox based on three informative variables: representation, logit, and the loss of space with $D_{ID}, D_{CS},$ and $D_{SS}$. We reveal that an ideal model should have a flatter neighborhood around the converging area ($i.e.,$ low sharpness) on both $D_{ID}, D_{CS}$ data. This motivated us to develop the DR-SAM to search for such an ideal model.

In general, the comparison and experiments justify DR-SAM's novelty and the necessity of $\text{aug}(\cdot)$, as simply cooperating SAM with OE cannot resolve the detection-generalization paradox.

We have added the above comparison and experiments to the Appendix A. We kindly refer the reviewer to check the latest version of our submission.

Q2: About the typos.

The analysis of the method is not detailed enough. In Algorithm 1, should $f_\theta + \epsilon$ in lines 3 be $f_\theta$?

Reply: Thanks for this valuable feedback. Yes, it should be $f_\theta$ in lines 3. We would like to explain the method by describing it in three steps as follows:

1. Acquire perturbation: We utilize the augmented samples with auxiliary outliers to enhance the model's ability to identify covariant-shifted samples from semantic-shifted ones.
2. Calculate the gradient: We then pose the perturbation to the model and calculate the gradient using in-distribution samples without aug(·) and outliers to preserve the benefits of the training trajectory for OOD-D.
3. Update model's parameters: We update the model's unperturbed parameters using the gradient signal from the previous step to improve generalization and detection more harmoniously.

The analysis in Sec.3 indicates that an ideal model should possess low sharpness on both $D_{ID}$ and $D_{CS}$ data. We thereby propose DR-SAM to enhance the model's detection and generalization ability simultaneously. The pipeline illustrated above allows us to create a challenging perturbation aware of the worst case for generalization under covariate shift while exploring the separation between $D_{ID}$ and $D_{SS}$.

Dear Reviewer iCQm,

Thank you very much for your time and valuable comments.

In the rebuttal period, we provided detailed responses to all your comments and questions point-by-point. Specifically, we

• clarify the difference between DR-SAM and standard SAM. (Q1)
• amend typos and explain our method further. (Q2)
• explain the effect of data augmentation on $D_\text{ID}^{\text{test}}$. (Q3)
• conduct ablation studies on data augmentation. (Q4)

Would you mind checking our responses and confirming whether you have any further questions?

Any comments and discussions are welcome!

Thanks for your attention and best regards.

Dear Reviewer iCQm,

The authors have submitted their rebuttal. Could you please take a moment to review their response and confirm whether your concerns have been addressed? Thank you for your efforts and contributions.

Best regards,

Your Area Chair