Why SAM finetuning can benefit Out-of-Distribution Detection?

1. This article seems to solely focus on adapting SAM for the task of OOD detection, a point of innovation that appears rather weak to support the entirety of the paper.
2. The authors claim that there is no need for real OOD data; however, looking at the form of the SAM loss, it seems that utilizing real OOD data could work just as well. I am curious if employing real OOD data for fine-tuning in conjunction with SAM loss could potentially yield better results.
3. The results presented in Tables 1, 2, and 3 indicate that SFT may not always achieve optimal performance across all OOD datasets, a phenomenon that the authors have not explained in detail.
4. A number of recent related works have not been compared in this study, including DICE [1], ViM [2], ASH [3], NPOS [4], and CIDER [5].

[1] Sun, Yiyou, and Yixuan Li. "Dice: Leveraging sparsification for out-of-distribution detection." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[2] Wang, Haoqi, et al. "Vim: Out-of-distribution with virtual-logit matching." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[3] Djurisic, Andrija, et al. "Extremely Simple Activation Shaping for Out-of-Distribution Detection." The Eleventh International Conference on Learning Representations. 2022.

[4] Tao, Leitian, et al. "Non-parametric Outlier Synthesis." The Eleventh International Conference on Learning Representations. 2022.

[5] Ming, Yifei, et al. "How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?." The Eleventh International Conference on Learning Representations. 2022.

1. this paper doesn't have enough novelty.
2. The compared methods mostly are before 2022.
3. This paper lack enough reference, like "Out-of-distribution detection with an adaptive likelihood ratio on informative hierarchical VAE"

• The paper does not provide a comparison with OE methods that rely on OOD data for training. Although SFT does not introduce external OOD data, it generates partial OOD data, and both SFT and OE methods belong to the category of incorporating OOD data into training. Therefore, a comparison with the OE method is indispensable. It is not mandatory to surpass the OE method, but the efficacy of the SFT method can be demonstrated.
• The main contribution of the paper stems from the generation of pseudo OOD data, which has achieved significant performance improvement. However, it is still necessary to elucidate the training and computational costs associated with SFT compared to other methods.

• The paper's "Related Work" section could provide more context and insights into how the proposed method relates to existing research. In particular, except for the OOD detection with discriminative models paragraph, all paragraphs are written as a catalog of related papers. Moreover, the link between generative models for OOD detection and the proposed method is unclear.
• The paper doesn't provide a clear distinction between the improvements from Outlier Exposure fine-tuning and the optimization benefits of Sharpness-aware Minimization. This point is critical as OE is a strong baseline and might be the main source for OOD detection improvement.
• Some parts of the paper lack clarity and are hard to follow. In particular, is Proposition 1 related to the upper bound of the Energy score or the upper bound of the variation of the Energy score? Moreover, the link between the bound and the better separability is not clear in the main paper and should be made explicit.
• The method appears to struggle in near-OOD scenarios, as seen in the results for the iNaturalist dataset, where it falls significantly below top-performing baselines. I guess this behavior is expected due to the proximity of near-OOD samples with ID ones thus limiting the impact of the Sharpness-aware Fine-Tuning.

Writing:

The paper is missing important definitions and descriptions of relevant algorithms and terms, and generally lacks the necessary precision to be able to fully understand the paper. This includes: SAM, energy, the "statistical distance" $D$ (from equation 2), the precise definition of $\mathcal{D}_\text{out}$, summaries of the relevant algorithms. Even when provided with a citation that includes the relevant definitions, I think it would be very helpful to provide at least a brief summary of each term and to clarify terminology.

In addition, at times, the notation is a bit confusing: $f$ is used as both a neural network and a loss function. In equation 3, the parameters are written as $w^t$ but $t$ is never used.

Theory:

The statement of Proposition 1 is lacking some formality. As mentioned above, "Energy" should be properly defined. In addition, it was unclear to me (before reading the proof) what the authors mean by "the change during the fine-tuning process", how $\Delta$ is defined, and what the authors mean by "an irreducible constant". In addition, the comment that "the upper bound of the change of Energy score can improve the separation between ID and OOD" should not be included in the statement of the proposition. In addition, the fine-tuning process should be clearly and precisely defined.

The statement is also only proved for a single step of fine-tuning, and not for the full fine-tuning process.

The proof (and statement) of Proposition 1 is extremely similar to Theorem C.5 in Mueller et al. (2023), and the majority of the proof is copied verbatim. I think it would help to clearly delineate the contribution of this paper and that of prior work and cite the relationship in the main paper.

I'm also not sure I understand the purpose of the Proposition: the bound establishes a relationship between $\Delta(\operatorname{Energy})$ and $\Delta(\operatorname{max}(g))$. As I understand, the authors argue that since $\Delta(\operatorname{max}(g))$ is empirically verified to be smaller for OOD points than ID points, and thus $\Delta(\operatorname{Energy})$ will also be smaller. However, this does not follow from establishing only an upper bound, and secondly, why not directly verify that $\Delta(\operatorname{Energy})$ is smaller, and skip the comparison with $\Delta(\operatorname{max}(g))$ (which the authors do in Figure 4)?

Empirical:

In order to have a fair comparison, it's necessary to tune hyperparameters (learning rate, batch size, weight decay, and $\rho$) which are likely to have a potentially large impact on the model, and may need to be tuned differently for different models, datasets, and algorithms. (It's worth noting that the authors do include an ablation where they vary $\rho$ for one architecture). There are a couple potential issues with the results (see questions section below), that cast doubt for me as to whether the results are sound.

References:

Maximilian Mueller, Tiffany Vlaar, David Rolnick, and Matthias Hein. Normalization layers are all that sharpness-aware minimization needs. arXiv preprint arXiv:2306.04226, 2023.