$\Delta \mathrm{Energy}$: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization

R1. Reply to ''The motivation is not sufficiently clear'':

We apologize for any confusion regarding the motivation and the issue we aim to address. We provide clarification as follows:

Problem Setting of Our Target Issue: In real-world deployment scenarios, models often encounter a mixture of distribution shifts, including both covariate shifts (e.g., changes in style or environment) and semantic shifts (e.g., unseen classes during fine-tuning). To reflect this reality, our work considers a more practical and challenging setting where the test distribution consists of both closed-set OOD data (covariate shifts) and open-set OOD data (semantic shifts).

Under this setting, the model is required to: (1) Maintain accurate classification of covariate-shifted samples—a capability referred to as OOD generalization; (2) Recognize semantic-shifted OOD samples and abstain from blindly classifying them into any ID class—highlighting the importance of OOD detection.

This setting has attracted increasing interest in recent years [1-4]. Importantly, this problem is especially critical for high-stakes applications (e.g., medical imaging or autonomous driving), where models must simultaneously generalize across domains and detect unseen semantic categories.

Clarification of the Motivation and Effectiveness of Our Method: We have explained the benefits of $\Delta\mathrm{Energy}$ and EBM for OOD detection and OOD generalization from both theoretical and empirical perspectives. The theoretical motivations are thoroughly discussed in R1 and R2 to Reviewer 3pyx, particularly through the logical flow behind the theorems. Due to space limitations, we kindly refer the reviewer to R2 to Reviewer 3pyx for the in-depth explanation of our theoretical motivations. Empirical evidence supporting the effectiveness of EBM is presented in Tables 3 and 4 of the manuscript, and in Tables 2-1 and 2-2 in response to Reviewer 3pyx, where consistent improvements are observed across both OOD tasks.

[1] Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection ICML, 2023.

[2] Aha: Human-assisted out-of-distribution generalization and detection. NeurIPS, 2024.

[3] Diversify: A general framework for time series out-of-distribution detection and generalization. TPAMI, 2024

[4] Bridging ood detection and generalization: A graph-theoretic view. NeurIPS, 2024

R2. Reply to ''Pruning-based methods are not discussed':

We apologize for any confusion caused by the lack of discussion on pruning-based OOD detection methods. We will include a corresponding discussion and cite all mentioned papers. Below, we highlight the key differences between our method and related works such as DICE and ASH:

• Different problem setting: Our method connects $\Delta\mathrm{Energy}$ with domain-consistent Hessians and is specifically designed to jointly optimize OOD generalization and OOD detection, backed by theoretical guarantees. In contrast, DICE and ASH primarily target OOD detection by removing noisy weights or activations, without addressing the generalization aspect. Moreover, our method performs V-L re-alignment and explicitly measures energy changes, while prior methods reshape the model outputs through pruning without considering such energy-change criteria.

• Different model assumptions: Our method is tailored for CLIP-based scenarios, where the output logits tend to be uniform-like, posing greater challenges for OOD detection. We particularly focus on hard OOD detection settings, where the semantic gap between ID and OOD samples is small. In contrast, prior works such as DICE and ASH primarily focused on traditional visual models, where the logit corresponding to the ground-truth label is often much higher than the others.

R3. Reply to ''Concerns about the method’s performance':

Reply to the concern on comparisons:

We have compared our method with recent CLIP-based state-of-the-art OOD detection methods, including NegLabel (ICLR 2024) and CSP (NeurIPS 2024). As shown in Tables 1,2, and 7 of our manuscript, our method significantly outperforms both NegLabel and CSP, demonstrating its superiority in distinguishing different semantics in hard OOD detection scenarios. The underlying reasons for the inferior performance of NegLabel and CSP have been discussed in Lines 239-246. In contrast, aligning with the results in Theorem B.1, our method is effective in hard OOD detection scenarios, where the number of classes is high and the similarity gap between closed-set and open-set samples is small.

Reply to the concern that gains are less significant on some data:

We attribute the modest gains on PACS and VLCS to two key factors: (1) the inherent discrepancy between ID and easily separable open-set OOD samples is already substantial in these benchmarks; and (2) for hard-to-detect open-set OOD samples, the maximum cosine similarities may not satisfy the condition: s_\hat{y_1}(x_{ID}) > s_\hat{y_1}(x_{OOD}). Thus, the benefit of amplifying ID-OOD separation using $\Delta\mathrm{Energy}$ is diminished. This outcome aligns with our theoretical findings in Theorem B.1, which suggest that $\Delta\mathrm{Energy}$ is more effective in challenging OOD detection scenarios—specifically, when the similarity gap between closed-set and open-set samples is small. Such cases are more prevalent in open-world settings, which we evaluate using ImageNet-based benchmarks. The results, presented in Tables 1, 3, 6, and 7, show substantial performance gains, demonstrating the effectiveness of $\Delta\mathrm{Energy}$ in handling hard OOD scenarios.

R4. Reply to ''Doubt on the claim that $\Delta\mathrm{Energy}$ outperforms MCM':

As shown in the experimental results in Tables 1, 2, and 6 in the manuscript, our method consistently outperforms the energy-based scoring baseline. The superior performance of $\Delta\mathrm{Energy}$ over the raw energy-based method indicates that the $\Delta\mathrm{Energy}$ formulation itself contributes meaningfully to enhanced OOD detection performance.

Moreover, in Theorem 3.2, we provide a theoretical justification showing that $\Delta\mathrm{Energy}$ leads to stronger separation between ID and open-set OOD samples than MCM. We emphasize that this theoretical analysis is conducted under CLIP-like model conditions—where the output logits are typically uniform-like (as noted in MCM). This differs significantly from visual models trained with cross-entropy loss, where the logit corresponding to the ground-truth label is often much higher than the others. As noted in the original MCM paper, for CLIP-like models, applying softmax helps sharpen the uniform-like logits output and increases the separation between ID and OOD samples. By tuning the temperature in the softmax, MCM significantly outperforms the standard energy-based scoring.

Taken together, both our theoretical and empirical results support the conclusion that, in the context of CLIP-like models, the $\Delta\mathrm{Energy}$ formulation itself plays a critical role in enhancing OOD detection.

R5. Reply to ''The method is merely an incremental combination of an ASH-style pruning technique with the Energy on top of MCM:

Our work offers both theoretical advancements and a broader application scope across two OOD tasks within the vision-language model (VLM) context—namely, OOD generalization and OOD detection—rather than simply combining ASH-style pruning with energy-based methods. The key distinctions are as follows:

• Different model assumptions and data settings: We focus on CLIP-based models, where the output logits tend to be uniform-like, making OOD detection more challenging. Moreover, we target hard OOD detection scenarios, where the semantic gap between closed-set and open-set OOD samples is small and the model must robustly handle both covariate and semantic distribution shifts.

• Different methodology: Our method performs vision-language re-alignment and introduces a novel post-hoc OOD score that explicitly measures energy changes. In contrast, ASH modifies model outputs by pruning activations and relies on existing post-hoc OOD scores, such as the energy score or MSP, without incorporating energy-change criteria.

• Unified framework for both OOD generalization and OOD detection: We theoretically prove that the $\Delta\mathrm{Energy}$-based lower-bound maximization (EBM) can theoretically improve both OOD tasks, explicitly increasing $\Delta\mathrm{Energy}$ for closed-set data and learning domain-invariant information between original data and masked data. In contrast, MCM and Energy Score focus solely on OOD detection without addressing OOD generalization, and ASH is an empirical, pruning-based method tailored to visual models trained with cross-entropy loss, lacking theoretical support for application to VLMs.

R6. Reply to ''Why do the Energy and MaxLogit performance differ:

In CLIP-like models, the output logits tend to be uniform-like, meaning that the similarity scores across different classes are relatively close in magnitude. In such cases:

• The Energy score, which is based on the log-sum-exp of all logits, becomes more sensitive to the overall distribution of logits output and less discriminative when the logits are not sharply separated. This can reduce its ability to distinguish between ID and OOD samples.
• In contrast, MaxLogit focuses solely on the maximum class score, which provides a sharper and more focused signal for confidence—even if all logits are close. This makes it more robust in detecting anomalies when the distribution of scores across classes is flat, as it picks out the highest response directly without aggregating all logits.

Additional empirical evidence of MaxLogit surpassing Energy in CLIP-based cases can be found in Table 1–2 of the CLIPN paper [5].

[5] Clipn for zero-shot ood detection: Teaching clip to say no. ICCV, 2023.

Reply to Q1:

The MSP and MCM are different in the paper that arises from the use of different temperature coefficients in the softmax computation. To clarify, we first define the CLIP similarities and classification logits as follows:

We denote the zero-shot text features as $z_T$ and the zero-shot image features as $z_I$. The CLIP similarity is computed as $\langle z_I, z_T \rangle$, and the CLIP classification logits are computed as $T \cdot \langle z_I, z_T \rangle$, where the temperature coefficient $T$ used by CLIP is 100. In Tables 1 and 2, methods including Energy, MSP, MaxLogit, ReAct, ODIN, and ΔEnergy are computed directly over CLIP’s classification logits, whereas the MCM method is computed based on CLIP similarities (i.e., $T=1$), as recommended in the original MCM paper. Therefore, the discrepancy between MSP and MCM arises from the use of different temperature coefficients in the softmax computation in Tables 1 and 2.

To mitigate the influence of temperature and to better demonstrate the contribution of ΔEnergy under different temperature coefficients, we conduct ablation studies on MSP, Energy, and ΔEnergy. From Table 4-2, our method obtains the best results when we use default CLIP classification logits (T=100). Compared to the original Energy scoring method, the performance gains of ΔEnergy over Energy are around 10%-35%, showcasing the effectiveness of ΔEnergy in CLIP-based zero-shot OOD detection.

Table 4-2

Reply to Q2:

When $c=1$, the ΔEnergy score is fundamentally different from the maximum CLIP similarity before softmax (i.e., MaxLogit)—both in formulation, theoretical justification, and empirical performance. Specifically, the maximum CLIP similarity does not account for the distribution of logits, while ΔEnergy explicitly captures this distribution by computing the energy change before and after re-aligning CLIP. We elaborate on the superiority of our method below:

1. Evidence that ΔEnergy outperforms MaxLogit: As noted in the original MCM paper, for CLIP-like models, applying softmax helps sharpen the uniform-like logits output and increases the separation between ID and OOD samples. As shown in Tables 1 and 2 in the paper, MCM significantly outperforms the MaxLogit method. Since our method theoretically outperforms MCM—as supported by Theorems 3.2 and 3.3—it naturally achieves better results than MaxLogit, which is also empirically demonstrated in Tables 1 and 2.

2. Furthermore, we emphasize our unified ΔEnergy-based lower bound maximization framework, which enhances both OOD detection and OOD generalization. This formulation is supported by theoretical guarantees and leads to improved robustness under various distribution shifts. Our theoretical contribution—which connects OOD generalization and OOD detection through a unified ΔEnergy-based loss—highlights the novelty and the practical significance of ΔEnergy. Detailed theoretical results can be found in Theorem 3.5 and Proposition 3.6.

Reply to Q3:

ΔEnergy is fundamentally different from using the sum of the top-k logits or similarity as OOD score—both in formulation and empirical performance. The top-k baseline does not consider the overall logit distribution, whereas ΔEnergy captures the distribution by measuring the energy change.

We conducted experiments using the sum of the top-k logits (with k = 1, 2, 3) as OOD scores for ImageNet-based OOD detection. Note that when we set the temperature $T=1$, the OOD score is computed directly from the CLIP similarities without any scaling. The results are reported in Table 4-3. As shown in the table, ΔEnergy consistently outperforms all top-k logit-based baselines by 30%-40%. The substantial performance gap indicates that modeling the variation in energy behaviour offers strong discriminative power for OOD detection—information that is not captured by raw logit or similarity magnitudes alone.

Table 4-3

R1. Reply to ''This work relies on the MCM method, and the improvement is not significant on some data'':

The proposed $\Delta\mathrm{Energy}$ computes the change in energy score resulting from re-aligning the vision-language modalities by setting the top-2 maximum cosine similarities to zero. We would like to clarify the difference between our $\Delta\mathrm{Energy}$ and the MCM method, and we also provide an explanation for why the performance improvements are less significant on PACS and VLCS. This discussion will be included in a later version of the manuscript to improve clarity and completeness.

We highlight the key differences between our method and MCM as follows:

1. Theoretical support for the OOD detection capability of $\Delta\mathrm{Energy}$ compared to MCM: We have theoretically demonstrated in Theorem 3.2 that the $\Delta\mathrm{Energy}$ score for ID data is consistently larger than that for OOD data, indicating its effectiveness and discriminative power for OOD detection. Compared to MCM and raw energy scores, $\Delta\mathrm{Energy}$ amplifies the gap between ID and OOD data, a property validated by both theoretical analysis (Theorem 3.2) and empirical results (Tables 1, 2, 5 and 7).
2. A $\Delta\mathrm{Energy}$-based optimization algorithm (EBM) for both OOD detection and generalization: Further performance gains can be achieved through our $\Delta\mathrm{Energy}$ lower-bound maximization algorithm (EBM). We have both theoretically and empirically shown that EBM not only enhances OOD detection but also secretly optimizes for improved OOD generalization. Theoretical support is provided in Theorem 3.5 and Proposition 3.6. This ''feed two birds with one scone” approach is novel compared to MCM, which focuses solely on detection.

Regarding the observation that improvements are less significant on PACS and VLCS, we attribute the more modest gains to two key factors: (1) the inherent discrepancy between ID and easily separable open-set OOD samples is already substantial in these benchmarks; and (2) for hard-to-detect open-set OOD samples, the maximum cosine similarities do not satisfy the condition: s_\hat{y_1}(x_{ID}) > s_\hat{y_1}(x_{OOD}). As a result, the benefit of amplifying this separation using $\Delta\mathrm{Energy}$ is reduced. This outcome is consistent with our theoretical analysis in Theorem B.1 (Lines 662–668), which suggests that $\Delta\mathrm{Energy}$ is more effective in hard OOD detection scenarios—specifically, when the similarity gap between closed-set and open-set samples is small. In contrast, when the distributional shift between closed-set and open-set data is already pronounced, the benefit of enhancing ID–OOD separability through $\Delta\mathrm{Energy}$ diminishes. However, we emphasize that hard OOD scenarios are more prevalent in open-world settings, which we evaluate using ImageNet-based hard OOD detection benchmarks. The results, presented in Tables 1, 3, 6, and 7, show substantial performance gains, demonstrating the effectiveness of our method.

R2. Reply to ''Briefly describe the idea of the theorem, and it is not clear why EBM can help OOD tasks'':

Following your suggestion, we will improve the writing of methodology in the revised paper to improve its readability. Below, we briefly describe the main ideas behind each theorem and explain why Equation 4 is beneficial for OOD tasks.

Brief description of the idea behind each theorem:

Theorem 3.2 provides a formal guarantee that $\Delta\mathrm{Energy}$ can provably surpass the widely-used OOD detection method MCM by amplifying the separation between ID and open-set OOD data.

Theorem 3.3 provides a formal guarantee that $\Delta\mathrm{Energy}$ can provably reduce the false positive rate (FPR) in open-set OOD detection compared to MCM.

Theorem 3.4 implies that minimizing the EBM loss can increase the lower bound of $\Delta\mathrm{Energy}$ for closed-set classes, which effectively enhances the separation between ID and open-set OOD data.

Theorem 3.5 demonstrates that minimizing the EBM loss can lead to domain-consistent Hessians of classification loss between original and masked samples, which serves as a strong indicator for OOD generalization.

Proposition 3.6 shows that minimizing EBM provides a bound on the performance gap between ID data and closed-set OOD data, indicating that optimizing $\Delta\mathrm{Energy}$ for ID data implicitly promotes OOD generalization.

Why Equation 4 can help OOD tasks:

Equation 4 introduces a $\Delta\mathrm{Energy}$-based bound maximization function (EBM) during the fine-tuning process, which aims at increasing the lower bound of $\Delta\mathrm{Energy}$ score for closed-set classes, as formally established in Theorem 3.4. We explain why Equation 4 is beneficial for OOD tasks from both theoretical and empirical evidence.

Theoretical Perspective: (a) For OOD detection: minimizing Equation 4 effectively increases the $\Delta\mathrm{Energy}$ score for ID data, thereby enhancing the separation between ID and open-set OOD samples. This improved separability facilitates more accurate OOD detection. (b) For OOD generalization: as shown in Theorem 3.5, minimizing Equation 4 leads to domain-consistent Hessians of the classification loss between original and masked samples, which is a strong indicator of OOD generalization. We have also provided detailed and intuitive explanations of the relationship between the EBM and domain-consistent Hessians, as well as the mathematical logic flow, in R2 to Reviewer 3pyx.

Empirical Evidence: We also empirically demonstrate the impact of the EBM loss on both OOD tasks through an ablation study on the coefficient $\lambda_0$. The results, shown in Table 2-2, indicate that models trained with EBM loss regularization (i.e., $\lambda_0 \neq 0$) achieve consistently better performance in both AUROC and OOD accuracy compared to the baseline without EBM. This confirms the effectiveness of EBM in jointly optimizing for OOD generalization and OOD detection.

R3. Reply to ''the application scenario of EBM'':

Our EBM framework follows a common practice in few-shot OOD generalization settings, where only limited source domain samples are available. Few-shot OOD generalization has been extensively studied in prior works. However, these existing methods primarily focus on generalization to closed-set OOD data. In contrast, real-world deployment environments typically involve a mixture of distribution shifts, including both covariate shifts (e.g., changes in style or domain) and semantic shifts (e.g., unseen classes during fine-tuning). To better reflect this reality, our work considers a more practical and challenging setting: the test distribution is composed of both closed-set OOD data (covariate shifts) and open-set OOD data (semantic shifts). This setting has attracted increasing interest in recent years [1-5]. Importantly, this problem is especially critical for high-risk applications (e.g., medical imaging or autonomous driving), where models must simultaneously generalize across domains and detect unseen semantic categories.

[1] Bai, Haoyue, et al. "Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection." ICML, 2023.

[2] Bai, Haoyue, et al. "Aha: Human-assisted out-of-distribution generalization and detection." NeurIPS, 2024.

[3] Gwon, Kyungpil, et al. "Out-of-distribution (OOD) detection and generalization improved by augmenting adversarial mixup samples." Electronics, 2023.

[4] Zhu, Lin, et al. "CRoFT: robust fine-tuning with concurrent optimization for OOD generalization and open-set OOD detection." ICML, 2024.

[5] Lu, Wang, et al. "Diversify: A general framework for time series out-of-distribution detection and generalization." TPAMI, 2024

R4. Reply to ''How to detect the satisfaction of theorem conditions'':

The key assumption underlying the EBM method is formalized in Equation 7. Since the cosine similarity after re-alignment is defined as $\tilde{s}_{\hat{y}_j}(\mathbf{x_i})= 0$, Equation 7 simplifies to:

$\sum_{i=1}^N e^{s_{\hat{y}_1}(\mathbf{x}_i)/\tau} \geq$

$\sum_{i=1}^N [(e^{\varepsilon_E} - 1) e^{-E_2(\mathbf{x}_i)}+1]$

where $s_{\hat{y}_1}(\mathbf{x_i})$ is the maximum cosine similarity for input $\mathbf{x_i}$, $\varepsilon_E$ is the upper bound of EBM loss, and $E_2(\mathbf{x_i})$ is the energy score for masked feature.

This inequality implies that, after fine-tuning, the maximum cosine similarity scores for ID samples (i.e., $s_{\hat{y_1}}$) must be sufficiently large. At the same time, it requires the bound $\varepsilon_E$ (in the constraint $\mathcal{L}_{\Delta E} \leq \varepsilon_E$) not to be too loose. These conditions are both reasonable: the first encourages confident classification, while the second ensures effective optimization of the EBM loss. Since all terms in Equation 7 are computable during fine-tuning, the theoretical condition is verifiable in practice. Overall, both the practical application scenarios and the verifiable theoretical conditions highlight the broad applicability of our method.

R5. Reply to ''Lack of parameter analysis experiments'':

In our manuscript, we have presented ablation study results for the hyperparameters $c$ and $p$ in Tables 9 and 10, respectively. Additionally, the ablation results for $\lambda_0$ are provided in Table 2-2 of R3 to Reviewer 3pyx. Due to space constraints, we kindly refer the reviewer to our response (R3) to Reviewer 3pyx for details regarding the hyperparameter selection strategy.

R6. Reply to Questions:

A1. The underlying rationale behind $\Delta\mathrm{Energy}$ and EBM is elaborated in R2.

A2. The hyperparameter selection strategy is detailed in R5.

A3. We discussed the reasons why the improvement is small on some datasets in R1.

A4. The practical application scenarios and how to verify theoretical conditions are explained in R3 and R4.

We thank the reviewer for the valuable feedback. Your suggestion is critical to strengthening our work! Regarding the concerns about the current state of our paper, we provided detailed explanations in our rebuttal. The relevant discussions will be incorporated into the revised version of the paper to improve clarity and completeness. We apologize for any remaining confusion and offer additional clarification below.

Regarding concern 1:

As shwon in in response R2, we discussed the correlation between EBM and improvements in classification performance from both theoretical and empirical perspectives. To further clarify this relationship, we will incorporate intuitive explanations into the revised version—particularly the connection between EBM and the learning of domain-invariant information, along with illustrative and intuitive explanations of the underlying mathematical reasoning. We thank the reviewer again for raising this important point.

Regarding concern 2:

Following your suggestion, in addition to the ablation studies in Tables 9 and 10 of the paper and Table 2-2 in our response to Reviewer 3pyx, we provide additional ablation experiments on the number of few-shot samples in the ImageNet-based setting. The results are reported in Table 3-1. The consistent improvements observed across both OOD tasks under various few-shot settings demonstrate the robustness of our method. To enhance clarity, we will also provide graphs to visualize the experimental results.

Table 3-1 Ablations on the few shot number.

Regarding concern 3:

We have explicitly explained how to verify the conditions (Eq 7) of Theorem 3.4 in response R4. Since all terms in Equation 7 are computable during fine-tuning, the theoretical condition is practically verifiable.

To further improve clarity, we outline the verification procedures for the theorems related to OOD detection as follows:

• For Theorem 3.2, it is straightforward to verify the condition that the maximum cosine similarity for an ID sample is greater than that of an open-set OOD sample, i.e., $s\_{\hat{y}\_1}(\mathbf{x}\_{\text{ID}}) > s_{\hat{y}\_1}(\mathbf{x}\_{\text{OOD}})$.
• For Theorem 3.3, under the same data assumptions as in MCM, we only need to check Assumption C.1, which is also easily verifiable in practice, as all required terms in Assumption C.1 are directly computable.

Regarding concern 4:

We apologize for the typo. The ∆Energy-based bound maximization is defined in Eq 4, and we will correct this in the revised version of the paper.

Regarding concern 5:

The key differences between our method and MCM as highlighted in R1 will be incorporated into our paper. Although our method requires computing the maximum similarity, it is fundamentally different from MCM in terms of formulation, theoretical justification, and empirical performance. The significant performance gap observed in Tables 1 and 2 indicates that modeling the variation in energy behavior provides strong discriminative power for OOD detection—information that cannot be captured by raw logit values or similarity magnitudes alone.

Regarding concern 6:

We have provided theoretical insights demonstrating that the proposed ΔEnergy improves both OOD detection and OOD generalization in this paper.

(a) OOD Detection: As shown in Theorem 3.2, ΔEnergy provably outperforms MCM by amplifying the separation between ID and open-set OOD data, i.e.,

Furthermore, ΔEnergy-based lower bound maximization (EBM) effectively increases the score for ID data, enhancing the separation between ID and open-set OOD samples. This improved separability facilitates more accurate OOD detection.

(b) For OOD generalization: As shown in Theorem 3.5, ΔEnergy-based lower bound maximization (EBM) leads to domain-consistent Hessians of the classification loss between original and masked samples, which is a strong indicator of OOD generalization. Proposition 3.6 shows that minimizing EBM provides a bound on the performance gap between ID data and closed-set OOD data, indicating that optimizing ΔEnergy implicitly promotes OOD generalization.

We sincerely appreciate your constructive feedback, which has greatly helped us strengthen our work. We will ensure that the revised manuscript fully addresses these points and are open to further suggestions.

We thank the reviewer for the constructive comments. Below, we provide detailed responses to all the questions raised:

R1. Reply to ''the performance under concept shift remains unclear'':

In our method, we primarily focus on enhancing the robustness of vision-language models under covariate shifts, while simultaneously enabling the detection of open-set semantic shifts. Specifically, we keep the class names consistent with those used during training in the test phase—a common and practical setting in real-world applications—and meanwhile aim to prevent the model from incorrectly assigning open-set semantic-shifted OOD samples to any ID class.

Following your valuable suggestion, we conduct experiments on the concept-shifted ImageNet-Superclass dataset [1], where each image is annotated with its corresponding superclass label. For the superclass labels, we use the open-source annotations provided in [1]. In Table 1-1, we demonstrate examples of ImageNet-1K training data alongside concept-shifted test samples. In Table 1-2, we report the performance of our ImageNet-1K-trained model on test sets annotated with superclasses. From Table 1-2, our EBM-based method outperforms baseline models under both covariate and concept shifts. These results demonstrate the effectiveness of our proposed approach in learning domain-invariant features, thereby enhancing the model's ability to generalize under various distribution shifts. All experimental results and related work [2, 3] on concept shifts will be incorporated into the later version of our manuscript to improve its overall quality.

[1] Soichiro Kumano, et al. Dataset Source: github.com/s-kumano/imagenet-superclass/blob/main/superclass_names.txt

[2] Xiao, Zehao, et al. "Any-shift prompting for generalization over distributions." CVPR, 2024.

[3] Santurkar, Shibani, et al. "Breeds: Benchmarks for subpopulation shift." arXiv preprint arXiv:2008.04859 (2020).

Table 1-1. Examples of the ImageNet-1K training data and the corresponding concept-shifted test data.

Table 1-2. Performance of ImageNet-1K-trained model on the test sets with covariate shifts and concept shifts.

R2. Reply to '' I am concerned that EBM may negatively impact classification performance on closed-set data. It is recommended to verify the fine-tuning accuracy of the proposed method on standard datasets used in CLIP'':

As defined in Equation 4, the proposed objective function—namely, the $\Delta\mathrm{Energy}$-based bound maximization (EBM)—aims to increase the lower bound of the $\Delta\mathrm{Energy}$ for closed-set classes, as demonstrated in Theorem 3.4. We have theoretically shown its impact on closed-set OOD data, particularly on data exhibiting covariate shifts, in Theorem 3.5. The results in Theorem 3.5 indicate that optimizing Equation 4 leads to domain-consistent Hessians of the classification loss, which implies improved model generalization on closed-set data [4, 5]. Empirically, this theoretical advantage is supported by the OOD accuracy gains observed in Tables 3 and 4 of our manuscript, as well as by the ablation study on EBM presented in Table 2-2 of our response to Reviewer 3pyx, further highlighting the effectiveness of EBM in enhancing OOD generalization.

Following your insightful suggestion, we evaluate the effect of the proposed EBM loss on fine-tuning accuracy across 11 standard datasets used in CLIP. We implement the EBM loss on PromptSRC [6] and train the models using 16-shot settings with a ViT-B/16 backbone. We train 50 epochs for Imagenet and 200 epochs for other datasets with SGD, following the same training setup as in [6]. The EBM hyperparameter was set to $p\\% = 0.6$, and we searched $\lambda_0$ over the values of $\\{0.1,0.5,1.0,2.0\\}$.

The performance results are reported in Table 1-3, demonstrating that the proposed EBM method further improves test accuracy on PromptSRC. Since the objective of EBM is to minimize $\mathcal{L}_{\Delta E}=E_2 - E_0$, it encourages the model to make equally high-confidence predictions for both the original and partially masked features, thus facilitating the learning of domain-invariant features between the original domain and the masked domain. The improvements observed in Table 1-3 empirically suggest that regularization on masked features effectively enhances the model’s generalization performance.

[4] Rame, Alexandre, et al. "Fishr: Invariant gradient variances for out-of-distribution generalization." ICML, 2022.

[5] Hemati, Sobhan, et al. "Understanding hessian alignment for domain generalization." ICCV, 2023.

[6] Khattak, Muhammad Uzair, et al. "Self-regulating prompts: Foundational model adaptation without forgetting." ICCV, 2023.

Table 1-3. Performances of fine-tuning accuracy on 11 standard datasets used in CLIP.

R3. Reply to ''Whether the proposed method can be applied to a broader range of OOD scenarios'':

We thank the reviewer for the insightful comment. Following your insightful suggestion, we have conducted additional experiments on concept shifts and evaluated model's fine-tuning accuracy on 11 standard datasets used in CLIP. The improvements observed in Tables 1-2 and Table 1-3 demonstrate that the proposed EBM loss, which is designed to facilitate the learning of domain-invariant features between the original and masked domains, can effectively enhance the model’s generalization performance. All experimental results reported in Tables 1-2 and 1-3, along with the related work on concept shifts [2, 3], will be incorporated into a later version of our manuscript to improve its overall quality.

We thank the reviewer for the positive comments and constructive suggestions. Below, we provide detailed responses to all the questions raised:

R1. Reply to ''Clarity of the Methodology'':

We apologize for the confusion and will revise the methodology section in a future version of our manuscript to improve readability. Following reviewers' constructive suggestions, we have provided detailed and intuitive explanations of the EBM loss and outlined the core ideas behind each theorem. A summary of the overall logic flow—illustrating the relationships among $\Delta\mathrm{Energy}$, the EBM loss, and domain-consistent Hessians—is presented in R2, with supporting empirical evidence shown in Table 2-1. Below, we briefly summarize the intuition behind each theorem:

Theorem 3.2 provides a formal guarantee that $\Delta\mathrm{Energy}$ can provably surpass MCM by amplifying the separation between ID and open-set OOD data.

Theorem 3.3 provides a formal guarantee that $\Delta\mathrm{Energy}$ can provably reduce the false positive rate (FPR) in open-set OOD detection compared to MCM.

Theorem 3.4 implies that minimizing the proposed EBM loss can increase the lower bound of $\Delta\mathrm{Energy}$ for closed-set classes, which effectively enhances the separation between ID and open-set OOD data.

Theorem 3.5 demonstrates that the EBM loss in Equation 4 can lead to domain-consistent Hessians of classification loss between original and masked samples—an indicator of improved OOD generalization.

Proposition 3.6 shows that minimizing EBM provides a bound on the performance gap between ID data and closed-set OOD data, indicating that optimizing $\Delta\mathrm{Energy}$ for ID data implicitly promotes OOD generalization.

All these intuitive explanations will be incorporated into the revised manuscript to enhance clarity and accessibility.

R2. Reply to ''Intuitive explanation of the EBM loss to guide the model to learn domain-consistent Hessians'':

As illustrated in the attention maps in Figure 1(B) and Figure 3 of our manuscript, retaining the top-$p$ proportion of elements in $P_j$ and masking the rest corresponds to masking environment-related information, after which we compute the resulting energy score, denoted as $E_2$. The EBM loss then measures the change between $E_2$ and the original energy $E_0$. The key insight is that masking environment-related information typically induces a small energy change, whereas masking domain-invariant information results in a large energy shift. By optimizing the EBM loss to minimize the energy change before and after masking, the model is in turn encouraged to focus on domain-invariant features, thereby promoting stable classification across domains. Thus, it encourages the Hessians of the classification loss to become domain-invariant as well.

To provide an intuitive understanding of EBM’s impact on learning domain-consistent Hessians and improving OOD generalization, we present the relationship between the EBM loss, Hessian distances between the original and masked domains, and OOD accuracy during the training process. We denote the Hessian of the classification loss with respect to the learnable model parameters for the original data as $H_\mathcal{s}$, and that for the masked data as $H_{\mathcal{s}^\prime}$. Following the prior work [1], we compute the Hessian distances between the original and masked domains as $\Vert H_\mathcal{s} − H_{\mathcal{s}^\prime}\Vert_F$, where $\Vert \cdot \Vert_F$ denotes the Frobenius norm. Table 2-1 reports these quantities throughout training on the ImageNet-based dataset. We observe that EBM consistently yields smaller Hessian distances and higher OOD accuracies, supporting our claim that EBM promotes domain-invariant optimization dynamics.

[1] Hemati, Sobhan, et al. "Understanding Hessian alignment for domain generalization." ICCV, 2023.

Table 2-1. Correlation between Hessian distances ($\Vert H_\mathcal{s} − H_{\mathcal{s}^\prime}\Vert_F$) and OOD accuracies / EBM losses under the EBM regularization during the training on the ImageNet-based dataset used in our main experiment. The OOD accuracies are evaluated on the closed-set subset of the ImageNet-R test data.

In addition, as shown in Theorem 3.5, we theoretically demonstrate that optimizing the EBM loss leads to domain-consistent Hessians. We briefly introduce the logic behind the theoretical relationships as follows: As shown in Equation 31, we establish a connection between the Hessians of the classification loss and the gradient of the energy score with respect to the model parameters. By comparing the gradients of the energy scores across the original and masked domains, we derive in Equation 33 that smaller differences in these gradients imply smaller differences in the Hessians of the classification loss across domains. Therefore, minimizing the EBM loss not only aligns energy score gradients but also encourages Hessian consistency, thereby improving OOD generalization. To make the logic flow clearer, we summarize the logic as follows:

A. Eq. (31): Classification loss Hessians ←→ Energy score gradients
↓
B. Domain-consistent Hessians ←→ Domain-consistent energy score gradients
↓
C. Optimizing EBM → Small distance between energy score gradients across original and masked domains
↓
D. Optimizing EBM → Small distance between Hessians across original and masked domains (i.e. domain consistent Hessians)

R3. Reply to ''Elaborate on the selection strategy for hyperparameters'':

In our main experiments on open-set discrimination using the large-scale ImageNet-1k dataset, we set $c = 2$ and search for the hyperparameters $p$ and $\lambda_0$ within the ranges $p\\% \in \\{0.4, 0.5, 0.6\\}$ and $\lambda_0 \in \\{0.09, 0.10, 0.11\\}$. Model selection follows the training-domain validation protocol, where models are trained using few-shot samples under different combinations of $p$ and $\lambda_0$, and the model that achieves the highest accuracy on 4-shot subsets—randomly sampled from the training data—is selected. In our manuscript, we have presented ablation study results for the hyperparameters $c$ and $p$ in Tables 9 and 10 of our manuscript, respectively. Here, we include the ablation results for $\lambda_0$, provided in Table 2-2. The results show that the EBM method achieves the best overall performance in terms of both AUROC and OOD accuracy when $\lambda_0 = 0.1$. Notably, models trained with EBM loss regularization (i.e., $\lambda_0 \neq 0$) achieve improved performance in both AUROC and OOD accuracy compared to the baseline model without EBM loss. This demonstrates the effectiveness of our EBM method in jointly optimizing for both OOD generalization and OOD detection.

Through empirical observations from the ablation study, we found that the masking proportion $p\\%$ exhibits a robust performance sweet spot within the range of 0.5 to 0.6, while the coefficient $\lambda_0$ performs best around 0.1. A masking proportion near 0.6 consistently balances the trade-off between retaining sufficient input information for accurate recognition and introducing meaningful corruption to enhance robustness. Similarly, setting $\lambda_0$ around 0.1 achieves a good balance between maintaining classification performance and enforcing energy consistency between the original and masked data. While these empirical observations provide a principled starting point for hyperparameter selection for other downstream tasks, we acknowledge that the optimal choices may vary slightly across different domains. Our ablation studies offer practical guidance for setting the search space of $p$ and $\lambda_0$ on other datasets.

Table 2-2. Ablations on the hyperparameter $\lambda_0$. In the table, we set $c=2$ and $p\\%=0.6$ according to the optimal hyperparameter observed in Table 9 and Table 10. The EBM method achieves the overall best performance on both AUROC and OOD accuracy when $\lambda_0=0.1$.