HYPO: Hyperspherical Out-Of-Distribution Generalization

We sincerely appreciate your comments and questions, which we address in detail below.

W1.1. Although Thm 5.1 provides insights into the upper bound of generalization variation, it does not conclusively demonstrate the superiority of the proposed method or loss.

We would like to clarify that our goal is not to theoretically establish HYPO's superiority over SupCon and PCL. Our primary contribution is to provide the first theoretical justification of prototype-based learning algorithms for OOD generalization. This is also acknowledged by Reviewer W1ps, who commented:

"...several learning methods have previously been proposed that utilize hyperspherical embeddings. But this paper is the first to provide a theoretical justification angled at OOD generalization."

Unlike PCL, our learning framework HYPO is motivated and guided by theory (Section 3.1), and provably guaranteed to reduce OOD generalization error (Thm 3.1 & Thm 5.1). Indeed, there may exist several alternative proxy-based learning algorithms such as PCL. However, it remains unclear if PCL's performance can be rigorously interpreted and bounded. We believe that our theoretical framework takes an important first step towards understanding domain generalization via hyperspherical learning formally, paving the way for more effective and theoretically grounded approaches in the field.

Empirically, unlike PCL which relies on SWAD optimization, HYPO is able to achieve strong performance by simple SGD optimization and the performance can be further improved with SWAD. Compared to PCL, HYPO achieves superior performance of an average accuracy of 89% on PACS, further demonstrating its effectiveness in practice.

W1.2. Thm 5.1 cannot be valid unless we explicitly specify the distribution distance $\rho$.

Indeed! We specified the distribution distance $\rho$ as shown in Definition 3.1 in the main manuscript. We have also added a footnote in the updated main manuscript for clarity.

W1.3. Thm 5.1 does not account for the influence of inter-class separation. Besides, in Ye et al (2021)'s Thm 4.1, function O(.) also depends on additional factors beyond just the variation.

We address the effect of inter-class separation in our theoretical analysis presented in Appendix J, which links to the second term in our loss eq. (5). Additionally, for the discussion on regularity conditions in Thm 4.1, please see our response to Reviewer W1ps.

W2. Clarification on the prototype update rule via EMA and the meaning of the second term in the loss.

The separation term's primary purpose is to maintain distinctiveness between class prototypes in the embedding space. While the separation term may not directly affect the gradients of $\mathbf{z}$, it indirectly influences how $\mathbf{z}$ is learned. By keeping the prototypes separate, the variation term has to adapt $\mathbf{z}$ in a way that aligns with these well-separated prototypes, leading to better OOD generalization. As detailed in Algorithm 1 in Appendix A, in this work, we choose to update the prototypes $\boldsymbol{\mu}_i$ via EMA, which leads to stable updates of prototypes. Alternatively, one can set class prototypes as learnable parameters and update prototypes via gradients. We compare the performance of our method with EMA update and learnable prototypes (LP) in Appendix H (P24). We observe our method with EMA achieves superior performance compared to LP, which empirically verifies the effectiveness of EMA-style update of prototypes in practice.

Thank you for your comprehensive and detailed rebuttal. I have carefully read through your responses to the comments and truly appreciate the time and effort you have put into addressing each concern. My further comments are listed as follows.

---

(1) This paper claims at the beginning of section 5 that

Our main Theorem 5.1 gives a provable understanding of how the learning objective effectively reduces the variation estimate.

However, Theorem 5.1 presupposes that the variation can be optimized to a small value under the proposed loss, specifically $\frac{1}{N}\sum_j\mu_{c(j)}^T z_j\ge 1-\varepsilon$. Notably, the paper lacks a clear explanation of how the learning objective in Eq (5) upper bounds $1-\frac{1}{N}\sum_j\mu_{c(j)}^T z_j$, which is crucial for justifying the claimed reduction in variation. Even for a special and simpler case where infinity samples are accessible $N\rightarrow\infty$, the theorem can still not tell why the learning objective is able to effectively reduce the variation estimate.

Therefore, the theoretical justification of prototype-based learning algorithms for OOD generalization is not sufficiently informative.

Moreover, it seems under this assumption, PCL and other methods can also offer the same theoretical insights.

---

(2) Why is there only a footnote directing to Appendix C for the theorem in the revised paper? Firstly, while Appendix C is extensive, it would be more helpful if the authors could provide a more specific reference within it for easier navigation. Additionally, why do the authors not directly include the distance $\rho$ used in Definition 3.1, as metentioned in the rebuttal?

Secondly, for the sake of rigor, it is essential for the theorem to be self-contained. Including specific details, such as the distance $\rho$ from Definition 3.1, within the theorem would enhance clarity and completeness.

---

(3) I appreciate the effort made to demonstrate that a simplex ETF can minimize the separation part of the proposed loss. However, my initial concern remains unanswered: how does the separation part of the proposed loss impact the OOD error?

---

(4) I cannot understand how the separation term can indirectly influence $\mathbf{z}$ if it does not directly affect the gradients of $\mathbf{z}$, as the authors mentioned in the rebuttal. A more detailed and mathematical explanation would be appreciated to clarify this aspect.

Moreover, my initial concern revolves around the potential redundancy of the separation term when EMA is employed. In the rebuttal, the experiments compare the loss with EMA and without EMA, which certainly introduces a distinction between the two settings. To address this concern directly, it would be beneficial to conduct experiments comparing the loss with the separation term and without the separation term specifically when employing EMA.

---

The manuscript would benefit from further refinement to fully meet the standards of ICLR. Thus, I believe this submission will be a good paper, but for now, it is not ready. I have decided to maintain my original score, and thank you for your effort once again.

Dear Authors,

I have reviewed your latest feedback, including the appendix, and I want to express my gratitude for your diligent response. My further comments are as follows.

(1) This paper highlights its theoretical significance as a main contribution (page 2, also the paper title), claiming to provide theoretical justification for how the proposed loss guarantees improved OOD generalization. Consequently, I expected a direct theorem demonstrating that the OOD error can be bounded by the proposed loss. However, this paper fails to provide this crucial argument, which largely limits its novelty.

(2) The rebuttal claims that minimizing the variation loss is "equivalent" to maximizing the negative first term of variation loss. This claim seems to lack sufficient mathematical rigor.

(3) In fact, the proposed loss comprises three components: L = first term of variation loss + second term of variation + separation loss. While minimizing the first term of the variation alone ensures that $\frac{1}{N}\sum_j\mu_{c(j)}^T z_j$ is close to 1, the presence of the last two parts tends to deviate it from 1. Therefore, the gap between $\frac{1}{N}\sum_j\mu_{c(j)}^T z_j$ and one should be the most important part of this paper and needs to be thoroughly investigated, instead of assuming its proximity to 1 directly (while this paper also claims that the introduction of separation loss is essential).

(4) Otherwise, if one uses only the first term of variation loss as the total loss, according to your theorem, the ood performance should also be good or even better (since smaller $\varepsilon$), which contradicts observed facts. This challenges the applicability of the theorem.

(5) Furthermore, based on your assumption, it appears that PCL and other methods could potentially provide similar theoretical insights, thereby largely diminishing the contribution of your approach compared to PCL.

(6) Similarly, since you claim that the proposed loss guarantees improved OOD generalization, you should point out how the separation loss affects the ood error in your theorem (the necessity of the separation loss). Otherwise, why can't one remove the separation loss? This concern does not appear to be adequately addressed in Appendix J.

(7) About EMA: The proposed loss can be written as $L(z,\mu)=variation(z,\mu)+separation(\mu)$. Since $\mu$ is updated by EMA, it is fixed when computating the gradients, i.e., $\nabla_z L(z,\mu)=\nabla_z variation(z,\mu)+\nabla_z separation(\mu)=\nabla_z variation(z,\mu)$. Then the encoder updates its weight based on $\nabla_z variation(z,\mu)$. After that, $\mu$ is updated by EMA, depending on the previous $\mu$ and current $z$, not depending on the loss. Therefore, the separation loss is not used in the training procedure.

---

In summary, this paper appears to overstate its theoretical contributions and demonstrates only marginal empirical improvements, possibly with technical flaws. Consequently, I am inclined to reject it.

We are grateful for your insightful comments and appreciation of our work. We address each question in detail and provide further clarifications below.

Q1. The motivation behind Equation 6

As you concur, we choose to update the class prototypes using exponential-moving-average (EMA) due to computational efficiency. Otherwise, computing the average over the entire dataset would have been very expensive. This practice has been commonly used in prior literature, see for example [3].

C1. Clarification on the significance of Thm 5.1

We would like to clarify that in Thm 5.1, the impact of sample-to-prototype alignment on the intra-class variation can be explicitly characterized via the first term $ε^{1\over 3}$ in the upper bound. In particular, $ε^{1\over 3}$ measures how sample embeddings are aligned with their class prototypes on the hyperspherical space (as we have $\frac{1}{N} \sum_{j=1}^N \boldsymbol{\mu}_{c(j)}^{\top} \mathbf{z}_j \geq 1-\epsilon$), which is optimized by our proposed loss. This Theorem implies that improved alignment (an illustration is shown in Figure 4(b)) leads to smaller upper bound of the intra-class variation $\mathcal{V}^{\text{sup}}$, a key term to upper bound the OOD generalization error by Thm 3.1. We provide empirical verifications below which further verify that the core assumption in our theoretical framework is satisfied in practice via our loss.

Q2 Empirical verification of Thm 5.1 and $\epsilon$ factor.

• Quantitative verification: Following the suggestion, we calculate the average intra-class variation over data from all environments $\frac{1}{N} \sum_{j=1}^N \boldsymbol{\mu}\_{c(j)}^{\top} \mathbf{z}\_j$ (Thm 5.1) models trained with HYPO. Then we obtain $\hat{\epsilon} := 1 - \frac{1}{N} \sum_{j=1}^N \boldsymbol{\mu}\_{c(j)}^{\top} \mathbf{z}\_j$. Due to the constraint of time, we evaluated on PACS, VLCS, and OfficeHome and summarize the results in the table below. We observe that training with HYPO significantly reduces the average intra-class variation, resulting in a small epsilon ($\hat{\epsilon} < 0.1$) in practice. This suggests that the first term $O(\epsilon^{1\over 3})$ in Thm 5.1 is indeed small for models trained with HYPO.

• Qualitative verification: We visualized the feature representations via UMAP in Figure 4. We can see that representations learned by HYPO become significantly more aligned with class prototypes across environments. This alignment further confirms the low variation in our learned representations, which is in close agreement with the regularity conditions on the density function required for the main theorem in Ye et al.

Q3. Ye et al. provide a specialized result for linear models (Theorem 4.2 in their work). I see this as a justification that the theoretical framework can be realized by some model...How do we know that the OOD generalization bounds can actually be computed for the choice of model used?

Great question! As specified in P9, we perform classification by identifying the closest class prototype: $\hat y = \text{argmax}_{c \in[C]} f_c$ where $f_c= \mathbf{z}^\top\boldsymbol{\mu}_c$ for embedding $\mathbf{z}$. This is equivalent to the linear top model as in Ye et al. (without the bias term), as we can reformulate as $f_c= (W\mathbf{z})_c$, where the $c$-th row of $W$ is $\boldsymbol{\mu}_c$. In this case, we have a linear convergence rate without the regularity conditions in Thm 4.1 (as shown in P7 in Ye et al.). In a more general case, the generalization bound in Thm 4.1 (pasted below for easy reference) depends on additional factors $\operatorname{err}(f) \leq O\left(s\left(\mathcal{V}\_\rho^{\text {sup }}\left(h, \mathcal{E}\_{\text {avail }}\right)\right)^{\frac{\alpha^2}{(\alpha+d)^2}}\right)$

such as $\alpha$ and $d$, where $d$ denotes the dimension of feature embeddings, and $\alpha$ is closely correlated to the density concentration. Such factors mainly affect the convergence rate of the error upper bound. The Theorem shows that, for any model, the generalization gap depends largely on the model’s variation captured by $\mathcal{V}^{\text {sup}}$. As HYPO aims to reduce the intra-class variation (verified in the previous response), it effectively reduces the OOD generalization bound.

Thank you for your detailed response.

I have taken the time to read through the discussions with other reviewers, in addition to your response here.

I appreciate the additional experiments that you've included. I feel that this helps to solidify the relevance of Theorem 5.1. The $\hat{\epsilon}$ values reported lead to an approximate additive error of $\epsilon^{1/3} \approx 0.4$ for the datasets explored. I'm having a hard time qualifying how large this value is relative to the loss bound (B) as I couldn't easily see how this boundedness assumption impacts the generalization error upper bound. It would also help to compare this value of $\epsilon$ achieved for other baseline methods --- to verify that an improvement is obtained from the proposed method.

I also share some of the concerns of kMBq, which I discussed in my original review: "I think it would be more valuable if one could show that the alignment assumption is satisfied by reducing the training loss directly, bringing the result more in line with typical PAC generalization bounds." I tend to agree with their final conclusion that this hasn't been adequately demonstrated; though the analysis you gave in response is a good start and I'd recommend that this be included.

Overall, I feel that the paper has been improved in rebuttal. The addition of new empirical results helps to support the claims of the paper. I believe that the theoretical contribution could be made more complete, but I think on balance there are some good contributions in this work. While I broadly agree with Reviewer kMBq, I have chosen to maintain my current score.

We are grateful for your insightful comments and appreciation of our work. We address each question in detail and provide further clarifications below.

Q1: How does Thm 5.1 connect back to the proposed loss function?

In Thm 5.1, we can see that the upper bound consists of three factors: (1) the optimization error $ε^{1\over 3}$, (2) the Rademacher complexity of the given neural network, and (3) the estimation error which goes down as the number of samples $N$ increases. While (2) and (3) are standard, the critical term, (1) $ε^{1\over 3}$, measures how sample embeddings are aligned with their class prototypes on the hyperspherical space (as we have $\frac{1}{N} \sum_{j=1}^N \boldsymbol{\mu}_{c(j)}^{\top} \mathbf{z}_j \geq 1-\epsilon$), which is optimized by our proposed loss.

Our Theorem implies that improved alignment (as illustrated in Figure 4(b)) leads to smaller upper bound of the intra-class variation $\mathcal{V}^{\text{sup}}$, a key term to upper bound the OOD generalization error by Thm 3.1.

Q2: How does our loss compare with more specialized algorithms on CIFAR10-C?

Thanks for the suggestion! Here we compare our loss (HYPO) with very recent algorithms: EQRM (NeurIPS'22) [3] and SharpDRO (SOTA from CVPR'23) [4], on the CIFAR10-C dataset (Gaussian noise). All three methods are trained on CIFAR-10 using ResNet-18, consistent with the architecture used in our main paper. The results are presented below.

Q3: A more in-depth comparison with the PCL loss

We provide a comparison in the paragraph Relation to PCL [1] on page 7. Here we would like to highlight notable distinctions and provide further clarifications:

• [Theoretical grounding] PCL offers no theoretical insights, while our loss HYPO is guided by theory. We provide a formal theoretical justification that our method reduces intra-class variation which is essential to bounding OOD generalization error (Section 5).
• [Statistical interpretation via vMF] While PCL and our loss share high-level intuitions, our loss formulation can be rigorously interpreted as shaping vMF distributions of hyperspherical embeddings (Section 4.2). In contrast, it remains unclear if PCL can be statistically interpreted.
• [Strong performance w/o specialized optimization technique SWAD] Unlike PCL which relies on SWAD [2], a dense and overfit-aware stochastic weight sampling strategy for OOD generalization, HYPO is able to achieve strong performance by simple SGD optimization (Avg Acc 88.0% without SWAD) and the performance can be further improved with SWAD.

Q4: Why the comparison of HYPO vs. PCL is in not included in Table 1?

Table 1 intends to contrast the effect of different loss functions under the standard SGD optimization. We exclude PCL from Table 1 since the performance reported in the original PCL paper (Table 3) is implicitly based on SWAD.

Due to the relevance of two losses and the choice of optimization strategies, we believe it is worthwhile to provide a detailed comparison of HYPO vs. PCL under four optimization strategies: PCL w/o SWAD, HYPO w/o SWAD, PCL w/ SWAD, and PCL w/ SWAD. Therefore, we summarize the results in a separate table (Table 2), as opposed to including them in Table 1 for better clarity. As shown in the table, compared to PCL, HYPO achieves superior performance with (Avg Acc 88.7% $\rightarrow$ 89%) and without SWAD (Avg Acc 86.3% $\rightarrow$ 88%), which further demonstrates its effectiveness and generality.

Table 2 in the main paper is pasted here for easy reference:

References

[1] Yao et al., PCL: Proxy-based Contrastive Learning for Domain Generalization, CVPR 2022.

[2] Cha et al., SWAD: Domain Generalization by Seeking Flat Minima, NeurIPS 2021.

[3] Eastwood et al., Probable domain generalization via quantile risk minimization. NeurIPS 2022.

[4] Huang et al., Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimization. CVPR 2023.

Q5: Typos

Thanks for pointing out! We have fixed this typo in the updated manuscript by adding a superscript that explicitly denotes the environment ($\mathbf{z}_i → \mathbf{z}_i^e$) to avoid potential misunderstanding.