Balanced Hyperbolic Embeddings Are Natural Out-of-Distribution Detectors

We thank the reviewer for their feedback. Below, we have addressed the reviewer's comments regarding difference over existing works and additional analyses. We have also gone over all individual questions. We believe that the review and rebuttal strengthen the paper and we hope to have answered all questions adequately.

Novelty over existing works. Indeed, [1] and [2] have previously highlighted the potential of hyperbolic or hierarchical embeddings for out-of-distribution detection. We take inspiration form such works and make following novel contributions:

• The proposed balanced hyperbolic embeddings approximate tree distances more accurately than existing optimization approaches and can also serve as a standalone embedding tool.
• We introduce generalizations of existing OOD detection scores to their hyperbolic variants and provide in-depth analyses to showcase its effectiveness in OOD detection across datasets.

We have added clarifications in the paper to highlight these key distinctions and novel contributions over existing works.

Changes in section 5. For better clarity, we added a short introduction in section 5 and updated the headings to be more descriptive.

Motivation for losses and how norm loss affects OOD detection. We aim to learn an embedding with minimal distortion to preserve the hierarchical information in the labels. This is achieved by jointly optimizing the distortion (Equation 7) and norm loss (Equation 9). Optimizing Equation (7) directly minimizes distortion. The reviewer is right in pointing out that in we should try to push all in-distribution samples away from the origin. The ID samples are indeed away from the origin when the distance to leaf nodes (corresponding to the classes) from the balanced hyperbolic embeddings is minimized. Consequently, it means we need norm loss in Equation (9) ensures that all nodes belonging to the same granularity of a hierarchy are equidistant from the origin.

Comparison to hyperbolic methods. We added Poincare embeddings and HEC for completeness, and didn't expect them to be better in classification performance, as also explained in lines 347-377. The AUROC and AUPR performance of Poincaré embeddings (PE) highlights the strong connection between hyperbolic learning and effective OOD detection. However, a downside of PE is its low classification performance. In contrast, our method achieves strong performance in both classification and OOD detection. We add suggested comparisons to other hyperbolic approaches below.

k = 300 in Table 4. We choose k=300 as to compare to ablations from CIDER [Ming et.al., 2023] and PALM [Lu et.al., 2024] [2]. As suggested, we also show below the results for k=200.

Eq (11) and comparing Euclidean vs Hyperbolic logits. Equation (11) ensures that embeddings of images are optimized to be closer to their class prototypes, $P_\mathbb{B}$. This results in tighter, more compact ID clusters. Both Euclidean and hyperbolic logits perform equally well in classifying ID samples, as shown in Fig. 1. However, hyperbolic logits are particularly advantageous for identifying OOD samples due to the exponential nature of distances in hyperbolic geometry. A key distinction of our method compared to hyperbolic OOD scoring approaches such as [1], which use distance hyperplane boundaries, or [Khrulkov et.al., 2020], which rely on sample norms to distinguish ID from OOD, is that our method differentiates samples based on their proximity to the prototypes. While [1] and [Khrulkov et.al., 2020] might misclassify OOD samples with high norms as ID, our approach leverages more nuanced information for better discrimination.

Line 157-159; Regarding [2] and CPCC loss. [2] and our method both enforce the hierarchical knowledge of the tree into the feature space but in different ways. [2] enforces hierarchy during training by using CPCC loss as a regularizer to learn tree distances between class means. In contrast, our method first explicitly embeds the label hierarchy into hyperbolic space. Using the leaf nodes corresponding to classes as prototypes, we then train the model on the dataset. We add the discussion to related work section.

Novelty over existing works

Although this might be somewhat subjective, I still want to give my own reason for why the novelty of this work is relatively limited. Let me copy the authors' response for the novelty statement in the rebuttal and elaborate:

“The proposed balanced hyperbolic embeddings approximate tree distances more accurately than existing optimization approaches and can also serve as a standalone embedding tool.”

After you clarified your distortion metrics for evaluation and training, this novelty argument is not very strong. First, your distortion metric is borrowed from [Sala et.al., 2018], which is a recent work under a similar context. [Sala et.al., 2018] proposed this metric under the context of learning hyperbolic embedding, and authors directly use almost the same formulation as their training objectives. Besides, when you say “approximate tree distances more accurately,” I do not think this is a fair comparison with other embedding methods. If you directly optimize the distortion metric, as long as the optimizer works well, it is hard to achieve anything better than direct optimization.

“We introduce generalizations of existing OOD detection scores to their hyperbolic variants and provide in-depth analyses to showcase its effectiveness in OOD detection across datasets.”

Generalizations of existing OOD scores are a very simple extension if you learn embeddings in hyperbolic spaces. However, I am not sure if this amount of extension is enough novelty for ICLR. For the in-depth analysis, I agree authors provide extensive experiments, but the authors mainly focus on describing the numbers in the tables. Since authors include many metrics, datasets, baselines, and settings in the experiment section, it is important to carefully interpret both performance gain and performance loss/outliers for some set of experiments, so researchers can use your work more effectively in the future.

Hierarchy for OOD detection?

The paper claimed that OOD detection will benefit from both hyperbolic embeddings and hierarchical information. I don’t have too many concerns about the hyperbolic embeddings, but for the hierarchical information, first, it relies on the quality of hierarchy as you mentioned in the limitation. However, in practice in a more general OOD setting, such kind of ground truth tree information is usually unavailable, so I cannot verify if your method will generalize well other than CIFAR100, ImageNet, etc. Second, I would imagine, even if you have access to the ground truth hierarchy, it may hurt OOD detection performance when the inD and OOD are too near. If they share some parts of the ground truth hierarchy, introducing tree information can cause confusion in the OOD detection. This problem becomes even more challenging when we do not have access to the ground truth tree for the datasets. So the benefit of using hierarchy for OOD detection, although mentioned in some previous literature, is not very strong either.

Furthermore, in the current submission, it is hard to see how much the OOD gain is from hierarchical information and how much is from hyperbolic embeddings. If the Euclidean baseline is just standard ResNet trained with cross-entropy, this baseline seems to be too weak for comparison. It is more reasonable to use the Euclidean version of your distortion loss (by replacing $d_\mathbb{B}$ with Euclidean distances) + norm loss for OOD detection, to prove the advantage of hyperbolic embeddings. And this will also help you show how much gain you get from including hierarchical information by comparing Euclidean hierarchical embedding with Euclidean + cross entropy embedding. If you are worried that Euclidean space cannot embed the tree very well, then another necessary baseline is just training with hyperbolic cross-entropy in Equation 11, and use this to compare with your final method. Some of the results are now included in Figure 2, but many OOD metrics are still missing.

If I have more questions about some details of the paper, I will add them later and ask for explanations if necessary. But personally, these concerns mentioned above are more fundamental weaknesses of this work, so I want to summarize them here and I am open to further discussion. Besides, if the authors feel some mentioned experiments require a huge amount of effort and find it challenging to finish all during the remaining rebuttal time, feel free to point them out.

Thanks for your reply.

Norm Loss Again your mathematical expression of norm loss cannot guarantee the position of inD data and OOD data for their distances to the origin. Therefore, I still do not understand why it can work as now it works magically. I believe my counterexamples provided before, no matter where my vertex is with respect to the origin, as long as all vertices are at the same depth/level, my norm loss will be small. It can be contradictory to the hyperbolic embeddings for OOD detection’s major advantage. Even if all inD nodes are leaf nodes, again, their distances to the root node/level is based on the ground truth tree, but they can be either large or just 1.

Besides, I don’t think all leaf nodes’ distance to root is the same for ImageNet100 is the same, based your figure 7 in the Appendix. If they are indeed placed equidistant to the origin, probably some of your implementation is wrong.

Novelty Because you claim your contribution to the combination of distortion and norm loss, while the norm loss’s motivation is still questionable, this novelty statement is not very strong in my point of view. This is the reason why I keep asking clarifying questions for the norm loss.

Hierarchy for OOD detection For the nearOOD performance on CIFAR-OSR split, the same concern for the weak baseline applies, and you need to provide more comprehensive analysis (with several controlled splits sharing the same hierarchy, stronger baselines, more inD datasets, more OOD metrics etc.) to illustrate the point that your method even works for nearOOD methods. I didn’t expect the authors to address them in the rebuttal time but this will be a good thing to add.

We thank the reviewer for their detailed feedback, we address the question below and fix minor mistakes in the paper directly and update Figure 1 to be bigger.

Learning a well defined hierarchy. We agree that distortion loss helps enforce the hierarchical structure. The reviewer is also correct in noting that the combination of distortion and norm loss helps prevent embeddings from collapsing. Measuring distortion of the learned graph (ex., distortion in Table 3, 7) shows that the hierarchical distances are preserved in the hyperbolic embeddings. To Appendix B, we add a plot of pairwise distances of each nodes which confirms that the hierarchy is indeed preserved. Additionally, we add a plot average norm for all the levels of hierarchy for CIFAR-100 and ImageNet-100.

Assumption of a known and correct hierarchy. We agree that the method assumes access to a known and correct hierarchy. Updated the limitations to include this.

Distance of OOD. The observations made by the reviewer in Figure 4 are correct, we add two clarifications:

• On average OOD samples are closer to origin. As suggested, a plot showing the distribution of norms is added to the Appendix C.
• OOD samples can have a high norm without necessarily being close to the prototypes.

If OOD is highly related to the semantic concepts. Using hierarchical prototypes makes it easier to handle OOD examples that are closely related to in-distribution data, as shown in hierarchical ablations (CIFAR-OOD-Split). It also leaves room to build on existing hierarchical datasets from other research or come up with new methods, making it a flexible and practical approach for future improvements.

Generalization to other curvatures. The method can generalize well to curvatures which can be dataset dependant. We keep the curvature fixed to -1 for our main experiments. We will add an ablation of curvature to Appendix C.

We thank the reviewer for suggesting related works and experiments, as well as visualizations to enhance the paper. We answer the questions below and correct minor mistakes directly in the paper.

On bias towards deeper and wider subtrees. The loss function in Equation (7) leads to embeddings for which the hyperbolic distances between the embedded vertices closely resemble the graph distances. Hierarchy imbalance is caused by the uneven distribution of labeled nodes which can cause uneven norms corresponding to the imbalance in works that don't explcitly correct for it [Nickel et.al, 2017, Ganea et.al. 2018]. We generate synthetic hierarchies for essential verification and compare with the hierarchies generated by previous works. We will add the visualization to Appendix A.

ID performance. The reported accuracy for our hyperbolic model is comparable to baseline CE models, while offering significant advantages in OOD detection. When training CIFAR-100 under a setting similar to CIDER (75.35), our method achieves nearly identical ID performance (75.24).

Possible scale match issue in loss function. The division by $d_G$ in Equation (7) actually prevents any distant pairs contributing large values. The non-linear growth of $d_\mathbb{B}$ is with respect to the Euclidean representation that is being used to model hyperbolic space, i.e. the Poincaré ball, which only influences the embedding through possible numerical errors.

Connection to previous papers. We agree and have added a reference to CIDER in the paper accordingly. In CIDER, OOD samples lie only between ID clusters because the embeddings are normalized onto a hypersphere. In contrast, in our method, OOD samples lie both between ID clusters, and between ID clusters and the origin, as hyperbolic space allows embeddings to exist anywhere within the space.

More details of experiment setup. Loss function for baseline experiments is cross-entropy loss, experimental setup is similar to OpenOOD benchmark. L250-253 is updated for clarity. L310-315, the models differ in how logits are computed but the loss is still cross entropy loss (Eq 11), so training hyperparameters remain the same. Distortion metric is defined in [Sala et.al., 2018] as referenced in Table 7 and now updated for Table 3 accordingly. We update necessary details in the paper and in Apendix C.

Using SupCon loss instead of CE loss. Methods using SupCon loss evaluate OOD using only distance-based functions (e.g., Mahalanobis score in SSD+ and KNN score in KNN+). In contrast, our method generalizes to both distance-based and logit-based scoring functions. Therefore, the baseline Euclidean setting uses CE loss to enable comparison across the scoring functions. For completeness, we have now included comparisons to SSD+ and KNN+ (in the context of table 4) to Appendix C.

How does norm loss affect the learning? By adding Norm loss, all ID prototypes in the label hierarchy are positioned at the same distance from the origin in hyperbolic space, even for imbalanced hierarchies. These prototypes are then fixed for the next stage. When training a ResNet-34, this ensures that all ID embeddings are farther from the origin, allowing OOD samples to map closer to the origin or between the prototypes.

Can the method generalize to other hyperbolic models? We first randomly initialize the model and learn Poincaré embeddings for 100 epochs, after which we continue training with our distortion loss. We now clarified this in the setup as well. Since a distance function is well-defined for various models of hyperbolic geometry, making it possible to generalize this formulation across different representations of hyperbolic space. Alternatively, prototypes can be learned in the Poincaré ball and mapped isometrically to other models.

Visualizing learnt hierarchies. Measuring distortion of the graph (ex. in Table 3 and 7) shows that the hierarchical distances are preserved in the hyperbolic embeddings. In the Appendix B, we also visualize the pairwise distances of each nodes in the hierarchy to show the hierarchical structure that emerges.

We thank the reviewer for their positive feedback and clarify our motivation below.

Motivation of hierarchical embeddings for Hyperbolic OOD. Previous research has demonstrated that hyperbolic spaces are highly effective for OOD detection, as briefly shown in Fig. 1 and further explored in the ablation study in Fig. 2 (non-hierarchical hyperbolic). This establishes a strong foundation for our approach to OOD detection using hyperbolic spaces. This forms a strong basis for our approach to OOD detection using hyperbolic spaces. Using hierarchical relationships for OOD adds two benefits:

1. Incorporating hierarchical relationships introduces meaningful structure to the embedding space, where the distances between class prototypes reflect their hierarchical relationships. This makes it easier for OOD detection methods to differentiate between samples from closely related classes and those that are genuinely out-of-distribution, particularly in challenging scenarios where OOD samples resemble in-distribution (ID) classes.
2. Learning with hierarchical prototypes enables our method to generalize to both distance-based and logit based scoring functions.

Adding recent related works. The paper is updated to include citations to the related works, thanks for pointing them out.

We again thank the reviewers for their thoughtful feedback, which has helped us further refine the paper. We have incorporated your suggestions and updated the PDF, with all changes highlighted in blue. Below is a summary of the revisions:

1. Improved Clarity and Readability: We refined the main text to enhance its clarity, readability, and completeness. Specifically, we increased the size of Figure 1, clarified equivalent edge distances in line 149, improved the description of Algorithm 1, added a citation for distortion in Table 3, and expanded the discussion on scoring function comparisons in lines 233–255. Additionally, we ensured consistent decimal formatting across all tables.
2. Structural Improvements: To improve structure and presentation, we added a brief overview paragraph before discussing the experiments in Section 5 and revised the headings for the hyperbolic comparisons and hierarchical ablations. We expanded the details of the hierarchical generalization in Section 5, with further elaboration provided in Appendix C. To accommodate these updates, we moved portions of the Related Work section to Appendix D.
3. Ablations and Analyses: We included multiple new ablations and analyses in Appendix C to strengthen our findings. These additions include expanded results on ImageNet-100, dataset-wise results on CIFAR-100, and ablations focusing on AUPR, AUROC (Figure 2), and curvature.
4. Additional Discussions and Visualizations: : Appendix A includes a discussion of biases toward deeper and wider subtrees. Appendix B provides visualizations of the learned hierarchies. Appendix C now includes detailed experimental setups for the Euclidean baseline and analyses on ID/OOD embedding norms. We also added further details on Figure 6 to clarify the motivation for our proposed losses.

We hope these revisions address the feedback and further enhance the quality and clarity of the paper. We welcome any additional questions or suggestions.