Learning Structured Representations with Hyperbolic Embeddings

We thank the reviewer for their detailed comments, and address each of the questions and weaknesses below.

Concerns about the novelty

We humbly disagree with the reviewers’ statement since our learning setting is different from the existing works in the hyperbolic geometry literature, where our focus is on using hyperbolic geometry to embed a given class hierarchy. We believe our work would be a suitable contribution to the space of structured representation learning and Hyperbolic geometry, and we will make key points to argue the novelty of our work, which will hopefully highlight our technical contributions to the field.

1. Implicit vs explicit incorporation of hierarchy in Hyperbolic geometry and complementary nature of HypStructure: Most previous works in Hyperbolic geometry assume a latent implicit hierarchy in distributions/parameters, while our work focuses on an explicit knowledge injection of external hierarchy into the representation space. So while the methods and motivations may seem similar, there are important differences in the nature of the problem, and the statement that the “method proposed by the author has been explored in many previous works” is an overly simplified generalization.

Specifically, the “Hyperbolic Image Embeddings” paper replaces Euclidean layers with Hyperbolic classifiers which constrained the layer parameters in the Hyperbolic space, and has no similarities with HypStructure which instead utilizes external hierarchy for embedding regularization. Furthermore, the “Hyperbolic Contrastive Learning for Visual Representations Beyond Objects” proposes a Hyperbolic supervised contrastive loss implicitly assuming the existence of an unknown scene-object hierarchy.

However, we want to emphasize that our approach offers a flexible framework that can be complementary and used in conjunction with other Hyperbolic losses/backbones assuming implicit hierarchies for performance gains. For instance, with a more recent variant of Hyperbolic classifier-based backbone (Clipped HNN), as well the Hyperbolic SupCon loss for further performance gains - please refer to the detailed results in the global response above, however we share these performance numbers here for reference.

2. Address limitations of the l2-CPCC with strong empirical performance: The central theme of our paper is accurately embedding external knowledge available in the form of hierarchical information, in the representation space. While our work draws inspiration from recent contributions in Hyperbolic machine learning, we propose a novel, practical and effective framework that addresses the severe limitations of the CPCC objective in embedding this hierarchical information. This is clearly demonstrated with the strong empirical performance on large-scale datasets and visualization of learned representations using HypStructure.

3. Theoretical understanding of structured representation learning and OOD detection - to the best of our knowledge, our work is one of the first to draw theoretical connections between the paradigm of hierarchy-informed representation learning and tasks such as OOD detection, and we provide a provable hierarchical embedding that fills a theoretical gap in the broader field. These insights can serve to be useful in designing theoretically grounded, efficient and accurate representation learning methods that can learn general representations useful across tasks in practice as we have demonstrated for classification, OOD detection, and maintaining interpretable representations.

Based on these grounds, we feel our work makes many novel contributions in terms of explicit modeling of structured hierarchy, improving and preserving representations with other Hyperbolic objectives and backbones, empirical effectiveness across tasks, and drawing theoretical connections to the OOD detection task (as you have rightly noted). We also foresee our contribution as motivating more works in the practical domain of embedding structured information explicitly using Hyperbolic geometry. We sincerely hope this provides a satisfactory justification to the reviewer regarding our contribution.

---

Concerns about writing and examples

We have included these examples in the motivation section of our paper to emphasize the practical concerns and severe limitations of l2-CPCC, which serves as our motivation to develop better methods. These examples related to our label tree setting, are very informative for understanding the problem setup and the challenges (as has also been noted by Reviewer dBDC as a strength of our paper). Regarding writing, thanks for your suggestions, we will proofread to improve readability and remove repetitive expressions in the camera ready version.

We thank the reviewer for their detailed comments, and address each of the questions and weaknesses below.

Hyperbolic $L_{flat}$

We choose the Supervised Contrastive Loss (SupCon) loss as the $L_{flat}$ loss in our experiments primarily to provide a fair comparison with the prior works, whereas our HypStructure methodology is flexible and can be combined with any other loss function.

To demonstrate the wider applicability of our method based on the reviewers suggestion, we evaluate the performance using the Hyperbolic Supervised Contrastive loss $L_{\text{hyp}}$ proposed in [1] instead of the SupCon loss, and provide a comparison with using HypStructure in conjunction with this Hyperbolic loss in the Table below (fine acc.)

In addition to the aforementioned loss, we vary the backbone network - the Clipped HNN model, a type of Hyperbolic Neural Network, instead of using the Euclidean ResNet style architectures and provide the results for that setting as below.

We observe that even with a classification loss function that optimizes for separability in the Hyperbolic space, or a Hyperbolic Neural Network as the backbone, our proposed methodology HypStructure is complementary and provides performance gains for better embedding the hierarchical relationship in the Hyperbolic space. (More details regarding the setup are in the global response above.)

We will include the results with the experiments on hyperbolic losses and backbone in the revised version of the paper as well.

[1] Hyperbolic Contrastive Learning for Visual Representations Beyond Objects, Ge et. al, CVPR ‘23

---

Definition of dataset distance $d(D_i, D_j)$

As we mention in line 75-76, the dataset distance $d(D_i, D_j)$ can vary depending on the choice of the distance metric and the design setup, hence we avoid writing an explicit definition in Sec. 2.1 and provide a general expression here instead. We follow this up with an example of the specific dataset distance for the l2-CPCC case in lines 97-98, where we see that $d(D_i, D_j) = \rho_{l2} =$ the Euclidean distance of two averages of all feature vectors in each subset. On the other hand, for our proposed methodology HypStructure, you can either compute the metric $d(D_i, D_j)$ on Hyperbolic space, which first uses exponential map (Eq. 4) and then Klein average (Eq. 6) followed by the Poincare distance computation (line 180-181), or compute the means of the Euclidean features as in $\rho_{l2}$ and project the centroids to the Hyperbolic space using the exponential map, followed by the Poincare distance computation.

---

Figure for Example 1 and 2

The accompanying figures for both Example 1 and Example 2 are provided in Figure 1. More specifically, Example 1 only uses the tree nodes on the left part of Figure 1, which is what we refer to as the tree $\mathcal{T}$.

The Example 2 uses both the left part of the Figure to describe nodes in the tree, C, D and E, and the right part of the Figure to visualize this corresponding setup in an attempt to embed the tree in the Euclidean space, based on the arguments described in the text in Example 2. We will make this more explicit in the paper description and the caption in the figure.

---

Grammatical errors

Thanks for the suggestions, we will proofread to improve readability and remove any grammatical errors in the camera ready version.

---

Clarification for limitations

The aforementioned comments misunderstand the limitations of our proposed method and we would like to provide a few clarifications regarding the limitations. First, the $l_2$-CPCC methodology suffers from the challenges of embedding a tree exactly, owing to the representational capacity of the Euclidean space which has zero curvature. This leads to high distortion when embedding a tree-like hierarchy using the $l_2$-CPCC, as also shown in the distortion metrics in Table 1. In contrast, Hyperbolic spaces allow for embedding tree-like data in finite dimensions with minimal distortion owing to the negative curvature of the spaces, and hence our proposed methodology - HypStructure does not suffer from the same limitation as the $l_2$-CPCC, and therefore reduces the distortion in embedding the hierarchy (Table 1).

Additionally, while our submitted work experimented with primarily a Euclidean classification loss, (hence the corresponding statement in the limitations), over the rebuttal period based on the reviewers’ suggestions, we have experimented with other Hyperbolic losses and backbones, including the Hyperbolic SupCon loss and Clipped Hyperbolic Neural Network, which also shows improvements when combined with our proposed method, and hence our work is not limited to Euclidean classification losses. We will rephrase this statement to avoid further confusion. We foresee a potential limitation - since our proposed method relies on the availability (or construction) of an external hierarchy for computing the HypCPCC objective, it can be challenging if the hierarchy is unavailable or noisy. We will include this in the limitations section.

We thank the reviewer for their detailed comments, and address each of the questions and weaknesses below.

Boundary collapse

Part of the design of our HypStructure methodology, including the centering loss and embedding of internal nodes mitigates boundary collapse. Embedding the internal tree nodes in HypStructure - $T_{\text{int}}$ (as compared to only leaf nodes in prior work CPCC) and placing the root node at the center of the poincare disk with $l_{\text{center}}$ loss (discussed in lines 100-104, 186-194 in the main paper, lines 931-946 in the Appendix) embeds the hierarchy more accurately and the mitigate aforementioned issues. To demonstrate this intuition and evaluate its impact on the learnt hierarchy and representations, we first visualize the learnt representations from HypStructure, with and without these components - i.e. embedding internal nodes and a centering loss vs leaf only nodes, via UMAP in Figures 14 (CIFAR100) and 15 (ImageNet100) of the rebuttal PDF. We also provide a performance comparison (fine acc.) in the Table below

First, based on Figures 14 and 15, one can note that in the leaf-only setting without embedding internal nodes + centering loss (figures on the left), the samples belonging to the fine classes which share the same parent (same color) are in close proximity, reflecting the hierarchy, however they are close to the boundary. With the embedding of internal nodes and a centering loss (right), we note that the representations are spread between the center (root) to the boundary as well as across the poincare disk, which is more representative of the original hierarchy. This also leads to performance improvements as can be seen in the table above.

Note that we used the same $\beta = 0.01$ penalty parameter across datasets for convenience and easy reproducibility, and with a more careful tuning of $\beta$, the boundary collapse can be mitigated to a further extent. However, forcing the representations to be too close to the origin can lead to inter-class confusion and performance degradation as discussed in prior work [1] (Figure 4, page 7 of the paper).

[1] Hyperbolic Busemann Learning with Ideal Prototype, Atigh et. al, NeurIPS ‘21

---

Novelty

Thanks for your comments in support of the simplicity of our work. We would also like to refer the reviewer to our response regarding technical novelty in the global response above.

---

Euclidean linear evaluation

We provide Euclidean linear evaluation on the Euclidean features, as an added evaluation in the Appendix. Besides, it allows us to perform a fair comparison between important baselines, particularly l2-CPCC vs. HypStructure. To clarify, let us revisit our HypStructure pipeline: we first train an Euclidean loss SupCon which gives us an Euclidean feature (Fig. 6c), and then apply the exponential map to get the Poincare feature for CPCC computation. Because SupCon takes Euclidean inputs, it is reasonable to use Euclidean space for classification evaluation. From the visualizations in Figure 6b vs 6c, we note that the Hyperbolic regularizer empirically affects the geometry of the features even in the Euclidean space to be more structured and leads to performance improvements (Table 1, Fig. 7a, c)

On the other hand, Hyperbolic classifiers are applicable if we change the backbone model. We experiment with a Hyperbolic model, Clipped HNN [1] and present the fine acc. results below (more details about the setup in the global response above), evaluated with Hyperbolic Multinomial Logistic Regression Layer in [2]:

We observe that HypStructure provides performance improvements with Hyperbolic backbones as well.

[1] Clipped hyperbolic classifiers are super-hyperbolic classifiers, Guo. et al, CVPR ‘22

[2] Hyperbolic neural networks, Ganea et. al, NeurIPS ‘18

---

Further Experimentation

We would also like to point the reviewer to additional experiments that we have performed with non-euclidean setups and we discuss more details about them in the global response.

---

Broader impact and limitations

Thanks for pointing this out, we would like to clarify that we experiment with the Euclidean loss primarily to provide a fair comparison with the prior works, and it is not a requirement (or limitation) for our HypStructure methodology. It can be combined with any other loss function (such as HypSupCon) or backbone (Clipped HNN) as we have demonstrated above, highlighting the wide applicability of our method.

In addition to the discussion in Section 7, we enumerate other limitations which we foresee for future research and building on our methodology - our proposed method relies on the availability (or construction) of an external hierarchy for computing the HypCPCC objective, which might be challenging if the hierarchy is unavailable or noisy. In terms of Broader Impact, one potential area of impact that we recognize is the AI for science domain, where HypStructure can be helpful in learning more interpretable representations which are more reflective of the underlying relationships in the domain space. We include this in the revised manuscript.

We thank the reviewer for detailed comments, and address each of the questions and weaknesses below.

Contribution

While our work builds on prior research, this does not diminish the contribution of our work and we request the reviewer to kindly refer to our note on novelty in the global response, we clearly outline the key differences between our work and prior works.

---

Reference to Hyperbolic ops.

Thanks for pointing this out, we will include references to common Hyperbolic operations in the revised manuscript [1]

[1] Hyperbolic trigonometry and its applications in the Poincare ball model of Hyperbolic geometry, Ungar A., Computers & Mathematics with Applications, 2001

---

Comparison to [r2]

Thanks for pointing out this work, indeed [r2] is a related work, we will cite it in our revised manuscript and have attempted to provide a comparison below. However, first we discuss the key differences between [r2] and our work.

1. The usage of hierarchy and “prototypes” differ significantly. [r2] is a two-step method, where first an action word Hyperbolic embedding is created based on action hierarchies (each word is a single tree node), and second a video embedding is learnt using positions of action words (called prototypes in [r2]) in the Hyperbolic space. In contrast, our method learns the hierarchical image embeddings end-to-end, does not depend on word embeddings, and performs a centroid computation on the label hierarchy tree nodes (many images belong to a single label node, hence class prototypes are computed by averaging).

2. The nature of components to learn hierarchical embeddings are different. In [r2], both losses $L_H$ (Eq. 2 in [r2], as in [2]) and $L_2$ (Eq. 6 in [r2], as in [3]) are ranking losses, whereas HypCPCC factors in absolute values of the node-to-node distances. The learned hierarchy with HypCPCC will not only (implicitly) encode the correct parent-child relation like [2, 3], but also learn more complex and weighted hierarchical relationships more accurately (Fig. 16 in the rebuttal PDF).

We attempt to compare HypStructure and [r2] now. Since losses $L_H$ and $L_2$ have similar motivation as HypStructure, we add them as regularizers to the Hyperbolic classification backbone Clipped HNN and compare with our method:

We see HypStructure achieves better performance than $L_H$. We face NaN issues while training with L_2 and other losses, and discuss the setup details in the global response.

[2] Poincaré embeddings for learning hierarchical representations, Nickel et. al, NIPS ‘17

[3] Hyperbolic entailment cones for learning hierarchical embeddings, Ganea et. al, ICML ‘18

---

Rationale behind Klein avg.

We experiment with both HypCPCC variants mentioned in line 177-185. The 2nd variant, however, given our understanding, is not a special case of [r2], since it still requires a Euclidean averaging step for class prototype computation. We empirically observe performance improvements in accuracy using the 1st variant across datasets below (fine acc.).

---

Theorem 5.1 entry of K

Since $K$ is $\mathbb{R}^{n\times n}$ (line 289), an entry of $K$ is - $K_{i,j}$, the $i$-th row and $j$-th column scalar value of K. Each entry of $K$ can be denoted as $r^h$, where $r^h$ is the lowest common ancestor (LCA) of two samples represented by the $i$-th row and $j$-th column respectively so it is indeed a scalar. We also refer the reviewer to the matrix in lines 714-715, where each entry of $K$, denoted as $r^h_{i,j}$, is a scalar value. (i,j subscript can be ignored, see 718-719, since every two leaves sharing the LCA of the same height have the same tree distance). We will make this more explicit in the revised version to avoid confusion.

---

“The proof used $|𝑢|=|𝑣|=1−𝜖$…”

Thanks for pointing out this typo due to our cluttered notation! We intended to mean $|u|^2 = 1 - \text{(a small number)}$, however the conclusion of the proof still holds with this change, as the reviewer noted. Let us rewrite the proof: in practice with $clip^1$ projection, to be consistent with the notation in the main body, we set $|u|=|v| = 1 - \epsilon$ where $\epsilon$ is a very small number. Then $|u|^2 = (1 - \epsilon)^2 = 1 - 2\epsilon + \epsilon^2$. Since $\epsilon$ is very small, we define $2\epsilon - \epsilon^2$ as $\xi$, making $|u|^2 := 1 - \xi$ where $\xi$ is still a small number. Then replacing all $\epsilon$ with $\xi$ in line 708-709, we get the same conclusion. We will correct this error in the revised version.

---

“Incorrect proof in Corollary A.1...”

Apologies for the confusion. With the statement “growing in the same trend”, we meant that Poincaré(u,v) is monotonically increasing with Euclidean(u,v), and the proof after Corollary A.1 will not depend on the metric, as can be seen from comments in 710. This is true since with $\text{clip}^1$ transformation, Poincaré is approximately the log-scalar transform of Euclidean distance. The monotonically increasing property ensures the relative order of any two entries for Euclidean CPCC and Poincare CPCC matrices in K to be the same. Then, we can argue about the structure of K, either Euclidean or Poincare, to have the hierarchical diagonalized structure as in Fig 8a. But indeed technically, due to the logarithm, this is not “proportional to”. We will remove this notation and make it more clear.

---

Limitations

Another potential limitation - our method relies on the availability (or construction) of an external hierarchy for computing the HypCPCC objective, which might be challenging if the hierarchy is unavailable or noisy. We will include this in the next iteration of the paper.

We are now sharing a more exhaustive set of results with the hyperbolic settings with more evaluation metrics used in our paper, with the results below (averaged over 3 seeds). We also perform the t-test for experiments with vs. without HypStructure, and highlighted the higher performance number only if the p-value is smaller than 0.05, for statistical significance

Experiments with the Hyperbolic Supervised Contrastive Loss

Fine accuracies, coarse accuracies, and distortion measurements with the L_hyp loss .

Comparative Evaluation of OOD detection performance on a suite of OOD datasets with different ID datasets. OOD Detection AUROC Results on CIFAR10 (ID)

OOD Detection AUROC Results on CIFAR100 (ID)

OOD Detection AUROC Results on ImageNet100 (ID)

Experiments using the Clipped Hyperbolic Neural Network Backbone

Fine accuracies, coarse accuracies, and distortion measurements with the Clipped HNN backbone

Comparative Evaluation of OOD detection performance on a suite of OOD datasets with different ID datasets. OOD Detection AUROC Results on CIFAR10 (ID)

OOD Detection AUROC Results on CIFAR100 (ID)

Summary: Based on the above results over multiple seeds, we can clearly observe that our proposed method leads to a statistically significant and consistent improvement in performance in terms of classification accuracies with a significantly more evident reduction in other metrics, including distortion of the representations, and a higher improvement on the mean AUROC in the OOD detection tasks, with improvements upto 3% across a suite of OOD datasets.