Hyperbolic Visual-Semantic Alignment for Structural Visual Recognition

We thank the reviewer for the feedback and suggestions, and appreciate the positive comments. We address your comments below, and start with the following question:

1. The definition of $q_{\phi}$ in Eq 7 and 8 are missing.

Sorry about this. Here we follow the conventional notation in VAE [ref1]: $q_{\phi}(z|x)$ represents the variational posterior distribution that is an approximation to the intractable true posterior $p_\theta(z|x)$. It typically refers to a probabilistic encoder, which produces a distribution (in our paper is a Gaussian) of latent representation $z$ given a feature $x$. Relevant parts have been carefully revised to reflect this.

[ref1] Auto-Encoding Variational Bayes. ICLR 2014

2. The training and inference computation efficiency is not presented.

To address your concern, we provide comparisons of training/inference cost in below table on Cityscapes val. The table has been incorporated into Appendix Table 11.

As seen, our model basically exhibits a similar level of training and inference efficiency as the other two methods. Though it has more parameters, a larger FLOPs, and runs slower (smaller FPS), the gap is minor. Considering the superior performance and generalization capability of our model, the additional cost is acceptable.

3. Minor issue: 3.1 Typo: Section 3.2, Task Setting: “An undirected edge $(v_i, v_j) \in E$ indicates that the class $i$ is a superclass of label $i$” It should be “label $j$” instead of “label $i$”. 3.2 Typo: Section 3.2.1 Hyperbolic Gaussian Mixture VAE: “we seek to align visual embedding with visual embedding to yield a shared representation space.” Is it “align visual embedding with LABEL embedding”?

Thank you so much for your careful review! They are fixed and a thorough proofreading is made.

“An undirected edge $(v_i, v_j) \in E$ indicates that the class $i$ is a superclass of label $i$” $\rightarrow$ “An undirected edge $(v_i, v_j) \in E$ indicates that the class $j$ is a superclass of label $i$”

“we seek to align visual embedding with visual embedding to yield a shared representation space.” $\rightarrow$ “we seek to align visual embedding with label embedding to yield a shared representation space.”

Are the authors planning to release the source code for this work?

Definitely! We will release the code and model weights to the public.

We appreciate the reviewer for pointing out certain issues of clarity in the technical part which we have revised in the latest revision and provide clarifications below.

To begin with, we address the following question that is relevant to our motivation:

What is the aim of hierarchy-agnostic alignment when we already have the hierarchy information? Why is the ``agnostic'' part important?

Sorry for the confusion. Please bear with our clarifications below. Our main idea is to learn a shared, hierarchical, multimodal feature space. Our model achieves this based on two major insights: (1) each visual entity (e.g., image, pixel) can be represented through a composition of multiple semantic labels, and (2) these labels are correlated in nature, collectively forming a tree-shaped hierarchy. Based on these insights, we develop two components: hierarchy-agnostic visual-semantic alignment addresses (1), and hierarchy-aware semantic learning addresses (2).

Difference and relation between the two components: the hierarchy-agnostic component handles multimodal alignment, but ignores hierarchical information; the hierarchy-aware component only interprets the hierarchical structure of semantic concepts (visual information is not taken into account). These two objectives operate on different granularity and synergistically contribute visual-semantic structural alignment. Notably, direct aligning the hierarchical structures between two modalities is very challenging, and our decoupled design alleviates this and helps model converge.

Next, we address your comments on paper writing:

The logic flow is broken, one can hardly see the connection between 3.1, 3.2. It will be very difficult (even for a hyperbolic learning researcher) to reproduce the method by reading the ``Method'' section.

Our apologies. We agree that the writing of mentioned parts can be improved upon rereading. We have worked to improve our writing.

• connection between 3.1 and 3.2: Sec. 3.1 is only to present the basic formulations for hyperbolic learning. It helps readers to understand basic concepts, and the formulations are also reused in Sec. 3.2 (e.g., Eq. 5, Eq. 6, Eq. 8). There are no inherent connections between them. To avoid confusion, we take an action to move Sec. 3.1 into a new Sec. 3 and the methodology part into a new Sec. 4. Thanks.

• writing improvements in the methodology part:

  (1) in Sec. 4.1, we have added missed details, including an explicit definition of covariance, i.e., $\Sigma_l=diag(\sigma_l^2)$, as well as the definition of the reconstruction loss.

  (2) in Sec. 4.2, we re-written most parts, and make the section more logical. Concretely, we compile hierarchy-aware semantic learning into two major parts, one is hyperbolic metric learning, and the other is how to construct samples for metric learning.

We believe that these improvements make the methodology more logical and easier to understand. Beyond this, we promise that we will release the code to help reproducibility. Thanks.

a) " ... the mean of the wrapped normal mixture distribution is calculated by Möbius addition. " is not logically correct.

We agree with the reviewer: this statement is not reasonable and is redundant; we removed it in the revised version.

b) " ... resulting in $y_{l,i} = \mu_l + \epsilon\Sigma_l \in \mathcal{Y}$...", you cannot add a mean vector to a covariance matrix.

Thank you for your careful review. In the original writing, we accidentally omitted the explicit definition of $\Sigma_l$, which is a diagonal matrix: $\Sigma_l=diag(\sigma_l^2)$. Here $\mu_l\in\mathbb{R}^d$ and $\sigma_l\in\mathbb{R}^d$ are derived from the semantic encoder. With this respect, the formula of the reparameterization trick should be more strictly revised to $y_{l,i} = {\mu}_l + \epsilon\sigma_l$. The relevant parts have been carefully revised.

c) The reconstruction loss is neither explained nor linked to a reference

The reconstruction loss is a standard negative log-likelihood. We provide its formal definition in Eq. 8 in the updated article. Thanks.

The tables basically show "we have better numbers", but the analysis lacks in-depth understanding of why each part works.

Thanks for pointing this out. Following your comments, we have carefully improved the experimental parts to offer more in-depth discussions on the results. Please refer to Sec. 5.2 and Sec. 5.3 for details.

• In response to For eq. (9) to eq. (11) are almost the same as in (Ganea et al., 2018), true, we use a similar loss function as in (Ganea et al., 2018), but our approach differs from it in two major aspects ...

In the submission, eq. (9) is in essence the same as eq. (32) in (Ganea et al.. 2018), eq. (10) is in essence the same as eq. (33) in (Ganea et al., 2018). The only notation difference is that

1. The authors use $(z_a, z_p) \in P$ for positive pairs and $(z_a, z_n)\in N$ for negative pairs
2. (Ganea et al., 2018) uses $(u, v) \in P$ for positive pairs and $(u', v') \in N$ for negative pairs.

In the submission, eq. (11) is in EXACTLY the same as eq.(28) in (Ganea et al., 2018).

However, in (Ganea et al., 2018), eq. (28) and eq. (33) is used for defining the cone, which is the core contribution, such that the word "cone" is in the title of (Ganea et al., 2018).

In this submission, those cone definitions are neither related to "visual-semantic alignment from a probabilistic perspective" nor "hierarchy-agnostic perspective", which I found conceptually novel.

• In response to The work (Liu et al., 2020) only studied hierarchical image embedding learning, and never explicitly addressed the composition issue discussed above, this solution is completely different from the one in (Liu et al., 2020).

In the authors' last comments, they defined the compositional label with an example: a bus might have three labels "vehicle, large-vehicle, bus" in Cityscapes taxonomy, which is the grantparent_class to parent_class to children_class relationship on the hierarchy. (Liu et al., 2020) did the same, but on the WordNet hierarchy, I didn't see the authors fundamental differences.

• In response to all reviewers have confirmed that our experiments are extensive. We have provided a detailed ablation study to provide in-depth understanding of the model.

I would like to reiterate that "we have better numbers" is NOT in-depth analysis. One of the authors' core contributions, Probabilistic Label Embedding's effectiveness against non-probabilistic embeddings, is NOT supported by any of those 10 Tables.

Thanks for the quick reply.

• In response to ... we indeed follow (Ganea et al.. 2018) for the loss function and have properly cited the paper in the article. However, they are not identical but differ in two aspects: 1) label embeddings in our method is probabilistic, and 2) we sample anchor-positive samples randomly...

Both of the authors' statements do not hold:

1. For label embeddings in our method is probabilistic is defined by eq. (3) and eq. (4) in the authors' submission, NOT in eq. (9) - eq. (11) that we are discussing.

2. For we sample anchor-positive samples randomly, rather than only linked categories in the hierarchy. the difference is minor, and the effect of this difference is NOT supported by any of the 10 tables.

To make it more specific, in (Ganea et al.. 2018), Ganea used very subtle controls over the sampling probability to control the ratio of positive and negative samples, I strongly doubt that generalizing their sampling choice to classes with higher semantic similarities (closer in the tree T ) to anchor are selected as positive samples would have a positive effect on the embedding quality.

• In response to ...our approach in Sec. 4.1 is hierarchy-agnostic, which means we ignore the hierarchical structure in the compositional labels, e.g., "vehicle, large-vehicle, bus" are treated as disjoint.

The word composition or compositional appears ZERO times in Section 4.1, while in Section 4.2, the authors claim §4.1 addresses the compositional properties of semantic concepts solely.... If it is one of the major differences from the existing methods, it is NOT supported by any of the sections, and it is also NOT supported by any of the 10 tables.

Also, the authors' comments CONTRADICTS with the last paragraph of the related work in their submission, where they wrote, ... Second, compositionality: hierarchies often emerge from the composition of basic elements. This observation has driven prior work to apply hierarchical representations learned in hyperbolic space, such as ... segmentation (Weng et al., 2021) and action recognition (Long et al., 2020)., where the references DO NOT treat the compositional labels as disjoint labels, instead they treat them as hierarchical labels.

Thanks for the reply.

• It is completely fine to use words other than composition to describe the concept. However, as one of the major concepts that is different from existing work, compositional labels is neither explained, nor explored in Section 4.1. Sections 4.1 only has three parts Hyperbolic Probabilistic Label Embedding, Hyperbolic Visual Embedding and Hyperbolic Gaussian Mixture VAE, could the authors point out where they model the compositional labels related concepts?

Thank you for your review. We have responded to your questions below, and wherever feasible, revised the paper to reflect these answers.

To begin with, we focus on the following comment:

The paper primarily discusses the advantages of conducting feature learning in hyperbolic space compared to Euclidean space in a theoretical manner, highlighting the inherent exponential growth characteristic of hyperbolic embeddings in capturing hierarchical structures. However, there is a lack of experimental evidence to support this design choice. A potential baseline approach could involve performing all the loss terms in Euclidean space, such as triplet loss in Euclidean space.

Sorry for the confusion. We had already provided the experiments to examine the choice of latent space. Please refer to Table 7 (copied below), in which Hyperbolic indicates our approach, while Euclidean means that all losses are computed in Euclidean space. The results show that using a hyperbolic space in our approach yields consistently and notably improvements against using a Euclidean space, across four datasets. This supports our theoretical analysis and evidences the efficacy of our algorithmic design. Thanks.

Next, we address the two comments on writing:

The paper does not provide explicit details on how the taxonomy of labels in different datasets is obtained. It would be beneficial to include information regarding the acquisition of label taxonomy and potentially include tree structures to visually illustrate the hierarchical relationships within the taxonomy.

Thanks for pointing this out. All the hierarchical structures we used are officially defined in respective datasets. We directly use them without any modification. Following your suggestion, we have illustrated detailed semantic hierarchies on the datasets in Appendix Fig. 2 and Fig. 3. Thanks.

The experiment section lacks a brief introduction to the metric terms CV and CMAP.

We had provided explanations to CV and CMAP in Appendix B.2. In the revised article, we additionally include formal definitions of the two terms (see Appendix Eq. 15 and Eq. 16). Thanks.

We guess that the reviewer missed this part due to the lack of proper reference in the main text, and we have fixed this. Thanks.

Further, we offer our thoughts to the open question:

This is not a question regarding this paper's issue, but more like a open discussion. Since the paper demonstrates the benefits of incorporating taxonomy as prior knowledge for recognition tasks in a close-vocabulary setting. It raises an intriguing question about the scalability of this approach to an open-vocabulary setting. Considering the superior performance achieved by visual-semantic alignment methods like CLIP on in-the-wild data, I am wondering whether this work can provide similar advantages in the open-vocabulary domain.

This is an interesting question and our insights are in two aspects:

First, theoretically, our algorithm can be promoted to open-vocabulary domain, since its basic idea is consistent with CLIP. However, in practice, there are challenges: 1) models like CLIP require very large-scale training data like image-text pairs, and explicit modelling the hierarchical structure within the data (as our method does) is hard if not impossible. Hence, implicit hierarchal structure learning appears to be more favorable and the proposed triplet loss that relies on pre-defined label taxonomy might be not suitable. 2) in comparison with contrastive-like loss used in CLIP, end-to-end optimization of VAE for large models is more difficult and expensive. Therefore, greedy optimization algorithms seem to be crucial as well.

Second, though not closely relevant to your question, we highlight that our model can serve as a more trustworthy system to handle open-vocabulary cases, as compared to current recognition models. Consider a simple label taxonomy: Animal $\rightarrow$ Okapi. For an Okapi image, even we might be not sufficiently trained to identify rare animal Okapi, our model can still hold a high confidence to interpret the image belonging to a broader category of `Animal'. This feature makes our algorithm fit to safety-critical applications (e.g., autonomous driving), even it itself is not purposely trained for open-vocabulary recognition. Thanks.