Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning

We thank the reviewer for the constructive feedback and suggestions. We answer the remaining questions by points as follows.

Q1. Ablation on hyperparameter sweeping

Due to restricted space, we report the ablation for the clipping threshold. We refer to our response to Reviewer fRer for the ablation with the curvature value. Remaining hyperparameter ablations (alignment weight, entailment cone weight, temperature) are included in the revision.

Q2. The use of the Lorentz model… it may be the case that this is purely a matter of convenience/numerical stability—if so, I would like the authors to clarify this.

Yes, we use the Lorentz model for its numerical stability, following recent work [1, 2]. Its space-time interpretation also fits our context, allowing us to use the time-like dimension to model continual updates and embedding uncertainty.

Q3. The derivation of Equation 10 is unclear: what is the relationship between this quantity and the distance from the origin of the hyperboloid?

We define Lorentz uncertainty via its isometric relation to the Poincaré model. Both share the same form, based on the Euclidean norm of the pre-exponential embedding.

$$\mathrm{Uncertainty}(\mathbf{h}^{\mathcal{L}}) = \mathrm{Uncertainty}(\mathbf{h}^{\mathbb P}) = 1 - \frac{1}{\sqrt{K}} \mathrm{tanh}(\sqrt{K} \|\| \mathbf z \|\|)$$

The distance from the origin of the hyperboloid is given by

$$d(\bar{\mathbf{0}}, \mathbf{h}) = \mathrm{cosh}^{-1} (\frac{1}{\sqrt{K}} cosh(\sqrt{K} \| \| \mathbf z \| \|)$$

Both the distance to the origin and our uncertainty measure depend explicitly on the Euclidean $\ell_2$-norm before the exponential map. While prior work uses the unbounded origin distance as a proxy for uncertainty, our bounded [0,1] measure is more interpretable and manageable.

Q4. What other metrics can be evaluated? classification performance?

We report CMC@1 and mAP to focus on retrieval performance. The table shows accuracy and retrieval results for various alignment methods, including cross-accuracy using the old classifier on new features. Accuracy trends generally follow retrieval, though with smaller gaps.

Q5. How does exponential volume growth relate to continual model updates, and why might it be preferable to polynomially growing spaces like Euclidean space?

We argue that exponential volume growth in hyperbolic space allows more room to represent new entities with smaller increases in radius, which is advantageous for continual updates. In contrast, Euclidean space requires rapidly growing radii to accommodate new entities.

Moreover, Lemma 1 in [3] shows that in Euclidean space, the probability of new class prototypes aligning with a trained model decreases exponentially with the number of dimensions and classes, leading to an impossibility result for backward compatibility. We hypothesize that hyperbolic geometry with its exponential growth property may mitigate this issue, though a formal analysis is left for future work.

Q6. Can this approach be used to train a new model to align embeddings between several different models? Assume we don’t know anything about how they were trained.

Aligning a new model to multiple old models is challenging without knowing their training processes. Some models may have incompatible embeddings, enforcing inherent trade-offs in the compatibility. A common solution is to add a projection layer to map embeddings into a shared space [4], which is orthogonal and can be integrated to our method.

Q7. What happens if you ablate the hyperbolic MLR (i.e. compute loss on Euclidean model, then exp map, then compute the other losses)—does this do better or worse?

We replaced the hyperbolic MLR with its Euclidean counterpart while keeping the entailment cone and RINCE loss in hyperbolic space. This led to a significant drop in both the new model's performance and its compatibility. This is likely due to conflicting signals between the Euclidean MLR and hyperbolic RINCE loss.

Q8. Writing structure and typos

We reorganized the "norm control" and "overall training" into a separate section and fixed the typos.

References:

[1] Bdeir et al. Fully Hyperbolic Convolutional…, ICLR’24. [2] Desai et al. Hyperbolic Image-Text…, ICML’23. [3] Biondi et al. Stationary Representations: Optimally…, CVPR’24. [4] Linia et al. Asymmetric Image Retrieval with Cross Model...

We thank the reviewer for the constructive feedback and suggestions. We answer the remaining questions by points as follows.

Q1: why the Lorentz uncertainty was defined the way it was.

We define the Lorentz uncertainty using the isometric relation between the Poincaré and Lorentz models. Both uncertainty measures in Poincaré and Lorentz models have the same form with respect to the norm 2 of the Euclidean embedding before the exponential map:

$$\mathrm{Uncertainty}(\mathbf{h}^{\mathcal{L}}) = \mathrm{Uncertainty}(\mathbf{h}^{\mathbb P}) = 1 - \frac{1}{\sqrt{K}} \mathrm{tanh}(\sqrt{K} \|\| \mathbf z \|\|)$$

Both the distance to the origin and our uncertainty measure depend explicitly on the Euclidean $\ell_2$-norm before the exponential map. While prior work uses the unbounded origin distance as a proxy for uncertainty, our bounded [0,1] measure is more interpretable and manageable. We will discuss this explicitly in the revision.

Q2+3: What is the CMC@1 and mAP performance of the models?… What do the self and cross columns mean in Table 1?

The self (new-to-new retrieval) and cross (new-to-old retrieval) columns in the Table refer to the original model performance (CMC or mAP). We thank the reviewer for pointing this out, and we will explain this explicitly in the paper to avoid future confusion.

Q4: Is there a reason why using the RINCE-based loss sometimes results in a large drop in compatibility?

During the experiments, we observed that training with RINCE-based loss sometimes is more difficult to converge in some cases, especially on the Vision Transformer model. This maybe because of the instability of the exponential map when applied to the large norm produced by ViT. However, when training with the entailment cone loss, the training becomes more stable.

Q5: Given how the norms grow with repeated model updates and the possible resulting instability, how does the performance of this method compare to Euclidean methods after many model updates?

We conducted an experiment with five consecutive model updates to test the performance of different compatibility methods on the CIFAR100. The training data will be split into five sets, each containing 20, 40, 60, 80, and 100 (all) classes, respectively. For the first two models (time steps 0, 1), we will use ResNet18 as the base model and we will update to ResNet50 at the last three steps (2, 3, 4). For each method, we report the CMC@1 compatibility matrix for HBCT, HOC and BCT. Columns are the encoders for the query, and rows are the encoders for the gallery set. It can be seen that the HBCT can maintain the compatibility with the old model after several updates ($\phi_1 / \phi_4$) = 0.32) , while HOC and BCT quickly deteriorate after a few model updates ($\phi_1 / \phi_4$ = 0.207 for HOC and 0.14 for BCT).

HBCT

HOC

BCT

Q6: It seems that a possible limitation of the model is that performance may degrade after many model updates.

Yes, we have discussed this in the limitations section. Increasing the clipping threshold after each update can lead to numerical instability when it becomes large (e.g., >10). One potential solution is rescaling the time dimension of all embeddings after a certain number of updates (e.g., >20). Another approach is online backfilling, where a subset of items is used to update the gallery set—this can help manage the growing time dimension.

We hope that we have addressed all your concerns adequately. We have updated the manuscript according to your comments. Please let us know if we can provide any further details and/or clarifications.

We thank the reviewer for the constructive feedback and suggestions. We answer the remaining questions by points as follows.

Q1: Training could be more expensive with hyperbolic geometry

We agree hyperbolic models may be more expensive, but our method only adds a simple exponential map to a Euclidean model, incurring minimal overhead. We report the running time of ResNet18 with a batch size of 256 in the table below.

Q2: Applicability could be limited as most methods are not assumed to be trained with hyperbolic embeddings spaces.

Our method is general and can be applied on top of existing Euclidean embedding models.

Q2: Some questions arise when the geometry of the space is not flat and nor with constant curvature

We consider Euclidean and hyperbolic geometries because they are most commonly used in existing retrieval systems, mostly because the geodesic distance between two points can be computed efficiently. In some cases, manifolds with non-constant curvature can better represent data, but the geodesic distance will be more computationally expensive, making it difficult to generalize to large-scale retrieval systems.

Q3: The comparison with these approaches that use hyperspherical geometry (with cosine similarity)?

Hot-Refresh and HOC are two alignment methods that employ hyperspherical geometry, as they align models using a contrastive objective based on cosine similarity. Since cosine similarity normalizes embeddings to have unit norm ($| z |_2 = 1$), these methods map embeddings onto a shared unit hypersphere. Pairwise embedding comparisons are then computed using the negative inner product as a distance measure. We will clarify this in the baseline discussion in the revision.

Q4: How does the similarity measure used in CMC affect results?

We observe that retrieval performance remains consistent across different similarity measures. Below are the results on CIFAR100:

• Euclidean-ResNet18:
  • Cosine: mAP 0.6543, CMC@1 0.7157
  • Euclidean: mAP 0.6543, CMC@1 0.7157
• Hyperbolic-ResNet18:
  • Geodesic: mAP 0.6548, CMC@1 0.7177
  • Lorentz inner product: mAP 0.6548, CMC@1 0.7177
• Using hyperbolic distance on Euclidean embeddings (via exponential map):
  • Hot-refresh original:
    • new-to-new retrieval (self): mAP: 0.649485, CMC@1: 0.7147
    • new-to-old retrieval (cross): mAP: 0.347584, CMC@1: 0.4977
  • Hot-refresh -> exponential map
    • new-to-new retrieval (self): mAP: 0.649482, CMC@1: 0.7147
    • new-to-old retrieval (cross): mAP: 0.347583, CMC@1: 0.4978

Q5: In Table 1 the columns corresponding to self and cross denote the base loss? what is this number referring to, the original model performance?

Yes, self (new-to-new retrieval) and cross (new-to-old retrieval) refer to the original model performance (CMC or mAP). We will explain this explicitly in the paper to avoid future confusion.

Q6. Experiments larger datasets

We mainly followed [1, 2] to test on CIFAR100 and Tiny-ImageNet, where hyperbolic geometry is shown to be effective. Given the short time frame for rebuttal responses, extensive experiments on large datasets are beyond our immediate scope but we acknowledge the valuable suggestion and plan to explore larger-scale evaluations in future research to further validate the approach.

Q7: How does curvature K affect the performance? why the choice of fixing it to 1 for all experiments?

We set a fixed curvature value following the existing literature [1]. We conducted an ablation study to examine how different curvature values affect compatibility performance. Generally, $K \in [0.5, 1.0]$ yields stable outcomes in both compatibility and the quality of the new model. When we also tested the learnable curvature, it showed good performance in the new model but negatively impacted compatibility performance.

Q8: How impactful is pretraining from scratch assuming hyperbolic geometry? How does this compare to finetuning from a pretrained model assuming hyperbolic geometry as opposed to training from scratch?

In our experiment, ResNet18 is trained from scratch, while ViT is finetuned from ImageNet21K pretrained weights to reflect a typical usage of open-source models. We do not pretrain ViT from scratch due to the poor convergence on CIFAR100 and TinyImagenet.

References: [1] Bdeir et al. Fully Hyperbolic Convolutional Neural Networks for Computer Vision, ICLR’24. [2] Biondi et al. Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements, CVPR'24.