Optimizing Learning for Robust Hyperbolic Deep Learning in Computer Vision

Thank you very much for your feedback on our paper.

On the Weaknesses:

1. While this was not clear before, we have updated our methodology section to better show that the derived AdamW is a very close adaptation of the Euclidean implementation. We also compared the results to that of Radam and RSGD for HIER and the show the results below:

2. Our main aim in this paper is to increase stability and improve the efficiency of the models while maintaining or increasing accuracy. We have updated the results tables to highlight the benefits of our approaches and components, where we get upto 2.5x memory and 3x runtime savings despite performing better or marginally similar. We also manage to learn the curvature in all settings, even float16 precision, without runtime errors, which demonstrates our stability benefits.
3. With the new draft of the paper we have hopefully better explained the theoretical reasoning behind the components in the methodology section and added visualizations for better clarity on key concepts.

On the Questions:

1. We have added theorems and some mathematical proofs both in the paper and the appendix. Specifically, we have done this for maximum distance rescaling derivation, the mapping, and parallel transport operations used in the new optimizer, and better clarity for the RAdamW and efficient convolution setup.
2. Clipping parameters normally leads to information loss and non-smooth gradients which causes some training issues. Additionally, Chen et al used the sigmoid function to limit the outputs to this radius but Bdeir et al. found that it saturates too quickly which also caused performance issues. Our approach is similar to Chen et al. in that we use a normalization function, we just provide finer control over the saturation limits and mapping slope. Gradient clipping is still used in this paper similar to Bdeir et al.
3. These improvements are crucial for future adaptation into bigger models and datasets. We are now able to run Resnet-50 with reasonable computation requirements for example. Additionally, with a lot of work moving towards float16 and float8 precision, the scaling function becomes crucial since the introduced errors become larger and cause much more instability. This is demonstrated in our work by being able to learn the curvature for the HIER experiments despite it being a float16 environment.

Thank you very much for the in-depth review. Based on your comments and shared sentiment with the other reviewers, we completely rewrote most of the methodology section and parts of the experimental setup and results sections. Hopefully the new draft is clearer and better presents our motivation and approach in designing these components.

On the Optimizer: We have followed your advice and replaced operations with direct notations to reduce clutter, which also helped save space for incorporating all the updates.

On Maximum Distance Rescaling

1. While errors are minimal under float64, its computational cost is prohibitive as most hardware is optimized for float32 operations in terms of FLOPs and efficiency. Additionally, many recent works, including the HIER experiments are moving towards float16 and float8 precisions in order to reduce memory and run time. Given that some of the hyperbolic functions require potentially undefined operations like the cosh function, the introduced imprecisions lead to both performance degradation and instability. The HIER experiments were a good validation of our scaling layer under float16, allowing us to learn curvature and components without runtime errors.

2. Aside from the empirical ablations (which showed NaN values without our modifications), there are two theoretical reasons why the existing approach falls short:

   • For the hyperboloid, the projection done by only recalculates the time value. While this could be acceptable for the model parameters it alters gradient magnitudes and possibly the momentum directions due to misaligned tangent spaces, as illustrated in the new figure in our revised draft. This causes some convergence issues and training instability.

• Some of the instability arises during the update steps before projection. Specifically, curvature updates are not synchronized with manifold parameters. This means that we use hyperbolic operations that rely on the new curvature with vectors that are only defined as hyperbolic under the old curvature, including the gradients and momentums. This can cause naan and infinite values.

3. We’ve rewritten this section to define the scaling term alone (with a visualization in the appendix) and detailed its integration with hyperbolic values. Let us know if this is still confusing so we edit it further.
4. The notational inconsistencies you mentioned, along with a few others, should now be fixed. Thank you for pointing them out.

On the Convolutional Layer:

1. We updated the convolutional layer section to clearly explain the differences from Bdeir et al. and provide stronger motivation for its necessity.
2. We have now added the definitions for Lorentz transformations in the paper appendix (due to issues with the page limit)
3. We have better clarified the goal of the rotation. We mention the Stiefel manifold since we offer the option to directly learn these semi-orthogonal weights (using geoopt) in the implementation directly rather than rely on a Cayley projection.
4. The problem with normal 2d convolutions is that their outputs are not necessarily hyperbolic vectors even if their inputs are. And while you could reproject the outputs after, Bdeir et al. tested this in their work and showed that it performs much worse. As for the problem with implementation by Bdeir et al., they rely on manual window patch creation followed by a lorentz linear layer. This is mathematically correct but we can no longer leverage the heavily optimized CUDA convolution implementations, significantly increasing memory and runtime costs. We now highlight these points better in the text. Our method retains CUDA compatibility by parametrizing the weights as rotation matrices before passing them into the convolution operation
5. The Cayley projection we use, available in the Torch library, computes an orthogonal matrix $**Q**$ for a skew matrix $**A**$ by solving for $**Q** = (I + **A**/2)(I-**A**/2)^{-1}$. $**A**$ is probably calculated as $**A** = **U**-**U**^* + (**V**^*)^T**V**$ where $**U**, **V** = **W**[…, :n], **W** […, n:]$, $**U**^*$ is the conjugate, and $**W**$ is the weight matrix. But we haven’t actually looked at the torch implementation. On Experiments:
6. We now properly explain the model notations in the experimental section.
7. The curvature is fixed to the literature default value of -1

On Questions:

12. This is definitely an interesting idea and we would be curious to try and add it if it is done in time.
13. Seeing as the update steps are inside the optimizer are done without calculating new gradients or storing them, the mapping and transport costs are similar to inference level time and memory consumption and is negligible compared to the other operations
14. Clipping parameters normally leads to information loss and non-smooth gradients which causes some training issues. Additionally, Chen et al used the sigmoid function to limit the outputs to this radius but Bdeir et al. found that it saturates too quickly which also caused performance issues. Our approach is similar to Chen in that we use a normalization function, we just provide finer control over the saturation limits and mapping slope.
15. Yes actually! It helped both stabilize the loss and lead to faster convergence even if the performance gains in some cases were marginal. We will be mentioning this in the paper and adding a graph to the appendix based on your feedback as well.

Thank you very much for reading the paper. We’re very glad you found it to be interesting!

1. In terms of larger datasets, we do run the ResNet-18 models on tiny-imagenet which is a 200-class subclass of the imagenet dataset. However, the current computational requirements of hyperbolic models limit the scalability to larger datasets. The primary goal of the new convolution formulation and the bottleneck block is to reduce these computational demands while maintaining similar or improved performance. We have already achieved significant progress with some models, demonstrating a 3x reduction training time and nearly 2x reduction in memory usage (we now highlight this better in the results tables). But, further optimizations are necessary before are able to scale to larger datasets reasonably.

2. For the hyperparameter details, we used the same setup as the Bdeir et al. experiments. We now mention this clearer in the experimental section. Additionally, we will make the code available for everyone to access after the paper is published.

6. We have updated the results to include the computational requirements of the models for CIFAR-100. Currently, these requirements limit the scalability to larger datasets. However, the primary goal of the new convolution formulation and the bottleneck block is to reduce these computational demands while maintaining similar or improved performance. We have already achieved significant progress with the ResNet-50 model, demonstrating a 3x reduction in training time and nearly 2x reduction in memory usage, enabling training with reasonable VRAM and time constraints. That said, further optimizations are necessary before scaling to larger datasets becomes feasible.

7. As we demonstrate, the previous curvature optimization method is fundamentally flawed. Furthermore, to the best of our knowledge, no implementation of RAdamW has been developed so far. Therefore, we believe the proposed components represent meaningful improvements over the existing approaches.

8. Many of your comments regarding the annotations have been addressed—thank you for your detailed feedback. We would just note that the L case is the annotation we used for the Lorentz manifold, and is defined in the background section. Additionally, we have included the mathematical definitions of the Lorentz boost and Lorentz rotation in the appendix, as requested by other reviewers as well.

9. The encoder and decoder are designed to have independently learnable curvatures, allowing for flexibility in their respective manifold representations. However, all blocks within the encoder share the same curvature. Exploring the use of different curvatures for individual blocks within the encoder presents an intriguing direction for future research.

Thank you for the time and effort you put into reading and reviewing the paper.

Just as a main point before going into the discussion, we would like to make clear any misconception about hyperbolic vectors on the hyperboloid. When describing points in the Lorentz model, we call the first dimension the time component $x_t$ and the remaining dimensions the space components $**x**_s$, such that $**x**\in\mathbb{L}^n_K=[x_t, **x**_s]^T$.

Given that, $\langle **x**, **x**\rangle_{\mathcal{L}}= \frac{-1}{K}$ and $\langle **x**, **y**\rangle_{\mathcal{L}} := -x_ty_t+**x**_s^T**y**_s$, we get $x_t = \sqrt{||**x**_s||^2+1/K}$. As such, the origin, where all the space values are equal to zero, becomes $\overline{**0**} = [\sqrt{1/K}, 0, \cdots, 0]^T$ and vectors cannot be directly interchanged between manifolds of different curvatures since the time value is reliant on the curvature.

As for the comments:

1. Although generalizing from RAdam to RAdamW is conceptually straightforward, it has not yet been addressed in the existing literature. By sharing our reasoning and implementation, we aim to establish a baseline and provide an accessible solution for future research that requires it.

2. Retractions and vector transport are actually used in the model, particularly in the residual connections, initial image projection, and activation functions. However, certain components, such as batch normalization, have shown empirical instability. The approximation errors introduced by retractions and vector transports can often result in invalid values. Additionally, we use the exact operations (exponential map and parallel transport) in the optimizers as they provide greater accuracy and do not impact the inference requirements. Furthermore, since gradients are neither stored nor calculated during optimizer update steps, the computational cost of these exact operations is significantly reduced during training.

3. As you mentioned, trivialization enables the use of Euclidean optimizers, making a trivialized AdamW unnecessary. However, a significant drawback of trivializations in some works is the need to manually handle the conversion between Euclidean and Riemannian gradients for each component, including writing a custom backpropagation function for these classes. To address this, we opted for a non-trivialized approach that only requires initializing the weights as manifold parameters, allowing the Riemannian optimizers to handle these conversions automatically.

4. Thank you for raising this point. Your feedback prompted us to revise parts of the methodology section, including the curvature learning approach, and to include additional visual aids for clarity. You are correct that the default implementation of GeoOpt leaves the manifold parameters unchanged, which caused highly unstable training. This instability arose because some update steps incorrectly treated parameters as belonging to the new manifold. As a result, the time values of the unchanged parameters became invalid, leading to NaN or infinite values when used with the new curvature operations that depend on a correct time value. Simply recalculating the new time values without proper mapping also introduces issues, as it alters the gradient magnitudes and the momentum directions. To address this, we chose to perform mapping and parallel transport onto the tangent plane of the origin then back onto the new manifold. This approach preserves both the magnitudes and relative directions of parameters, as well as their gradients and momentum. We have detailed this process, along with its theoretical basis, in the revised draft.

• Specifically, we used the tangent plane of the origin because it remains parallel to the tangent spaces after curvature updates, enabling simpler vector projection and allowing direct reuse of origin tangent mappings without modification.

• Finally, you are correct that it is not strictly necessary to update the curvature last. As long as we track both the old and new curvatures, update parameters based on the old curvature, and then map them onto the new manifold, the process remains valid.

5. The proposed convolutional layer ensures that any transformation is strictly a rotation, preserving the norms of vectors while altering only their orientations. This guarantees that the output remains within the Lorentz space (since the norm of the space values remain unchanged which does not require a change in the time component), aligning with the definition of a hyperbolic operation as mentioned by Bdeir et al. and prior research. To achieve a complete hyperbolic transformation, a boost operation can then be applied, modifying the norms while keeping the orientations unchanged.

The model we based our work on is the one defined in [1] including the batchnorm layer. We understand that it is based on the method by [2] but replaces the frechet mean with the centroid for faster calculation. Additionally they provided a theoretical decomposition for the batch norm as a recentering and rescaling approach, and provide the operations for each (recentering using centroid and parallel transport, and rescaling on the tangent of the origin).

However, we were not aware of [3], although our paper does not target the batchnorm, that is very helpful insight for future work or at least to mention in this one to encourage alternatives. From a quick view, the proposed BN seems similar to that of algorithm 1 in [3], we'll take a deeper look and perform some tests to see if the BN from [1] can normalize sample statistics.

[1] Bdeir, Ahmad, Kristian Schwethelm, and Niels Landwehr. "Fully Hyperbolic Convolutional Neural Networks for Computer Vision." ICLR 2024.

[2] Lou, Aaron, et al. "Differentiating through the fréchet mean." International conference on machine learning. PMLR, 2020.

[3] Riemannian batch normalization for SPD neural networks

Thank you for going over the rebuttal!

We might have misspoken a bit, by manual backpropagation we did not mean explicitly calculating gradients by hand. As you pointed out, the operations are definitely auto-differentiable. However, when optimizing these parameters, the auto-computed Euclidean gradients must be transformed into Riemannian gradients. While this transformation is very straightforward, it must be explicitly defined and applied before the optimization step.

As examples of this:

• [1] and [2] Perform this during the ToPoincare projection operation that creates a class for the parameter that projects the gradient into Riemannian everytime.
• [3] and [5] Iterate over all the parameters before optimization, check if the variable parameterizes a hyperbolic operation, then cast the gradient accordingly.
• [4] Creates two different optimizers, during operation definition, it adds the trivialized hyperbolic parameters to a list and then optimizes that list separately with a Riemannian optimizer.

Other approaches also modify the automatic backdrop function for every hyperbolic module to perform the gradient casting.

While the process is theoretically very straightforward, the literature varies significantly on how and where to perform these gradient updates. We chose a centralized approach that minimizes the effort for future researchers to adopt.

Additionally, we did consider using trivializations, which is a valid method we've employed in other hyperbolic research. We also found that the existing code for [6] includes a trivialized batch normalization with cast Euclidean parameters. However, in our implementation, this approach yielded worse empirical results (in our models at least). It remains a very interesting direction to explore trivializations for the entire model, as some researchers report improvements, particularly when learning curvature (although these improvements might also be attributed to the poor update scheme in this particular case).

While we encourage dedicated research to compare the two possible parameterization approaches directly, our work here at least aimed to support the literature on direct hyperbolic parameter learning, so it can also benefit from stable curvature learning.

We even have the option in the library and the paper to choose between a trivialised Cayley projection approach for the convolutions or learn the rotation matrices directly on the stiefel manifold. However we did not see significant performance differences between the two.

[1] Ermolov, Aleksandr, et al. "Hyperbolic vision transformers: Combining improvements in metric learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Kim, Sungyeon, Boseung Jeong, and Suha Kwak. "Hier: Metric learning beyond class labels via hierarchical regularization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3] Atigh, Mina Ghadimi, et al. "Hyperbolic image segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[4] Dai, Jindou, et al. "A hyperbolic-to-hyperbolic graph convolutional network." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[5] Ganea, Octavian, Gary Bécigneul, and Thomas Hofmann. "Hyperbolic neural networks." Advances in neural information processing systems 31 (2018).

[6] Bdeir, Ahmad, Kristian Schwethelm, and Niels Landwehr. "Fully Hyperbolic Convolutional Neural Networks for Computer Vision." arXiv preprint arXiv:2303.15919 (2023).

Another issue concerns the BN layer.

I could not find the explicit formulation of the BN layer in your work. Do you follow Eq. (6) in [1]? Can BN normalize sample statistics? As noted in [2], the combination of exp-log-pt operations cannot normalize the sample distribution, which is crucial for BN. Only in certain specific cases, such as on the SPD manifold, where it coincides with the GL(n)-action, can it achieve normalization [3].

[1] Fully Hyperbolic Convolutional Neural Networks for Computer Vision

[2] Manifoldnorm: Extending normalizations on Riemannian manifolds

[3] Riemannian batch normalization for SPD neural networks

Thanks for the authors' reply.

Why do you say that trivialization will involve manual backpropagation? Trivialization should be much simpler, as it uses Riemannian exp or other surjective to parameterize the manifold parameters. I know and have experimented with several geometries that it might be better. In hyperbolic space, the Riemannian exp can be auto-graded by torch. I know manual BP might be necessary for a few computations in matrix manifolds. But vector manifolds, like the hyperbolic, are much simpler, as Riemannian computations are mostly decomposition-free. Currently, I do not see any operation that cannot be auto-differentiated.

Besides, even if you need manual BP, it is quite easy. This is a quite basic exercise in geometry. I do not think this is a hard issue.

In many cases, together with trivialization, the final expression can be further simplified, such as HNN++ and ones on matrix manifolds:

Matrix Manifold Neural Networks++

Building neural networks on matrix manifolds: A Gyrovector space approach

The Gyro-structure of some matrix manifolds

I may have come across a bit strongly; my intention was merely to discuss the underlying nuances of several related issues, which I believe could be important factors. Personally, I value more in-depth insights and encourage the authors to delve deeper into this line of research.

Thank you for your efforts.

I thank the authors for the further clarification.

• For trivialization. Only with empirical results, can we safely say it is not working well on hyperbolic against the Riemannian counterparts. The geometries that I experimented on tell me in several cases, it is easier and better, as it can use lots of optimizers to tune, without the efforts to re-code the Euclidean ones following the Riemannian rule.

• For the BN issues. The Riemannian BN in Lou is a natural variant of SPDNetBN. It is thoroughly discussed in ManifoldNorm. The key issue of log-exp-pt BN lies in its incapability of normalizing sample statistics in general, which was discussed in ManifoldNorm several years ago (see the homogeneous case Algs. 1-2). Without the ability to control sample statistics, it is simply a transformation, which looks like BN but fundamentally does different things. This is an interesting and complex part of geometries.

My main concern is that this work lacks in-depth insight in quite a few ways. It is more like a stack of existing techniques, although the author made efforts on several aspects. Therefore, I will keep my original score.

Thank you for the time you put into reading and reviewing the paper.

You are correct on a lot of the points you make so to answer your questions:

1. The experimental setup section lacked clarity, particularly regarding where the core-bottleneck block was applied. We have rewritten this section to explicitly state that the core-bottleneck block is used in the ResNet-50 HECNN+ model. In our implementation, we replace all the alternating Euclidean and hyperbolic bottlenecks from Bdeir et al. with the Lorentz core block and directly compare their performance and computational costs. These details are now emphasized in the ResNet-50 table, alongside the computation savings achieved.

2. AdamW has become a standard optimizer for many vision tasks, and the HIER paper uses AdamW for its experiments. To ensure fair comparisons, we also adopted AdamW in our study. Retraining HIER with Adam or comparing results obtained with different optimizers would not provide an ideal benchmark. Moreover, since the differences between Adam and AdamW are relatively minor, we adapted AdamW for hyperbolic learning. The revised paper now includes a more detailed explanation of the theoretical reasoning behind these changes. Additionally, to provide comprehensive comparisons, we reran the ResNet HIER experiments with Adam, AdamW, and SGD, and we have included these results here:

These results are based on only 2 run averages but we will update them and add them to the appendix as soon as the experiments are done.

3. Finally, while some empirical accuracy improvements are modest, we have revised the experiment section to better highlight the overall performance/efficiency gains that a lot of our components aim for. We mainly focus on maintaining or improving accuracy while significantly reducing computational requirements, as well as enabling stable curvature learning without crashes or divergence during training.