Hyperbolic Embeddings of Supervised Models

We thank the reviewer for a review whose strengths (1-3) perfectly capture our key contributions, in particular the fact that we embed models, not data.

We would like to highlight the common point to all three weaknesses mentioned (1-3): they all come down to putting substantial material in a space that was extremely constrained at submission time. Fortunately, there would be a +1 page in the camera ready. We believe we could at least partially cover (3) on more diverse use cases, assuming we can label our proposals [t8YB-B][t8YB-C] as such (see above).

Weakness (2) (extension to other supervised models) could be partially addressed in conclusion: for example, one can use our technique to embed branching programs without any modification (apart from a few tweaks in the Sarkar part, because we replace binary trees by directed acyclic graphs).

Weakness (1) would require a lot more space to deliver the reviewer’s suggestions, we are afraid. We would propose to use part of the +1 space to slightly expand Section 5, making a better connection with Figure 3 (right).

"I agree with the other reviewer's comments about the evaluation of contributions. The datasets used are indeed not very large. I am maintaining a positive score

We thank the reviewer for providing a uniform “good” rating on all three paper metrics, highlighting that our approach is novel, our method is mathematically grounded and visualization is intuitive.

weaknesses

The authors present their method primarily on small-scale UCI datasets [...] lack of comparison with existing tree visualization techniques, [...] scikit-learn or XGBoost,

[Ry4X-A] We believe there is a misunderstanding here, see [ALL-2] above. The software suggested work in the same way as the one we quote in our footnote ”*” (bottom page 2). It is not a criticism: all softwares proceeds in the same way – embed a graph (binary rooted tree), superimpose the classification information. The only differences between softwares are “cosmetic” details to represent the full models: use charts or pies or boxes to represent local class probabilities, add text / color, etc. . We write cosmetic because it all end up to the same conclusion when one realizes what we write in [ALL-1]: it is largely impossible to represent a tree with 200+ nodes (that we use in our experiments) and get any meaningful information of how classification changes as we descend the tree, how nodes / leaves in different parts of the tree compare in term of confidence, etc . Monotonic Decision Trees (that we introduce) do not solve the problem because the issue is not on the “cosmetic” details (that is why we use the term “cosmetic”): the issue is that the backbone comes from the graph theory representation and is suboptimal to grasp models that big when applied to ML. We believe the hyperbolic embedding is very helpful to address the issue. We refer the reviewer to the attached file above to see for themselves.

[...] Potential computational overhead: The paper does not adequately address the computational complexity of transforming decision trees into their hyperbolic representations. [...]

[Ry4X-B] We would be happy to add in the camera-ready (using the +1 page) the fact that the complexity of computing the MDT is linear in the size of the tree (DT: see algorithm getMDT in our paper; class method PROCESS_TREE_GRAPHS in file MonotonicTreeGraph.java) and our variant of Sarkar’s algorithm is a cheap as Sarkar’s, so there is no computational overhead compared to a standard visualization of a DT.

[...] Ambiguity in interpretability gains: While the authors claim improved interpretability, they do not provide a quantitative measure or user study [...]

[Ry4X-C] While we hope that our argument in [ALL-2] makes the case for “not small” models at least, we emphasize that the whole trend of hyperbolic geometry in ML has been built from the mathematical properties and benefits of tree/hierarchical representations in the model. All the papers that we cite (either published in ICML or NeurIPS) make this case. We quote, in order of our reference numbering, the first ones:

[6]: “[...] hyperbolic embeddings [...] are more aligned with the geometry of trees than standard Euclidean embeddings [...]”;

[7]: “[...] Learning representations of data in hyperbolic spaces has recently attracted important interest in Machine Learning (ML) due to their ability to represent hierarchical data with high fidelity in low dimensions [...]”

[8]: “[...] The adoption of hyperbolic geometry for graph embeddings has sparked a vibrant and rapidly growing body of machine learning research [...] driven by the compelling advantages offered by hyperbolic spaces, particularly in capturing hierarchical and tree-like structures [...]”

(and so on) The motivation for our approach is no different from these.

questions

[...] How does the method perform on regression trees as opposed to classification trees? [...]

One would have to change the first step (the node embedding) of our approach because the losses we use are fundamentally for supervised classification. Then, the rest could be just about the same because of the similar tree structure (getMDT, modified Sarkar).

[...] How does this hyperbolic embedding method perform on highly imbalanced datasets, particularly in visualizing minority class decision boundaries? [...]

We suspect there is a misunderstanding here: we do not represent data but a model; therefore there is no such boundaries because there is no “training point cloud” mapped. However there is an effect of class imbalance, which is to (predictably) move the root of the tree far from the center of the disk. Thus class imbalance can be spotted “at lightning speed”, see L273-L274.

We really appreciate the comment that our method “[...] addresses many fundamental issues in hyperbolic learning presenting [...]”. This is a strong selling point for our method, which unfortunately comes with a challenging presentation problem. We hope our rebuttal helps in clarifying views and questions.

We also really appreciate the comment that one part of our contribution (the t-self) indeed addresses a general, “[...] key problem across hyperbolic distance metrics that seemingly helps avoid collapse to the boundary of the ball [...]” (indeed, it avoids collapse).

weaknesses

“[...] It is not clear [...] Some examples of fully hyperbolic classification models include [1,2]. I may be misunderstanding [...]”

[j9dG-A] Notations and naming can be misleading. Indeed, there is a misunderstanding: please have a look at Figure 4 in [1] and Figure 2 in [2]. The legend makes it clear that it is data which is indeed embedded. Note that some details of the model can be embedded as well, like decision boundaries (Figure 2 in [2]), but at the end of the day, the hyperbolic space is the data’s in those papers, while it is the model’s in ours. We also refer to [t8YB-A] for a limitation of [1][2] in the data space.

“[...] The presentation and writing overall [,,,] impedes the readability of the work. [...]”

We apologize for the inconvenience. We believe that we can make use of the +1 page in the camera ready to expand some of the technical material’s explanation in the main file.

“[...] The experimental results are purely demonstrated as visual comparisons, and no quantitive measures are presented. [...]”

There are actually qualitative measures of the quality of the embeddings displayed on each, the error $\rho$, but we do understand the point made, to perhaps instead (or in addition) make a separate quantitative analysis. We have made such a proposal, that would be straightforward to implement using our code, in [t8YB-B][t8YB-C].

“[...] Arguably the t-self method you propose to improve visualisation [...] should not be positioned in the contributions as a method for visualisation.[...]”

It is right, it is a transformation that can be used in any use of the Poincaré disk model. We were hoping that our remarks in L305-L308 would help grasp this. To improve this part of the readability, we would be happy to include part of it in the introduction as well, using part of the +1 page camera ready.

“[...]Experimental/implementation settings are omitted from the work limiting the ability of replication[...]”

We strongly disagree on this point: the code we provide contains a UCI dataset and the resource file to run experiments exactly as we did for this dataset (all is explained in the README). Using another dataset requires changing just one line in the resource file (“@PREFIX,abalone”).

On minor comments, we appreciate the suggestions and we are sorry that the small space allocated probably caused potential misunderstandings (ex: L106 goes with L113 for the bold face statement), we are confident that the +1 page can allow us to improve readability.

questions

“[...]On Line 60 you state insist that these models do not represent models in hyperbolic space[...]”

This is related to [j9dG-A] above and we refer to it. In short, in all those papers, the hyperbolic space is the data’s in those papers, while it is the model’s in ours.

“[...]How does the proposed method compare to Euclidean models? [...]”

The substantial benefits come from the properties of the hyperbolic space, and is common to all papers using such geometric models. See for example [t8YB-C] for a concrete example, that was also part of a paper [29], albeit in a different context. There is, however, an additional benefit in our case, which comes from the singular property that there is a relationship between the link function of the log loss and the Poincaré distance which makes a tree node embedding not just natural: it provides a direct visualization of classification confidences in the disk ! (though simple, we have not seen this connection elsewhere).

“[...]You suggest the t-self may be useful as a standard encoding in hyperbolic space, have you made any empirical findings to support this? [...]”

Excellent question: at the moment, with our paper cramming so much content, the sole observation that we have is experimental in our context (the remark appears in L292-L293), and its usefulness hopefully becomes clear in Table 1. We use the conditional (might) instead of may because we indeed suspect a much more thorough analysis can be carried out, which would require not just digging in experiments but also in various encodings and against additional geometric properties of the Poincaré disk model, to establish the potential for a broader standard. This is out of the scope (and size !) of our paper.

“[...]The limitations of only embedding the monotonic DT are not discussed? It may assist for clean embeddings but what are the drawbacks?[...]”

As we understand the question, we see three interpretations for “limitations” that specialize in three different questions:

• Are there limitations in terms of accuracy for models (MDT vs DT) ? We kindly refer the reviewer to Appendix Section D.II Line 689 and Table II. Here the effects of reducing a DT to its monotonic form is explored. Here we find that in most cases the accuracy / behaviour of the MDTs are similar in performance to the original DTs. Note that we suspect there is a formal rationale to this observation, but it is out of the scope of this paper.

• Are there limitations in terms of computational cost for creating the MDT ? The answer is clearly no, see [Ry4X-B].

• Are there limitations for the user, any pricey “entry ticket” for using MDTs (vs DTs) ? This question is about the easiness of using MDTs vs DTs. Knowing DTs, the entry ticket to get to MDTs would arguably be minimal compared e.g. to get to kernels.

We thank the reviewer for noting that our approach is “[...] very original [...]” – it is indeed the first of its kind and we believe the problem we address is of broad importance. We also appreciate the comments on the formal part of our approach.

strengths

In my opinion, the methods for embeddings of discrete data into hyperbolic space is less covered in the literature, [...] adding more methods covering this topic [...]

[t8YB-A] This is an excellent observation that we did not think about and that would surely make our contribution stronger: the approaches [8][11] rely on real data embeddings, making it tricky to embed discrete or categorical data. Because we embed models that are naturally fit to handling any kind of numerical / nominal data, our technique encompasses any kind of such data.

weaknesses

“[...] In my opinion, the paper would have benefited from a more clear presentation of quantitative results [...]”

[t8YB-B] A fair comment, due to the fact that we mainly focuses on the “visualization part”; fortunately, this can easily be addressed as each visualization also includes an error term: we propose to summarize (mean $\pm$ std dev) the errors ($\rho$) made on each dataset, in a separate table, that would hopefully fit in the +1 page allocated for camera-ready.

[t8YB-C] Reading this comment from the reviewer, we came up with another idea to test one of the benefits of hyperbolic embeddings to our setting, which would be trivial to code and run: it is well known that, as points move closer to the border, the hyperbolic distance $d_{B}(**z**, **z**’)/2$ approaches the average $(d_{B}(**z**, **0**)+d_{B}(**z**’, **0**))/2$ (see [29] Figure 1). For our application, the benefit is substantial: the expectation is indeed the expectation of absolute confidences of the two nodes mapped on $**z**$ and $**z**’$ (the nodes’ quality) and what the property says is that it can be approximated by just looking locally at (half) the distance between the two points (in terms of visualization, think zooming on a part of the disk or having a hovering tool providing the information of this distance “on the fly”). We would be happy to compute a discrepancy between the two terms above, as a function of the localization of the nodes on the disk.

questions

“[...] A small remark: the most part of plots use red/green colours which can be challenging to distinguish for colour blind people. [...]”

The reviewer is absolutely right: we propose, in addition to the color, to differentiate using shapes (e.g. circle for red, square for green, for all nodes). This would be a one-liner change in the code.

“[...] The sentence in line 697 is unclear. [...]”

Our experiments involve a 10-folds stratified cross validation (L672) to compute average errors $\rho$. In our visualizations, we can only show one of the ten models. In the spirit of fairness, we display the model learned on the first fold (instead of e.g. picking the best model out of the ten). This is what is summarized in L697 and we would be happy to be more specific.

“[...] My main concern regarding the paper is related to the difficulties while evaluating the performance of the approach and it mainly justifies my rating. However, I am open for discussions with the authors.[...]”

We are glad the reviewer signals their openness for discussion. Regarding evaluation, we hope that the suggestion we make in [t8YB-B][t8YB-C] would lead to a purely quantitative evaluation of our embeddings that would satisfy the reviewer.