Intrinsic Mesh CNNs

We are very grateful for the provided review. In the following, we would like to address the points and questions.

The authors focus on a limited body of work in the geometric deep learning literature, namely, three early formulations of intrinsic convolutions. Furthermore, they only cite one reference regarding all the theoretical framework of gauge equivariant CNNs, Bronstein et al., 2021.

The theoretical contribution of our work focuses on the discovery of previously unknown but insightful properties of intrinsic surface convolutions which depend only on the definition of priors and is as such more fundamental and independent of gauge transformations. Therefore, while we think our theory also applies to gauge-equivariant intrinsic surface convolutions, this topic exceeds the scope of this work.

The mathematical derivations contain typos (the limit of Gaussian functions for decreasing variance should be for $n \to 0$ not $n \to \infty$, and the use of the notation $\Delta_\theta$ and $\Delta_{\theta, w}$ should be introduced.

We thank the reviewer for pointing out typos in our manuscript and will of course correct those, as well as add the full definition for the anisotropic Laplacian $\Delta_{\alpha \theta}$ into the appendix to ensure completeness.

As noted by the authors themselves, the Dirac distribution is not a function. This weakens the link with the theoretical results presented before.

While mathematically possible to expand the domain of priors to transition kernels and with that allowing for several new priors such as the Dirac prior, this redefines the Riemann integrals into Lebesgue integrals which come with a significant amount of technical overhead, as stated in the manuscript, and whose description would not help to understand our contribution. Thereby, our choice of describing the Dirac prior allows to compare it to all previous models that use priors, which can be expressed in terms of Riemann integrals, for example by comparing their sets of learnable features. This, in turn, again shows the potential of reformulating IMCNNs in terms of using a prior rather than a patch operator.

The authors claim to "we show that existing definitions for surface convolutions only differ in their prior assumptions about local surface information". It is known in the community, and the authors show it for the formulation of Monti only, who had already shown their formulation encompasses the previous Geodesic CNN and Anisotropic CNN.

We generalize the framework of Monti et al., 2017, thereby obviously allowing for the definition of models mentioned by Masci et al., 2015, and Boscaini et al., 2016, but additionally also the model of Bronstein et al., 2021, and new models that use priors in the sense Theorem 1.

Can the theoretical framework be strengthened by reformulating it in terms of distributions and extended to the class of test functions?

While the discussion about distributions and test functions definitely is possible, we are convinced that the choice of transition kernels is more natural as we think of signals on the manifold in a statistical fashion. If we assume that our IMCNNs only learn test functions, we would restrict our research on a comparably limited domain.

The network architecture used in the experiments uses angular max pooling. Can the authors clarify why this is needed? Isn’t it a step backwards compared to the work of Monti, or other mesh CNNs such as FeastNet that do not require angular max pooling?

Angular max-pooling (AMP) is required as a means to resolve the angular coordinate ambiguity problem, a direct consequence of gauge-ambiguity, in practice. It was also used by Monti et al., 2017. FeastNet (Verma et al., 2018), while representing interesting work, is a fundamentally different approach to surface convolutions.

How did the authors choose the priors compared against the Dirac prior?

The normal distribution has also been used by Monti et al., 2017. The other priors portray valid instances for our class of IMCNNs and were selected for further comparison.

Sources

Masci, Jonathan, et al. "Geodesic convolutional neural networks on riemannian manifolds." Proceedings of the IEEE international conference on computer vision workshops. 2015.

Boscaini, Davide, et al. "Learning shape correspondence with anisotropic convolutional neural networks." Advances in neural information processing systems 29 (2016).

Monti, Federico, et al. "Geometric deep learning on graphs and manifolds using mixture model cnns." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

Verma, Nitika, Edmond Boyer, and Jakob Verbeek. "Feastnet: Feature-steered graph convolutions for 3d shape analysis." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

Bronstein, Michael M., et al. "Geometric deep learning: Grids, groups, graphs, geodesics, and gauges." arXiv preprint arXiv:2104.13478 (2021).

We appreciate the provided feedback of the reviewer and would like to address the raised points.

My primary concern with this paper is that it’s not clear to me why it is called a prior and how using a different prior helps. It simply seems like different instances of the patch operator, akin to the Gaussian in Masci et al. 2015.

The prior can be seen as a formal tool for encoding assumptions that you have about the signal on the surface. Our contribution formally describes the effects of priors. Key to the description is the introduction of the notion of learnable features in Section 4 of our manuscript. Depending on the chosen prior, the set of learnable features of an IMCNN can be more or less comprehensive. Given a problem, like for example classification, we search for features within the set of learnable features that cause the IMCNN to perform well. If the set of learnable features is smaller, we expect that good features within that set are found faster. However, a small set of learnable features might exclude features which lead to better performances. Larger sets of learnable features, in turn, imply more unstable training as searching for good features becomes more complicated due to the increased amount of possible features to learn.

If the Dirac prior is the most expressive, why should one consider other priors? The results don’t seem to show much difference between the other priors.

Our paper introduced the Dirac prior as a means to connect the framework of Monti et al., 2017, with the definition of intrinsic surface convolutions from Bronstein et al., 2021. We subsequently discovered that the Dirac prior is very comprehensive as it is associated to a large set of learnable features. This comes with a higher probability of finding more suitable features for the underlying problem, as indicated by the experiment results in which Dirac outperforms the other priors. However, this comes at the cost of a large search space. If unstable training is observed and domain knowledge is available, we have the option to limit our network onto a certain set of learnable features to stabilize training by using a more restrictive prior.

Are the shot descriptors orientation invariant?

SHOT-descriptors are rotation-invariant as they are computed with the help of relative angles between vertex normals, which do not vary under the application of isometries. However, our work does not discuss SHOT-descriptors. Therefore, we kindly refer to the original paper (Salti et al., 2014) for further questions about SHOT-descriptors.

Does a global gauge exist?

A global gauge does not exist in the general case. A gauge $\omega$ defines a local basis for a tangent space $T_u M$ with $u \in M$ of the manifold $M$. In the general case, $T_u M$ and $T_v M$ are different vector spaces for $u \not = v$, which already shows that a gauge usually cannot be globally equal for multiple tangent spaces. However, in the special case that the underlying manifold is parallelizable, we at least have a global continuously changing gauge.

Why was a gauge-equivariant method such as [1] not considered? It seems very relevant to the proposed method and has the advantage that the output transforms accordingly with the gauge transformations.

The theoretical contribution of our work focuses on the discovery of previously unknown but insightful properties of intrinsic surface convolutions which depend only on the definition of priors and is as such more fundamental and independent from gauge transformations. Therefore, while we think our theory also applies to gauge-equivariant intrinsic surface convolutions, this topic exceeds the scope of this work.

Sources

Salti, Samuele, Federico Tombari, and Luigi Di Stefano. "SHOT: Unique signatures of histograms for surface and texture description." Computer Vision and Image Understanding 125 (2014): 251-264.

Monti, Federico, et al. "Geometric deep learning on graphs and manifolds using mixture model cnns." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

Bronstein, Michael M., et al. "Geometric deep learning: Grids, groups, graphs, geodesics, and gauges." arXiv preprint arXiv:2104.13478 (2021).

We thank the reviewer for the valuable feedback. In the following, we would like to briefly address the raised points and questions.

Point 1:

While it is true that earlier experiments report better training results, a direct comparison with these experiments is misleading as they applied post-processing methods to the network output, such as functional maps in Masci et al., 2015, and Boscaini et al., 2016, or intrinsic Bayesian filters in Monti et al., 2017. Furthermore, the Dirac prior is a proposal of our work, which is fundamental to establish a connection between the theory of Bronstein et al., 2021, to the practical framework of Monti et al, 2017. Notably, it was not discussed in Monti et al, 2017.

Point 2 and 3:

Learning the parameters directly has been suggested by Bronstein et al., 2021. Pre-convolving the template $t(·)$ with a handcrafted kernel $w(·)$ has implicitly been done in what previously has been referred to as the patch operator (Masci et al., 2015; Monti et al, 2017). Our paper proposes a reformulation of the parametric framework from Monti et al, 2017, which stresses the theoretical implications of handcrafted kernels $w(·)$ by giving rise to the notion of learnable features. This notion not only explains why the handcrafted kernel encodes prior knowledge, which in previous work has merely been referred to as either interpolation coefficients, weighting function or kernel (Masci et al., 2015; Monti et al, 2017), but also allows us to observe a partial order on priors in the context of intrinsic mesh CNNs. Therefore, we now know that different priors can vary in their comprehensiveness. One very interesting example is given by our proposed Dirac prior, which on the one hand, when inserted into our reformulation, yields the theoretical definition for intrinsic surface convolutions of Bronstein et al., 2021, and on the other hand renders the pre-convolution irrelevant.

Point 4:

In order to simplify the repeatability of our experiments, we want to be precise about our experiment setup.

Point 5:

While we agree that there exist more interesting benchmarks, we want to clarify that our benchmark is no artificial toy benchmark. For instance, compared to Hanocka et al., 2019, our input meshes have 41328 edges. Their benchmark used considerable smaller meshes with roughly 750 edges for mesh classification and roughly 2250 edges for mesh segmentation. Additionally, in difference to Hanocka et al., 2019, we have trained our IMCNN to predict dense point-correspondences. Predicting dense point-correspondences is commonly regarded as a notoriously hard problem in the computer vision community (Van Kaick et al., 2011). Lastly, the benchmark is a common benchmark in the background papers (Masci et al., 2015; Boscaini et al., 2016; Monti et al, 2017) .

Question 1:

The Dirac-prior effectively gets rid of a matrix multiplication, therefore requiring less computation than the IMCNNs which use a specific prior.

Question 2:

In order to get an intuition for what difference the hyperparameter make, it is helpful to take a look on Figure 4 in the appendix. While it is difficult to quantify how much difference varying hyperparameter cause in general, we expect certain aspects to hold true. For instance, the GPC- system radius should be chosen smaller or equal to the injectvity radius of the exponential map.

Sources:

Van Kaick, Oliver, et al. "A survey on shape correspondence." Computer graphics forum. Vol. 30. No. 6. Oxford, UK: Blackwell Publishing Ltd, 2011.

Masci, Jonathan, et al. "Geodesic convolutional neural networks on riemannian manifolds." Proceedings of the IEEE international conference on computer vision workshops. 2015.

Boscaini, Davide, et al. "Learning shape correspondence with anisotropic convolutional neural networks." Advances in neural information processing systems 29 (2016).

Monti, Federico, et al. "Geometric deep learning on graphs and manifolds using mixture model cnns." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

Hanocka, Rana, et al. "Meshcnn: a network with an edge." ACM Transactions on Graphics (ToG) 38.4 (2019): 1-12.

Bronstein, Michael M., et al. "Geometric deep learning: Grids, groups, graphs, geodesics, and gauges." arXiv preprint arXiv:2104.13478 (2021).

Dear authors,

I thank you for your reply.

Can you refer exactly to where Monti [4] refers to $J$ as the number of neighbours, in the context of their proposed approach? Under equation (9) in [4], they write

$J$ represents the dimensionality of the extracted patch

Later they write that there are $2Jd$ parameters. The index $J$ is thus a statement about their parametrization and has nothing to do with the number of neighbours. In fact, it's the number of mixture components in their CNN, thus it's a discrete index. I therefore fail to see why it makes sense to take that index to have values in the tangent space, and also fail to see why your description theorem 1 has anything to do with Monti's formulation.

Could you please elaborate on why your formulation in theorem 1 is a sensible "continuous version" of Monti [4]?

[4] Monti, Federico, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M. Bronstein. 2016. “Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs.” http://arxiv.org/abs/1611.08402.

We thank the reviewer for the undertaken efforts and want to address the raised concerns.

Any linear map $s: M \to \mathbb{R}$ from a scalar signal on a $n$-manifold, to another scalar signal can be represented by a function [...]

The formulation of the intrinsic manifold convolution follows a different intention. Intrinsic manifold convolutions (Bronstein et al., 2021) represent a generalization from standard Euclidean convolutions to convolutions on Riemannian manifolds. For this we require a notion of relative direction between points within the manifold. Bronstein et al., 2021, proposes to use tangent vectors. We associate tangent vectors to coordinates by selecting one from multiple possible gauges for each tangent plane. Eventually, we map said tangent vectors via the exponential map onto the surface to retrieve manifold points on which the surface signal is defined.

Monti et al (2017) [4] recognize that the gauge can not be chosen uniquely, so choose $J$ frames apply [...]

Monti et al., 2017, proposes a general algorithmic framework for the construction of previous geometric CNN architectures such as, but not limited to, geodesic CNNs (Masci et al., 2015). Within said framework, when specifying for the geodesic CNNs, $J$ refers to the amount of neighbors around surface point $x$, at which we calculate the convolution. In the continuous case, the sum becomes an integral over the disc $B_R(**0**)$, that contains geodesic polar coordinates which portray the input to the selected gauge. Thereby, $R$ is pre-specified radius that typically refers to the injectivity radius of the earlier discussed exponential map.

However, what's done in theorem 1 in the equation of, is very different from any of the above. [...]

Our convolution is equivalent to how Monti et al., 2017, and Bronstein et al., 2021, would compute intrinsic surface convolutions. We have shown this in Theorem 1. The proof is provided in the appendix and comes down to the application of Fubini’s theorem. The function $t : \mathbb{R}^n \to \mathbb{R}$ represents the template that shall be learned, the function $w : \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$ references a hand-crafted prior.

The authors appear to suggest that this is similar to what Monti et al (2017) [4] does, but this appears to me as very different. [...] So what's the point of analyzing the double convolution, which no one uses? It's very different from what [4] proposes, so how does it bridge a gap between anything?

As noted before, Theorem 1 states that the framework of Monti et al., 2017, is equivalent to our notation. Selecting Dirac as a prior does not make the intrinsic surface convolution an identity operation, but returns the formulation for intrinsic surface convolutions of Bronstein et al., 2021. That is, selecting Dirac as a prior within our reformulation of Monti et al., 2017, which we have proven to be equivalent, closes the theoretical gap between the algorithmic framework of Monti et al., 2017, and the theoretical derivation of the intrinsic surface convolution from Bronstein et al., 2021.

The authors cite [1] for the gauge equivariant convolution. This should be a citation to [2] and also [3] in the context of meshes.

We are aware of the literature by Taco et al., 2019, and De Haan et al., 2020. We thank the reviewer for pointing out that these citations are missing and will of course add them to the manuscript.

The authors should compare to [3] in their experiments. Those authors found close to 100 correspondence at 0 geodesic distance for the the FAUST experiment [...]

The theoretical contribution of our work focuses on the discovery of previously unknown but insightful properties of intrinsic surface convolutions and not on extensions of them such as gauge-equivariant convolutions.

It’s very confusing to use $w$ (w) and $\omega$ (omega) in the same equation, the first referring to weights and the latter to the gauge.

Thank you for your feedback on the readability of our paper. We can rephrase $w(·)$ to $p(·)$, highlighting that $w(·)$ does not contain trainable weights but represents a handcrafted prior.

Sources:

Masci, Jonathan, et al. "Geodesic convolutional neural networks on riemannian manifolds." Proceedings of the IEEE international conference on computer vision workshops. 2015.

Monti, Federico, et al. "Geometric deep learning on graphs and manifolds using mixture model cnns." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

Cohen, Taco, et al. "Gauge equivariant convolutional networks and the icosahedral CNN." International conference on Machine learning. PMLR, 2019.

De Haan, Pim, et al. "Gauge equivariant mesh CNNs: Anisotropic convolutions on geometric graphs." arXiv preprint arXiv:2003.05425 (2020).

Bronstein, Michael M., et al. "Geometric deep learning: Grids, groups, graphs, geodesics, and gauges." arXiv preprint arXiv:2104.13478 (2021).