Latent Functional Maps: a spectral framework for representation alignment

We thank the reviewer for their feedback and suggestions. We are gonna address their questions and concerns in the following, and remain available for any additional questions.

Know the correspondence

We fully agree with the reviewer that when the correspondence is known there is no need to optimize for the functional map C. We would like to clarify however, that the correspondence between the two domains is not (fully) available in most of our experiments. In general we would like to address both the cases in which the correspondence is known (for example measuring similarity between projections of the same samples in different networks, when inputs are available) and settings in which are not as stitching and retrieval tasks.

In the experiment proposed (retrieval and stitching) our goal is to recover a good correspondence starting from very few or coarse (labels) correspondences between the domains..

We were glad to see however that the optimization recovered very good estimates of the ground truth functional maps.

Geodesic distance descriptor

We remark that the geodesic distance descriptor on the graph is computed just w.r.t to the available correspondences (anchors) on each node of the graph. Therefore the descriptor just encodes the anchors correspondence as distance functions, which is instead provided explicitly to fit the transformation in [22]. We chose to employ the geodesic distance descriptor as it is agnostic to the metric used to build the graph (in this case cosine distance). Therefore the geodesics (shortest paths) on the graph will be a discretization of the distance metric used to build the knn graph, and are expressed just w.r.t. to the anchors.We clarified this in the paper. In general we explored different strategies for the choice of descriptor functions on top of the KNN graph, and choose the geodesic distance on the graph as it is agnostic of the choice of the metric used to construct the KNN graph and provides a good choice in terms of performance. We included an ablation experiment on the choice of descriptor in the PDF of the general answer.

Experiments and baselines

We included many additional experiments in the general answers on diverse and large scale datasets. In Figure 1 we test stitching on diverse datasets (ImageNet, MNIST, AGNews, DBPedia14, CUB) to validate the performance of LFM on multiple settings, including large scale, complex, datasets (Imagenet).

Partiality limitation

We agree that the partiality case must be still explored and will include it in the limitation section.

Notation

We fixed the notation in the introduction. We thank the reviewer for spotting this.

We thank the reviewer for their feedback and suggestions. We are gonna address their questions and concerns in the following, and remain available for any additional questions.

Comparison with [29]

The method in [29] is one of many recent works [13,18, 20,22, 25, 27] that focus on the emergence of similar representation independently to either measure similarity or solving for a map (implicitly or explicitly) between the two domains. The conceptual differences with [29] are in:

• Solving the alignment problem by optimizing for a map between the eigenbasis of the two domains (functional map). Therefore our framework optimizes for a map between the spaces, while [29] maps both spaces independently to a shared relative space and the mapping is implicit.
• By employing cosine similarity, [29] assumes that the transformation is a global rotation of the space plus a local rescaling, so that cosine similarity is invariant to the mapping. This means that when there is a more complex transformation [29] will not be able to handle it well, while a functional map can in principle handle complex non linear transformations by expressing the map as a linear sparse matrix in the spectral domain.
• Our framework comprises at the same time (i) an interpretable similarity measure, (ii) a way to find correspondences, and (iii) solving for an explicit mapping between different spaces in a single framework, differently from [29] which attempts to solve just the latter implicitly. We included a discussion in the manuscript. In terms of qualitative and quantitative performance we added experiments in Table 2 comparing with [29] in the general Answer, demonstrating better qualitative and quantitative performance.

Descriptors

In general in our framework descriptor functions can either be semi supervised (e.g. express on each node of the ) weakly supervised (use the indicator functions of labels ) or fully unsupervised ( unsupervised descriptors of the graph, such as the heat kernel signature[c]).

Neural models, compared to 3D shapes, involve many quantities that can be assumed to be preserved across latent spaces, as indicated in [13, 29, 22, b]. Labels are a clear example, but unsupervised properties related to distributions in latent spaces can also be preserved. For instance, in models that assume a prior distribution in latent spaces, such as Variational Autoencoders and derived models, specific quantities can be conserved and exploited to develop useful descriptors.

We agree that designing better unsupervised descriptors tailored to the data and task at hand is an intriguing direction. This is a key focus area for our future work. We report an ablation study on the choice of descriptors in the General answer.

[a] Sun, J. et al.. A concise and provably informative multi‐scale signature based on heat diffusion. In Computer graphics forum [b]Damodaran, B.et al.. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation.ECCV

Figure 2

We adjusted the colorbars in Figure 2 for improved visualization.

Baselines

We added a comparison with different baselines [18,25,22] on the CUB dataset on a retrieval task in the general answer. With this experiment, we remark that LFM can be used on top of any off-the shelf method as [18, 22, 25, 27, 20, 13].

We did compare with the method in [22], referred to as “ortho” in Figures 3 and 5 of the manuscript, and in the additional experiments of the general answer. In the main paper we chose to compare mostly with it as it holds the state of art in the stitching and retrieval benchmarks considered. With respect to [18,25,22,17] the assumptions are pretty similar, in that they assume a linear mapping and optimize for it. We stress that with respect to methods like [13], that compute a similarity kernel between latent spaces without solving for any mapping, can be easily incorporated in our framework as descriptors of the graph nodes.

Regarding the visualization of stitched autoencoder reconstructions, we have added an experiment using the MNIST, FashionMNIST, and CIFAR-10 datasets. The results, presented in the attached PDF in the general answer section, demonstrate superior reconstruction quality w.r.t. [22].

Clarity on Figure 3 and 5

We improved the experimental section stating better the goal and takeaways from each experiment. We will add the update in the camera ready.

Functional Map visualization

We added functional maps visualization for experiment in the pdf attached in the general answer in two experiments. In Figure, we conduct a synthetic experiment by perturbing the space with simple transformations, such as rotation. We observe that the map remains perfectly diagonal when recovering an isometry of the space. Additionally, we note that it preserves a diagonal-like structure even when noise is added. In Figure we show functional maps visualizations in real case scenarios.

Comparison with Ortho [22] clarification

In both Figures 3 and 5 the purpose of comparing between Ortho and LFM + Ortho is to show how our framework can expand the set of correspondences from very few (e.g 3.) to many one and outperform the direct estimation of the Orthogonal mapping by just using the same initial information. We choose to fit the same orthogonal transformation on top of the correspondence found by LFM to make the comparison with [22] the fairest possible, although in principle any off-the-shelf methods could be used to estimate the transformation once the new correspondence is found.

Variance fine labeling

The fine labels case is more complex than the coarse label one, as the latter correspond to assign the superclass information to each sample, therefore resulting in having less classes. Therefore estimating the map can be more difficult in some settings. We observed this especially when trying to match self supervised vision models to classification based ones (i.e. DINO vs ViT).

We thank again the Reviewer for their feedback.

We hope that our rebuttal has adequately addressed their concerns, and we kindly request for feedback on this, particularly in light of the positive responses to the rebuttal from the other Reviewers: 5Rrs, yAi6 and neES.

We remain available to clarify any further questions or concerns during the remainder of the discussion period.

I thank the authors for the rebuttal. Broadly, I am ok with the response and mostly like what I see (Figure 2,4 and Table 2 in the response pdf). I am still concerned with the descriptor problem and, it appears one needs good concrete landmarks to get a good FMap. It is unclear how to generally make such descriptors for representation spaces - fully unsupervised as rightfully pointed out.

I would still critique the authors for a less favorable presentation (even in the rebuttal pdf, there are no captions for each figure and the fonts are too small - making it rather cumbersome to see what issue was being resolved with these additional experiments). Nevertheless, this is an interesting paper that could be compiled much better for long-term impact. Although I am not 100% convinced, I think I can live with a borderline accept score.

We thank the reviewer for their feedback and suggestions. We are gonna address their questions and concerns in the following and remain available for any additional questions.

Theoretical contribution We would like to respectfully point out that the goal of the paper is to make aware of the representation learning community of the functional map framework, which can have many impactful applications (retrieval, stitching) that were not primarily explored in the geometry processing setting. From the theoretical perspective the dimensionality of the problem is very different.

According to the NeurIPS submission guidelines, combining established methods to address new problems is a valuable contribution. Applying well-studied tools with theoretical guarantees can often lead to significant advancements and insights. In our study, we applied the Functional Map framework with minor adaptations, specifically in graph building within the latent space. Surprisingly, this approach already yielded considerable improvements over previous methods, demonstrating its practical efficacy and potential. We believe these findings are noteworthy and beneficial to the community. Furthermore, our results pave the way for future research, where we plan to explore more tailored modifications of the method.

Self-containment

We improved clarity in the paper, in particular: line 152: we explained and reported the definition of the operators in Appendix A of the paper in the original submission. We made the link clearer adding references to the main text and details in Appendix. We also incorporated part of it in the main paper to improve the flow of the writing. line 158: We reported an in-detail explanation of the zoom-out iterative refinement strategy of [24] in Appendix A. In the main text, we briefly described how the algorithm works and referenced the detailed explanation in the Appendix. line 235: We added a self-contained explanation for all experiments, including the definition of zero-shot stitching, which, for clarity, we report below. line 185: Adding an in more detailed explanation of the [29], we hope the concept of functional “relative” space is now more clear.

LFM similarity score

We regard the LFM similarity score as a pseudometric: concerning symmetry, as the functional map commutes with the laplacian eigenbases of the two spaces (when optimizing for it, this is enforced by the regularizer in equation 3 in appendix A) . This enforces that the inverse of the functional map will correspond to its transpose. The formula for the similarity is invariant to the transpose operator, making the score symmetric. Concerning triangle inequality we believe that the score does not satisfy this property as distinct perturbations applied to space may have similar scores. These properties are discussed in the manuscript, and we have included the formal proofs in the Appendix for clarity and completeness.

Descriptor Operator

As defined in Appendix A, the descriptor operator expresses the set of descriptor functions \mathbf{F}{G_X} (likewise \mathbf{F}{G_Y} ) in eq 2 as an operator in the spectral basis of the graph. The intuition behind the commutativity constraint in the regularizer eq3 of Appendix A, is for indicator functions of regions of constant values of each pair of descriptors f^{G_X}_i, f^{G_Y}_i to be preserved. We expanded the explanation of the definition, and made the reference to the appendix and the text in the main clearer.

Stitching

We define a generic stitched model as the composition of an encoder model that embeds data with a decoder specialized in a downstream task (classification, reconstruction). The stitching operation is always performed without training or fine-tuning of the encoder/decoder, in a zero-shot fashion. In [29], the notion of zero-shot stitching different neural components was introduced, treating them as frozen black-box modules. Nevertheless we remark that [29] still needed to train a decoder module once in order to process relative representations, before being able to perform stitching, while our method is fully zero-shot.

Comparison with [29]

We compare with [29] in the retrieval experiment in Figure 5 of the paper. We added further comparison in the retrieval experiment on the CUB dataset reported in the general answer for this rebuttal. In the stitching experiments, we do not report comparison with [29] for two reasons: (i) [29] requires training decoder modules once before performing stitching, while we directly estimate the map between the two latent spaces, without requiring any additional training; (ii) We compare with [22] (under the name of ortho), however, whose performance is superior to the one of [29], as reported in the experimental section in [22]. By demonstrating better performance w.r.t. [22] our method is therefore superior to [29].

Choice of descriptor operators

In general in our framework descriptor functions can either be semi supervised (e.g. express on each node of the ) weakly supervised (use the indicator functions of labels ) or fully unsupervised ( unsupervised descriptors of the graph, such as the heat kernel signature). We explored different strategies for the choice of descriptor functions on top of the KNN graph, and chose the geodesic distance on the graph as it is agnostic of the choice of the metric used to construct the KNN graph and provides a good choice in terms of performance. We included an ablation experiment on the choice of descriptor in the PDF of the general answer. We clarified this in the main manuscript and reported the ablation experiment in the Appendix.

Correspondence between domains

In general in our framework we don’t assume to have access to a direct correspondence between the 2 domains, and we can generalize to setting when this information is accessible for some samples (semi supervised) or not available at all (unsupervised).

We thank again the Reviewer for their feedback.

We hope that our rebuttal has adequately addressed their concerns, and we kindly request for feedback on this, particularly in light of the positive responses to the rebuttal from Reviewers 5Rrs, yAi6.

We remain available for any further questions or concerns during the remainder of the discussion period.

We thank the reviewer for their feedback and suggestions. We are gonna address their questions and concerns in the following and remain available for every additional question.

Writing quality

We have significantly improved the manuscript's clarity, making it self-contained and clear. In particular: We structured the experimental section by adding details about the settings in the experimental setting and adding a smaller subsection depicting the experiment's goal and takeaway. In the experimental section, we added an in-detail description for [29] and [22], making it clear that we are in the most similar setting to have a fair comparison with them. We added more details in section 3 and section 4.2, also adding more information in the appendix, complementary to the one already present in Appendix A, to make everything self contained and avoid pointing the reader to past work. Although we thank and agree with the Reviewer that our manuscript is clearer right now, we would like to respectfully point out that Reviewer yAi6 highlighted writing clarity, with confidence 5.

Limited Experiments

We included multiple new experiments in the paper with the objective of: (i) Validating our stitching performance on multiple datasets (CUB, ImageNet,MNIST, AgNews), including suggested datasets and datasets that lead to more complex latent spaces such as Imagenet; (ii) validating our retrieval performance on multiple methods, showcasing the applicability of LFM to state-of-the-art methods; (iii) perform experiments on multiple modalities to show how the LFM can be applied successfully to different representations. The experiments and their analysis are reported in the attached PDF in the general response.

ImageNet performance

As detailed in the general answer, we show stitching experiments on Imagenet, validating the usefulness of the LFM despite the dataset being complex. We would like to respectfully point out that the performance is also dependent on the architecture, for which we employed pre-trained state-of-the-art vision models for all the datasets. The models have been pre-trained with different objectives (classification, self-supervision) and on multiple modalities (image, text) and can handle complex and large-scale vision and text datasets.