Q1: The range of tasks and the diversity of object categories provided in the dataset are limited.

Please kindly refer to Common Question in General Response.

Q2: Line 853 mentions the total time expenditure without delving into specific details, such as the time required for rendering images, particularly the computation comsumption of the function map.

We thank the reviewer for this valuable suggestion! We have performed inference runtime benchmarking for our model accordingly. We found that for a pair of meshes that are remeshed to ~2000 vertices, computing 2D SD-DINO features for 5 views each consumes ~3.6 seconds, performing DiffusionNet forward pass for each consumes ~0.01 seconds, and computing functional map consumes ~2.2 seconds. For a pair of meshes that are remeshed to ~500 vertices, computing functional map consumes ~0.8 seconds while the other parts remain unchanged. The rendering time for 5 views depends on meshing of the original textured assets and averages to ~0.2 seconds per mesh. We reflected this change in Appendix A.4.3 and Table 5.

Q3: The paper lacks an ablation study for the DINO and SD components. Previous zero-shot methods shows that the features provided by SD VAE may not be optimal. An ablation analysis for the feature backbone should be included in the experimental tables.

We acknowledge that SD VAE features lack semantic richness, as discussed in prior works (e.g., Appendix B in DIFT [A]). However, our approach does not utilize VAE features from Stable Diffusion. Instead, we rely on features extracted from the UNet decoder, specifically layers 2, 5, and 8, as outlined in SD-DINO [B] and GeoAware-SC [C], which is shown to be most informative regarding spatial and semantic understanding. Our pipeline follows GeoAware-SC for extracting geometry-aware 2D representations and employs their pretrained feature aggregation network, which was trained jointly on both SD and DINOv2 features. Conducting an ablation study for individual feature sets would require retraining the aggregation network for each specific feature set, which is computationally infeasible. We direct the reviewer to Table 3 in SD-DINO [B], where a similar ablation study on 2D representations is presented. Their results highlight the significant performance gains achieved by fusing SD and DINO features compared to using either feature set individually.

[A] Emergent Correspondence from Image Diffusion. Tang, L., et al. NeurIPS, 2023.

[B] A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence. Zhang, J., et al. NeurIPS, 2024.

[C] Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence. Zhang, J., et al. CVPR, 2024.

Q4: There is no discussion on whether the model incorporates augmentations for the pose of the mesh. Research has shown that semantic features can easily overfit to spatial position-related scenarios. If the input mesh's position changes, the resulting semantic map may become inaccurate. Therefore, it would be beneficial to include experiments that apply random rotations to the mesh as input.

In our original experiments, we randomly rotated the mesh as a part of our training augmentation procedure. We have updated Appendix A.4.2 with this detail.

Q5: Additionally, it would be constructive to present examples of failure cases to provide a more comprehensive evaluation.

Our unsuccessful cases primarily arise from two sources: one is the inaccurate generation of poses, and the other is the imprecision of the waypoints we provide, which leads to task failure. We have uploaded a case where peeling a banana failed due to the waypoint issue to the linked website（https://densematcher.github.io/）, but this failure is unrelated to the performance of our model. Our model predominantly influences the grasping pose.

Thank you for your response. Are there any specific concerns that you feel still place the paper on the borderline for acceptance? We would be happy to discuss and address any remaining issues to further clarify or strengthen our work.

Additional notes on Q4:

In Figure 3 of the updated paper, we randomly rotate the mesh before the test procedure to ensure that the model is not reliant on canonical spatial poses. It showcases that our model is robust to random rotations during testing.

Q1: The experimental evaluation is limited to a self-contributed dataset and very few qualitative runs on a robotic application (where it is unclear if the method difference is statistically significant).

For the number of trials in real-world experiments, we follow previous work like Robo-ABC [A] and RAM [B], which is five times.

[A] Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation. Ju, Y., et al. ECCV, 2024.

[B] RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation. Kuang, Y., et al. CoRL, 2024.

Q2: The method design contains a couple of non-straightforward design choices without justifications or experimental evidence to back up these choices: Q2(1): Using the XYZ coordinates of the mesh vertices makes the method sensible to random transformations on the input mesh. There is no experiment evaluating whether the model is able to learn invariance over such random coordinate system changes.

In our original experiments, we randomly rotated the mesh as a part of our training augmentation procedure. In addition to sinusoidal-encoded XYZ coordinates, the 3D refiner network’s input contains heat kernel signature(HKS) descriptors of the scale-normalized mesh, which is invariant to scaling, rotation, and translation. We have updated Appendix A.4.2 to include this detail. Additionally, we showcase in Figure 3 that the model is robust to random rotation during test.

Q2(2): The choice of negative cosine similarity in $L_\text{semantic}$ is quite particular. The authors do not explain why they would choose this over e.g. L1 or L2 distances and also do not ablate this choice.

We appreciate the reviewer's thoughtful comments regarding our choice of the negative cosine similarity. We chose this metric because both $||f(v_i) - f(v_j)||$ and $D_\text{semantic}(v_i, v_j)$ are normalized . Under this normalization, optimizing the negative cosine similarity is mathematically equivalent to optimizing the squared $L_2$ distance, while enjoying the benefit of being more interpretable.

To elaborate, for any two normalized vectors $a$ and $b$, their cosine similarity is defined as:

$$\cos(\theta) = \frac{a \cdot b}{||a|| ||b||}.$$

Since $a$ and $b$ are normalized, $||a|| = ||b|| = 1$. The negative cosine similarity becomes:

$$-\cos(\theta) = -a \cdot b.$$

Expanding $||a - b||^2$ for normalized vectors, we have:

$$||a - b||^2 = ||a||^2 + ||b||^2 - 2a \cdot b = 2 - 2\cos(\theta).$$

Thus, minimizing $-\cos(\theta)$ is equivalent to minimizing the squared $L_2$ distance $||a - b||^2$, up to a constant factor.

In addition, since a maximal cosine similarity implies linear correlation between $||f(v_i) - f(v_j)||$ and $D_\text{semantic}(v_i, v_j)$, we added proof in Appendix A.5.2 to show that optimizing the functional map objective is equivalent to optimizing total $D_\text{semantic}(v_i, v_j)$ beteween matched vertices, and referred to it in Section 4.3.1.

Q2(3): Similarly, for $L_\text{preservation}$, the choice of a single linear layer for reconstruction might hinder the encoder network to learn a more useful non-linear function. The more standard choice would probably be to mirror the encoder architecture like in an autoencoder, but this is neither discussed nor evaluated.

We thank the reviewer for this insightful suggestion. We have taken the suggestion and performed experiments with 3 variants of the reconstructor module, where each variant's parameters are optimized together with the model during training. The three variants include (i) a linear layer (ii) a 4-layer MLP, corresponding to the depth of our 3D refiner (iii) a DiffusionNet mirroring the architecture of our 3D refiner. The results are presented below.

We found using reconstructor variant (iii) resulted in poor training loss convergence. We surmise this is because DiffusionNet simulates the heat diffusion process to propagates features on the mesh surface, and attempting to reverse this process forces the diffusion time constant to be near-zero, causing numerical instability.

We observe slightly better performance on benchmark albeit higher training loss when using a linear reconstructor layer compared to a 4-layer MLP. We suspect that this is due to the more powerful MLP reconstructor being prone to overfitting, thus "cheating" the information preservation problem.

Q3: The method requires supervised training with an expensive 3D annotation workflow.

We asked the annotators to time their workflow and found that by annotating sparse keypoints and using heuristics to process them into dense groups, the annotation time is around 10 seconds per mesh, which is on par with labeling image keypoints in 2D. For more complex meshes such as four legged animals which require dense annotation using blender’s vertex brush, the annotation time is around 5 minutes per mesh, which is faster than coarsely-labeled dense annotations [C] on 2D images (~7 minutes per image). We have included the details in Appendix A.2.4 and A.2.5.

In addition, as explained in the paper (Appendix Sec. A.3.2), the only trainable component of our model is the lightweight(~5M parameters) 3D refiner, whose input already contains rich semantic features from frozen 2D foundation models. As a result, our method is highly data-efficient and only requires a few dozen meshes per category to work.

[C] Urban Scene Semantic Segmentation with Low-Cost Coarse Annotation. Das, A., et al. WACV, 2023.

Q4(1): Section 4.1: I am not super familiar with the prior work on 3D dense matching, but this optimization formulation seems computationally expensive and as Section 4.4 shows also unstable.

The original motivation of functional map [D] is its computational efficiency, as it decomposes a high-dimensional space of vertices into low-dimensional frequency representations. The theoretical runtime of function map is scales linearly with the number of vertices, and our GPU-accelerated implementation of functional map takes ~0.8 seconds for a pair of meshes with 500 vertices each, and ~2.2 seconds for a pair of meshes with 2000 vertices each. In contrast, other state-of-the-art 3D shape matching methods that solve for globally optimal solutions such as [E] are polynomial time and take >200 seconds for a pair of 2000-vertex meshes. We have added runtime comparisons in Appendix A.4.3 and Table 5.

[D] Functional Maps: A Flexible Representation of Maps Between Shapes. Ovsjanikov, M., et al. ACM Trans. Graph., 2012.

[E] SpiderMatch: 3D Shape Matching with Global Optimality and Geometric Consistency. Roetzer, P., et al. CVPR, 2024.

Q4(2): Why are other assignment and matching methods not compared as beaseline or ablation? e.g. Hungarian matching or the double-softmax used in LightGlue [F]?

We have added a visual comparison of functional map with Hungarian matching and nearest neighbor in Figure 11. Although the two suggested formulations are valuable, we believe they may not be the most suitable for the 3D matching problem. Many 3D objects exhibit circular symmetry, and finding point-to-point correspondences does not necessarily result in a continuous mapping between surfaces.

In addition, following the reviewer's suggestion, we benchmarked the runtime of Hungarian matching on the pairwise vertex feature distance matrix without accounting for spatial consistency, which consumes ~0.01-0.4 seconds for a pair of 500-vertex meshes, and ~0.5-2.5 seconds for a pair 2000-vertex meshes. We have added Table 5 for a straightforward view.

Finally, we have also explored setting up an optimization program that solves for a point-to-point mapping matrix using double softmax while incorporating spatial constraints such as isometry. However, the results did not converge, possibly due to the large search space dimension associated with a point-to-point matrix.

[F] LightGlue: Local Feature Matching at Light Speed. Lindenberger, P., et al. ICCV, 2023.

Q5: line 200: The requirement of textured 3D assets is very limiting. It seems to me the method could also work from an untextured geometry asset and posed images, or am I missing something?

The reviewer is correct that our method would also work from an untextured geometry asset and posed images, as long as they are consistent with each other. We trained our model on textured assets since they are easily sourced from existing datasets. In addition, for future works wishing to scale up this method, state-of-the-art methods in 3D generation such as LRM [G] can already generate high quality textured 3D assets from posed images within seconds, so the limitation is not significant.

[G] LRM: Large Reconstruction Model for Single Image to 3D. Hong, Y., et al. ICLR, 2024.

Q6: line 242: Since the negative cosine distance is such an odd choice I suspect the authors were inspired here by related work? In that case it would be important to attribute this here with a reference.

Please kindly refer to Q2(2).

Q7: line 252: "object type and material" is misleading. Neither one of the frozen backbones captures this information, both are self-supervised encoders of visual appearance that might correlate with this information in some cases.

Thank you for your insightful question. We would like to clarify that the Stable Diffusion model is not trained in a self-supervised manner but with billions of text-image pairs, which enables it to distinguish object types and materials. In practice, a line of work has shown that material information (e.g., albedo, normal, roughness, and metallic) can be extracted from the Stable Diffusion model [H, I, J, K, L] or the DINOv2 model [K, L], via (LoRA) fine-tuning [H, I, J] or feature probing [K, L]. These studies suggest that material information is present within these visual foundation models.

[H] Intrinsic Image Diffusion for Indoor Single-view Material Estimation, Kocsis et al., CVPR 2024.

[I] RGB↔X: Image Decomposition and Synthesis Using Material-and Lighting-Aware Diffusion Models, Zeng et al., SIGGRAPH 2024.

[J] MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors, Litman et al., arXiv 2024.

[K] Probing the 3D Awareness of Visual Foundation Models, El Banani et al., CVPR 2024.

[L] Generative Models: What Do They Know? Do They Know Things? Let's Find Out!, Du et al., arXiv 2023.

Q8: line 254: What norm is used in the equation for $|| \cdot ||$? Why is that one choosen? Thanks for the feedback. $|| \cdot ||$ denotes L2 norm, following the convention for feature distances. We have updated the paper to use $|| \cdot ||_2$ for clarity.

Q9: Table 1: Please explain better the different ablation variants. Is "w/o Diffusion Net" directly matching the concatenation of and the HKS features? Or is it also using the XYZ features and therefore failing because of coordinate system change?

The experiment "w/o DiffusionNet" uses only $f_\text{multiview}$ and does not concatenate it with XYZ/HKS features. This was originally mentioned in section 6.4. We have updated Table 1's caption to refer to section 6.4.

Q10：Section 6.2.3: I dont't think the comparison to Robo-ABC is entirely fair. It would be good to show both variants, with the full affordance memory and with the reduced form that is currently presented. The proposed method is very expensive in terms of the 3D data it requires, so really it needs to show that this additional information can compete with methods that are only based on cheaper and more abundant image data.

In Table 3, we have added a comparison of the two variants of DenseMatcher and Robo-ABC: one with full memory capabilities and another where Robo-ABC's affordance memory is only allowed to be collected from the corresponding human demos we provide, while keeping Robo-ABC's original retrieval-and-transfer framework intact.

From Table 3, it can be seen that the success rate of Robo-ABC with full memory capabilities is lower than that of Robo-ABC where its affordance memory is only allowed to be collected from the corresponding human demos we provide. Although the amount of data in Robo-ABC's memory is huge, the categories of objects are too different from the object categories used in our experiment. Robo-ABC will retrieve the entire memory for objects it has never seen. In fact, the simplified version of Robo-ABC reduces the error rate of the retrieval process and improves the success rate of the robot experiment. We hope to resolve the reviewer's doubts through the comprehensive experimental results.

Q11: Section 6.2.4: How is success determined in the experiments? Given the low number of overall trials, what level of statistical significance does the experiment currently have?

Our real-world deployment process is the same as Robo-ABC's, which involves obtaining grasp points on the target object and generating grasp pose at the grasp point using AnyGrasp [M]. The criteria for determining the success of robotic experiments are also the same as Robo-ABC's, which is based on whether the robot grasping is successful.

[M] AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains. Fang, H., et al. IEEE Trans. Robotics, 2023.

Thank you for the insightful feedback.

Our focus is on real-world applications, specifically using human hand demonstrations as they are more practical to collect and present greater challenges compared to simulation. These real-world scenarios better reflect whether the method is effective in practical settings. This is why we chose to supplement with additional real-world experiments during the rebuttal.

We agree that testing in simulation could provide additional validation. However, as you mentioned, it is beyond the scope of our work and what we can address within the rebuttal period. We plan to conduct simulation experiments as you suggested, such as those in the SAPIEN simulator, after the discussion phase and will include them in future work.

Q1: Performance on Varying Topologies: How does DenseMatcher perform with objects of varying topologies? Are there specific object structures or topological variations where its performance significantly degrades?

The daily object subset of our dataset contains chairs that each have 4 legs and a backrest made of planks with holes in between, animals with 4 legs, and cars that are empty inside and have holes in windows. We have added Figure 3 and Table 6 (as below) to provided more visualizations and quantitative test results for those categories.

Table: 3D correspondence performance (Error $\downarrow$) on categories with complex topologies.

Q2: Handling Severe Occlusion: Is DenseMatcher able to be adapted or extended to handle severe occlusion more effectively? What potential modifications could mitigate its reliance on multiview feature extraction and functional maps in such cases?

In the original training procedure, with a 50% probability, we sliced the mesh along random directions and removed half of it, in order to simulate occlusion. We have added this detail in our appendix along with other training augmentations. During inference, without any extra modification, our model is capable of establishing correspondences from one partial mesh to another partial mesh. As a result, in our real-world robot experiments, we did not use a multi-perspective camera setup but relied solely on a single L515 camera, which only captures the camera-facing part of the mesh. Following the review, we ran additional robotic experiments by intentionally putting obstacles that occluded the object. We have updated Figure 12 in the appendix to demonstrate that even under severe occlusions, our model can achieve successful grasps. In Figure 12, the red dots represent contact points, and the blue poses represent the poses generated at these contact points. Further, to handle cases where we need to match whole objects to partial objects, we have re-implemented a version of partial functional map [A], which co-optimizes a mask on the full mesh with the functional map itself. We have provided more explanations and visualizations in Section A.6 and Figure 13.

[A] Partial Functional Correspondence Rodolà, E., et al. Computer Graphics Forum, 2017.

Q3: More Benchmark Validation: Are there any benchmarks or experiments that could further validate DenseMatcher’s robustness against topological diversity and occlusion? How might these additional evaluations impact its overall effectiveness and applicability in real-world scenarios?

Please kindly refer to Common Question in General Response.

Q1: The statements of regularization terms in the methodology part are unclear and may cause ambiguity. In Sec 4.1 Preliminary, Functional Map, please give a detailed justification about how to regularize the term C as isometric in your context.

We appreciate the insightful feedback. We added derivations for isometric regularization of the term C in Appendix A.5.3 and refered to it in Sec 4.1. This addition aims to enhance clarity and address any potential ambiguities in our methodology. Thank you for highlighting this concern.

Q2: In the appendix, please provide a detailed explanation, with proofs, showing how previous constraint terms ensure that the output is minimized in the semantic distance function.

Thank you for your valuable input. In Appendix A.5.2, we added detailed derivations for the previous constraint terms. In addition, we showed with proofs that our proposed semantic distance function is minimized under the functional map framework.

Q3: Some experiment details, like the description for each task, can be placed in the appendix and give a more precise visualization. The images in the robotic manipulation task are too undersized.

Following your suggestion, we have reorganized the structure of the paper, placed the description of tasks in the appendix, and re-uploaded Figure 8 to provide a clearer view of the robotic experiments.