Thanks for your great suggestion. In the rebuttal phase, we address the reviewers' concerns about the effectiveness of our method on partial point clouds, such as those obtained from back-projecting depth map sand point clouds containing outliers. We also provide a detailed explanation of the convergence issues with the SE(3)-invariant model and highlight the advantages of using grid heatmaps over graphs.

$\color{Indigo}Q1$: The visualized results presented in this paper all feature uniform and complete point clouds. Can this method handle partial point clouds, such as those obtained from back-projecting depth maps? Can it deal with point clouds that contain severe outliers, not just those with some Gaussian noise? This is very important for practical use.

$\color{red}A$: In Figure 1(a) and (b), we present the visualization results of keypoints detected by Key-Grid on both point clouds with outlines and partial point clouds sampling from the depth map. We observe that Key-Grid identifies keypoints with strong semantic consistency, even when dealing with point clouds containing ten outliers in Figure 1(a). For the point clouds obtained from depth maps, we first generate depth maps of objects from multiple-angle photographs and then sample point clouds based on the depth maps. We find that when facing the low-quality/partial point clouds, Key-Grid identifies keypoints whose locations are similar to those detected in the complete point clouds.

$\color{Indigo}Q2$: In 289, However, if using a SPRIN module which is a SOTA SE(3)-invariant backbone to replace PointNet++ [22] directly, the training process of Key-Grid does not converge. Why? How about using other backbones? Like Vector Neurons?

$\color{red}A$: Figure 3 in our new submission material provides the training loss of Key-Grid using SPRIN and Vector Neurons instead of PointNet++ to detect keypoints on the fold pants. By examining the curves, we observe that Key-Grid does not converge by using SPRIN and Vector Neurons in the training phase. We think the SE(3) backbone model is not converging, primarily because of the stringent training conditions. Our designed loss function fails to provide sufficient information to enable the SE(3) model to converge. Like USEEK, to achieve the convergence of the SE(3) model, the authors compute the difference between features outputted by SPRIN and features from the pre-trained PointNet++ in SM as a loss function to optimize the SE(3) model. If the SE(3) model is directly used as the backbone in SM and Key-Grid, we only optimize it by locating the keypoints instead of providing a standard feature for the SE(3) model to learn from. Therefore, we think this is the main reason why the SE(3) model fails to converge.

$\color{Indigo}Q3$: This proposed representation is based on Euclidean distance between the grid point towards each line segment, right? Why not first establish a graph and then aggregate features along the graph? I saw many papers using graph to model deformable objects.

$\color{red}A$: Thank you for your valuable perspective. Currently, applying graph networks to deformable objects typically involves constructing a graph structure from point clouds, where nodes represent points on the object and edges capture mesh information between these points. Subsequently, graph neural networks input graph structures and output the required points [1]. However, the role of the grid heatmap is to use keypoints to build the skeletal structure of the deformable object, aiding the network in reconstructing the original point cloud. This process is contrary to the current application of graph neural networks for deformable objects. Therefore, we think that replacing grid heatmaps with graph structures is not suitable for our approach.

[1] Learning language-conditioned deformable object manipulation with graph dynamics. arXiv:2303.01310, 2023.

Thank you for your feedback on our paper, particularly regarding the applicability of the Key-Grid to partial and noisy point clouds. We will include this section in the next version of the paper. We greatly appreciate your consideration of the paper as ready for "weak accept". We notice that the rating is not updated. We kindly hope that you update the rating of our paper to reflect the improvements made to the paper.

Thanks for your insightful comments. We address your concerns as follows: We explain the rationale of the grid heatmap from a theoretical perspective and highlight its advantages over the skeleton structure proposed by SM. Subsequently, we demonstrate that the grid heatmap is more effective for keypoint detection than skeleton structure on deformable objects through experimental validation. Furthermore, we clarify some confusions, such as the importance of the grid heatmap to the Key-Grid and whether Key-Grid can adaptively generate keypoints.

$\color{Indigo}Q1$：My main concern is whether it can be empirically or theoretically justified why the grid heatmap can represent a dense extension of the skeleton, and why the current method of generating the grid heatmap is reasonable. Could using the density of the 'skeleton' or the density of the point cloud directly represent the grid heatmap instead?

$\color{red}A$: Skeleton structures proposed by SM does not accurately represent the skeletal structure of deformed objects. For example, in Figure 2 of our paper, if you connect the red keypoint with the black keypoint, this skeleton does not rationally represent the structure of the folded pants. Unlike SM, which directly uses the lines connecting keypoints as the object's skeleton, we characterize the overall skeletal structure of deformed objects more delicately by depicting the distances between grid points and keypoints pair line. Figure 4(C) of our paper shows the visualization results of the grid heatmaps and skeleton structures proposed by SM， denoting that the grid heatmap provides a more accurate depiction of the underlying skeletal structure of deformed objects. We also report the performance of Key-Grid which replaces the grid heatmap with the skeleton on the fold deformation in the following table. We can conclude that the grid heatmap helps Key-Grid identify keypoints with more semantic relevance on the fold deformation of clothes.

$\color{Indigo}Q2$: In the ablation study (Table 4), the accuracy drop after "No Grid Heatmap" is minimal, yet the Grid Heatmap is the most significant contribution of this paper. Could the authors explain this phenomenon and provide ablation studies across more categories?

$\color{red}A$: Key-Grid utilizes the farthest point keypoint loss to pre-select several initial positions for keypoints. Subsequently, we employ the Grid Heatmap and leverage encoder information for hierarchical decoding to assist the model in finalizing these pre-selected positions. In other words, the Grid Heatmap fundamentally aids the model in selecting keypoint positions rather than directly generate their positions. Therefore, when removing the grid heatmap, its overall impact on the model's effectiveness is less significant compared to removing the farthest point keypoint loss, which provides initial candidate keypoint positions. However, in Table 4 from our paper, “No Grid Heatmap” actually has a greater impact on reducing the semantic relevance of keypoints compared to “No Encoder Information”, which also illustrates the significant role of grid heatmap in the decoder. In the following table, we present an ablation study of more diverse categories of deformable and rigid objects under the DAS metric and also show the average impact of ablating different modules on the performance of Key-Grid.

$\color{Indigo}Q3$: Can this paper adaptively select the number of detected keypoints for different categories of objects?

$\color{red}A$: Thank you for your suggestion. Currently, in Key-Grid, we need to manually set the total number of predicted keypoints beforehand. In future research, we will propose an adaptive keypoint detection method that can generate varying numbers of keypoints with accurate positions for different samples.

$\color{Indigo}Q4$: The quality of figures 1 and 2 could be enhanced. For example, the information content of figure 1 is too minimal to serve as a teaser.

$\color{red}A$: Thanks for your advice. In the come ready version, we will submit the new version of Figure 1 and Figure 2. The new version of Figure 1 not only provides the visualizations of keypoints detected on both deformable and rigid objects but also adds the grid heatmap and skeleton structure visualizations like Figure 4(c) in our article, which demonstrates that grid heatmap can more accurately represent the geometric information of the object.

Thank you for your valuable feedback. We address your concerns as follows: we provide demonstrations of our method on some less successful objects and investigate whether keypoints maintain semantic consistency across various deformations of objects.

$\color{Indigo} Q1$：A small neat-picking addition would be to add one or two visual examples where the method does not work super well, to contrast with the excellent results presented in the paper

$\color{red}A$: We present the visual results of the drag deformation for 'Mask' and 'Skirt' in Figure 4 of our submission material, which exhibit lower DAS values than other categories on the ClothesNet dataset. Although the semantic consistency of keypoints detected on Masks and Skirts is relatively lower than other categories, keypoints identified by Key-Grid uniformly distribute across the object's surface, located at positions rich in geometric information compared to other methods. For instance, keypoints detected by Key-Grid are evenly distributed along the straps of the mask.

$\color{Indigo} Q2$：Perhaps I missed this point, but how does the author ensure that the key points are semantically the same across different folding/deformation of the objects?

$\color{red}A$: Figure 2 of our submission material shows the visualization results of keypoints detected by Key-Grid on long pants under the dropping, pulling, and dragging deformation. We can observe that keypoints detected by Key-Grid keep great semantic consistency during this long-term deformation process. At the same time, we observe that keypoints identified by Key-Grid are uniformly distributed across deformable objects, with their positions containing substantial geometric information, such as the hem and waist of long pants.

Thank you for providing thoughtful and detailed feedback, which greatly enhances the quality of our article. Based on your comments regarding related work, clarity, and grammar & style nits, we will revise our paper in the came ready section. Regarding your inquiries about experimental results and significance sections, we address each one individually. Additionally, we provide the new visualization results of keypoint detected by Key-Grid on the SUN3D dataset in the new submission material.

$\color{Indigo}Q1$: Compared to SM, is the keypoint heatmap the main difference, how about the use of hierarchical decoder.

$\color{red}A$: The main distinction between SM and Key-Grid lies in the decoder design. We incorporate information from the encoder into the decoder to reconstruct point clouds through hierarchical decoding. Additionally, we integrate grid heatmap information during the reconstruction process. Therefore, we consider the hierarchical decoder and the grid heatmap as the two significant innovations of our method. In our paper, we conduct ablation experiments on the hierarchical decoder in Section 4.5. In Table 4 from our paper, the label “No Encoder Information” means the ablation of the hierarchical decoder. We observe that when we do not use a hierarchical decoder, our method's performance will decrease on both the deformable and rigid objects.

$\color{Indigo}Q2$: Wouldn't a richer shape descriptor heatmap do even better than a max function? (e.g. K channels, distances to the keypoints, etc).

$\color{red}A$: In the following table, we display the impact of grid heatmaps with different distance definitions on keypoint recognition in deformable objects. These definitions are the K-channel distances from each grid point to all segments or keypoints. For the folding deformation, the distance defined in our paper achieves great performance on the semantic consistency of keypoints. We think that adopting a K-channel distance will result in some overlap of distance information between the point clouds outside the deformable objects and those inside. This condition reduces the ability of the grid heatmap to depict the structure of the deformable objects.

$\color{Indigo}Q3$: How do you get keypoint correspondences across shapes for your model, do they emerge naturally?

$\color{red}A$: The positions of keypoints are outputted in an orderly manner by Key-Grid and other baselines(SC3K, SM, and KD). In our article's visualization of keypoints, we use different colors to represent keypoints outputted in different orders. During the entire deformation process, if keypoints of the same color (with the same output order) maintain their positions unchanged, we refer to these keypoints as having good semantic consistency. The output of keypoints with good semantic consistency can trace various positions that contain essential geometric information in objects when objects undergo deformation. This is crucial for robot manipulation of deformable objects and understanding the deformation process of objects. In our paper, Key-Grid naturally outputs keypoints with semantic consistency.

$\color{Indigo}Q4$: What are the method results on KeypointNet dataset? Can you provide some additional comparisons to supervised baselines on your datasets, if there are such?

$\color{red}A$: Thank you for pointing out this issue. In the following table, we present various supervised networks and self-supervised methods to compare Key-Grid on the KeypointNet dataset based on mIoU metric. Several standard networks, PointNet [1], SpiderCNN [2], and PointConv [3], are trained on KeypointNet to predict the probability of each point being a keypoint in a supervised manner. We can observe that compared with other self-supervised methods, Key-Grid recognizes the more accurate location of keypoint, and it performs even better than some supervised methods using PointNet and SpiderCNN as backbones.

$\color{Indigo}Q5$: Can this method be applied on datasets where shapes are significantly occluded (e.g. viewed from a specific direction)?

$\color{red}A$: Thanks for your insightful suggestion. Keypoints detected by Key-Grid estimated on the deformation of objects from side views are depicted in Figure 1 from the new submission material. Meanwhile, we also show the keypoint detection results when objects' shapes are occluded in Figure 1 from the new submission material. We find that Key-Grid exhibits robustness on the occluded point clouds and point clouds obtained from different views.

[1] Pointnet: Deep learning on point sets for 3d classification and segmentation. CVPR 2017

[2] Spidercnn: Deep learning on point sets with parameterize convolutional filters. ECCV 2018

[3] Pointconv: Deep convolutional networks on 3d point clouds. CVPR 2019

Thank you for your detailed review and comments on our paper. We will revise our paper according to your suggestions. We are also pleased to hear that you believe our paper has reached the ’weak accept‘ stage. However, we observe that the rating is not updated. We would be grateful if you could revise the rating to better reflect the improvements made to the paper.