Active Perception for Grasp Detection via Neural Graspness Field

Response to Reviewer DbvE

Thanks for your valuable feedback. We understand your concerns about the goal of active grasp detection, the performance gain of our method and some other details. We address your concerns below:

Q1: Meaning of NBV for grasp perception.

While finding one feasible grasp pose allows a robot to pick up objects, grasp detection often serves as a part of semantically diverse downstream manipulation tasks [1, 2]. In such cases, it is crucial to provide diverse feasible grasp poses in the scene. For instance, grasping the cap area of a water bottle does not enable the pouring task. Similarly, when tidying up a table, the robot must select the grasping pose based on the object's placement pose. Therefore, it is reasonable for active perception to set the goal of grasp detection as finding all feasible grasps in the scene.

Q2: Performance gain of our method.

To evaluate the performance for active perception, we consider the grasp detection results with different perception steps, where our method totally show significantly superior results compared to other active perception method given same steps. As shown in Figure 4 of our original paper, our method consistently outperforms other NBV strategies by at least 1 AP starting from the 6th step across the seen, similar, and novel sets. In the ablation study of neural graspness sampling illustrated in Figure 5 of our original paper, the incorporation of this sampling method demonstrates significant improvements across almost all planning steps. Furthermore, Table 1 of our original paper presents the results of different active grasp detection methods under a 10-step planning scenario. Our method shows notable improvements compared to the previous state-of-the-art ACE-NBV, with increases of 8.38, 6.71, and 2.98 AP on the seen, similar, and novel sets, respectively. In the context of the Graspnet-1billion benchmark, these improvements can be considered substantial.

Q3: Problem about the process of NGF optimization.

In the optimization process of NGF, there indeed exists inconsistent graspness results for each observed view, which can be primarily controlled through two aspects: (1) During the optimization process, both the new and previously observed views are jointly involved. This approach fully utilizes multi-view information and promotes consistency in graspness across different views given the same spatial location. (2) According to previous research [3], neural representations inherently possess the capability to reconstruct relatively smooth distributions from sparse and noisy multi-view information.

As for the depth error in the unseen views, becuase the camera moves continuously in small distance for each step, it is relative small in most cases and won't influence the calculation of the information gain. Figure1 in the rebuttal PDF also illustrate the effectiveness of using the rendered depth for the calculation of grasp information gain.

Q4: Explaination about Section3.4.

We apologize for any confusion. In Section 3.4, we introduce neural graspness sampling, which directly queries the graspness from NGF rather than predicting it from a grasp detection network. Following previous work [4], we incorporate FPS (Farthest Point Sampling) for two reasons: (1) To ensure that grasp positions are distributed throughout the entire scene. (2)To control the number of generated grasp poses, as there can be numerous positions with high graspness values.

The graspness scores are generated by the neural field, and we generate grasp poses at positions where the graspness score exceeds a threshold.

Q5: Missing of the colorbar.

Thanks for your suggestion. In Figure 3 and Figure 8 of the original paper, blue means low graspness value while yellow represents high value. We will add it in the next version.

Q6: Deviation between NGF and predicted graspness.

During the training of NGF, the rendering process of graspness is similar to the color. We sample rays $r \in R$ from the given view and use L2 loss to calculate the deviation, which can be formulated as:

$$L_g = \frac{1}{|R|}\sum_{r\in R}(\hat{g}(r)-g(r))^2$$

where $\hat{g}(r)$ is the graspness rendered by NGF and $g(r)$ is the predicted graspness. During NBV planning, we employ Equation (6) in the original paper to calculate the deviation as the information gain.

References

[1] Learning task-oriented grasping for tool manipulation from simulated self-supervision, Fang et al., IJRR2020

[2] Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping, Rashid et al., CoRL 2023

[3] In-Place Scene Labelling and Understanding with Implicit Scene Representation, Zhi et al., ICCV2021

[4] Graspness discovery in clutters for fast and accurate grasp detection, Wang et al., ICCV2021

Response to Reviewer 4AgP

Thanks for your valuable feedback and we address your concerns below:

Q1: Video recording of real-world experiments.

Thanks for your suggestion. In Figure 3 of the rebuttal PDF, we provide keyframe screenshots of one execution, including the active perception part and robotic grasping part. The videos will also be attached then.

Q2: Experiments on data captured from Kinect camera.

Thanks for your suggestion. We conduct additional experiments with the Kinect camera and make comparison with the previous SOTA method (i.e. ACE-NBV), as shown in Figure 5 of the rebuttal PDF, demonstrating the effectiveness of our method under different cameras. Due to time and page limitation of the rebuttal, we only finish the experiments on the seen set and will supplement the results in the camer-ready version.

Q3: Analysis experiments on hyperparameters, such as the maximum NBV step.

In fact, we have presented the results with a varying number of steps for both the NBV policy comparison (Figure 4 in the original paper) and the Neural Graspness Sampling experiments (Figure 5 in the original paper). The horizontal axis denotes the number of planning steps. As the complete planning policy greedily chooses the view yielding the highest information gain at each step, the results with 7 or fewer steps can be directly inferred from the graph. Regarding the other hyperparameters, such as the number of iterations for NGF, we emprically know that they do not substantially influence the performance of our method. Anyway, we will improve our experiments on hyperparameter sensitivity in the next version.

Q4: More detials about the real-world experiment setting.

In this setting, we consider 5 scenes, each containing 5 objects, and for each scene, we repeat the experiment 3 times by changing the poses of the objects within the scene. We provide the detail recording of the real-world experiment in the table below.

Although we use a relatively small number of objects (5) to construct each scene, the objects in the scene are arranged in a crowded manner, with the presence of view occlusions. Therefore, our scene setting reflects the characteristics of cluttered scenes. In the rebuttal PDF, we provide the object setting of the real-world experiment scenarios in Figure 4.

Q5: Differences between the trajectories generated by Close-loop NBV and ours .

Close-loop NBV is a heuristic active perception method based on TSDF Fusion that calculates the number of potentially unobserved voxels as information gain without involving a training process. However, this type of methods do not directly relate to grasp detection, as the view selection primarily depends on the overlap between the candidate views and the observed region. Therefore, the trajectory generated by the Close-loop NBV planner tends to uniformly surround the scene to maximize the observation area. More examples are shown in Figure 6 of the rebuttal PDF.

Q6: Problem about Equation (3).

We apologize for the missing closing parenthesis and any confusion caused by Equation (3). Regarding the calculation for ray color $\hat{c}(r)$, depth $\hat{d}(r)$, and graspness $\hat{g}(r)$, the NGF follows the previous NeRF-SLAM mapping method [1], where only the 3D location is involved. However, unlike the vanilla NeRF, which directly decodes the color and depth from a single MLP, the mapping framework that we employ first queries the position-corresponding feature from the learnable axis-aligned feature planes and uses separate MLPs to decode the color, depth, and graspness. This approach can significantly enhance the convergence speed. Although not introducing view directions when rendering RGB may lead to a slight decrease in quality, we typically only focus on depth and graspness for grasp detection. Therefore, the mapping system based on the NeRF-SLAM framework is suitable for NGF.

Q7: Meaning of "All views" in Table 1.

In Table 1 of the original paper, "All views" refers to using all the 256 views captured for a scene in the Graspnet-1Billion dataset to perform a complete reconstruction of the scene and then infer the grasping results. This result serves as an upper-bound reference for active perception methods.

Q8: Ablation study about Neural Graspness Sampling.

We would like to clarify although the definition of graspness is proposed in GSNet, our method significantly differs in the way of graspness generation and achieves better results. In GSNet, a grasp detection network is used to infer the graspness value of each point from the input point cloud, which depends on the quality of the point cloud and does not effectively utilize multi-view information. Our proposed neural graspness sampling directly queries the graspness value by the position from the NGF, which fully leverages the NGF's capability to accurately model the scene's grasp distribution, leading to improved results. Therefore, it is essential to conduct an ablation study on neural graspness sampling to validate its contribution. For other parts of the NBV planner, we will strive to provide ablation studies in the next version.

References

[1] ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields, Johari et al., CVPR 2023

Response to Reviewer peVt

Thanks for your valuable feedback. We understand your concerns regarding the computation of information gain with NGP and the other issues you have raised. We address your concerns below:

Q1: Incremental training approach of the NGF.

In the initial stage of NGF, there are indeed errors in both the rendered depth $\hat{d}$ and graspness $\hat{g}$ but have few influence on the calculation of information gain. Our inconsistency-guided NBV policy adopts the pseudo-label paradigm by substituting the ground-truth grasp distribution $g(d)$ with the pseudo-graspness $g(\hat{d})$, which is widely employed in other semi-supervised and active learning vision tasks [1,2]. The effectiveness of our NBV policy relies on the premise that the $g(\hat{d})$ predicted by the graspness network is closer to the ground-truth $g(d)$ compared to $\hat{g}$. The smaller error in $g(\hat{d})$ can be attributed to two factors: (1) The robot-mounted camera moves continuously in small steps, resulting in minimal differences in the observed geometric information between views, leading to insignificant errors in the rendered depth $\hat{d}$. (2) The graspness prediction network, trained on a large dataset of real point clouds, inherently provides robustness to depth noise. As a supplementary, we visualize the rendered graspness error E_\hat{g} = |g(d)-\hat{g}| and the pseudo-graspness error $E_{g(\hat{d})} = |g(d)-g(\hat{d})|$ in Figure 1 of the rebuttal PDF, where E_\hat{g} is significantly larger than $E_{g(\hat{d})}$ and the difference decreases with more steps.

Q2: The maximum achievable performance of our method.

We are not certain the definition of "single-shot":

If "single-shot" refers to using a single-view depth map for grasp detection, our active perception method aims to reconstruct the scene geometry and grasping representation with as few steps as possible. The scene point-cloud reconstructed by active perception is used for final grasp poses prediction,, resulting in significantly better performance.

If "single-shot" refers to not using neural graspness sampling, it's important to note that while the grasping knowledge of NGF originates from the teacher network's output, NGF enhances the multi-view consistency and smoothness of the graspness distribution through implicit multi-view fusion. This is particularly beneficial when the single-view output is noisy and views are sparse, as demonstrated in similar scenarios such as 3D segmentation [3]. Therefore, sampling grasping positions from NGF yields better results compared to directly sampling from the network's graspness prediction.

Q3: The inference pipeline with neural graspness sampling.

We are sorry for the confusion caused. To clarify, we still rely on a grasp detection network to perform inference on the reconstructed scene point-cloud after active perception, rather than directly sampling grasp poses from NGF. For neural graspness sampling, we replace the graspness output of the grasp detection network with values obtained from NGF based on the position, while parameters such as rotation and gripper width are still inferred using the grasp detection network.

Q4: Visualization in Figure 3 in original paper.

In robotic grasping scenarios, it is difficult to divide the objects and background in 3D space from a single view perception. Therefore, given a candidate view, the graspness values of both foreground and background regions are used to compute the information gain. For our methods, not all selected views with high information gain arise from background errors. We provide more visualization results in the Figure 2 of rebuttal PDF for illustration, where we use yellow rectangle box to highlight the forground inconsistency region.

Q5: The mapping time for our framework.

The mapping method is an extension of an efficient dense SLAM method [4] (citation [12] in our paper) that exploits axis-aligned feature planes and implicit Truncated Signed Distance Fields to reduce the number of training iterations, whereas previous dense neural mapping methods require more iterations for convergence.

Q6: Explaination about Equation(6).

We are sorry for the confusion caused. $g(r)$ means the rendered graspness of sampled ray $r$ and the summation symbol means the rendered graspness of the whole view. The definition follows Equation (3) in our paper and we will clarify it in the camera-ready version.

References

[1] Rethinking pseudo labels for semi-supervised object detection, Li et al., AAAI2022.

[2] Learning From Synthetic Images via Active Pseudo-Labeling, Song et al., TIP2020

[3] In-Place Scene Labelling and Understanding with Implicit Scene Representation, Zhi et al., ICCV2021.

[4] ESLAM: efficient dense SLAM system based on hybrid representation of signed distance fields, Johari et al., CVPR2023.

I appreciate the authors' response, which addresses most of my concerns and helps me understand this work better. Here are some of my follow-up questions:

1. Since you already have the observed depth map, why do you use $g(\hat{d})$ to guide the training of $\phi_g(p)$ rather than $g(d)$?

2. For Q2, 'single-shot' refers to the first case mentioned above by the authors. It seems the single-shot method [1] already achieves better APs compared to the results reported in Table 1. Could the authors elaborate on the advantages of the proposed method compared to [1]?

[1] Wang, Chenxi, et al. "Graspness discovery in clutters for fast and accurate grasp detection." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

Response to Reviewer DQft

Thanks for your valuable feedback. We appreciate your acknowledgment of our active perception method based on neural graspness field and the performance improvements achieved. We address your concerns below:

Q1: The relative weak improvement on novel set.

In our experiments, the improvement of the proposed method compared to other NBV strategies is less pronounced on the novel set than on the seen and similar sets. We attribute this to several factors: (1) The grasp detection model for inference demonstrates relatively lower performance on the novel set compared to the seen and similar sets. This inherent limitation constrains the potential improvements through active perception. (2) The graspness prediction network we employed has not been exposed to objects from the novel set during training. Consequently, the view graspness prediction used for training the NGF may lack accuracy for novel objects. This potential inaccuracy in graspness prediction may impact the performance of our active perception approach.

Our method also experiences a performance drop between step 3 and 4 in Figure 4 (c) of the original paper. We attribute this primarily to the grasp detection model's suboptimal performance on the novel set, which leads to a scarcity of positive grasp samples and makes the final results unstable. As the number of view gradually increases, our active perception method still achieves stable performance improvement on novel set.

Q2: Generalization and distraction experiments.

Thanks for your suggestion. In future work, we will strive to supplement our study with ablation experiments on different modules and hyperparameters of our method to demonstrate its robustness and reliability.