We express our gratitude to the reviewer for their valuable feedback on our manuscript. Here is our detailed response to the concerns raised. We believe that we have addressed all the major concerns that the reviewer had. We are happy to address more concerns given a score of 3.

Weakness and Response

Weakness 1:

The paper claims that the proposed inductive bias leads to high sample efficiency, but there is no direct evidence in the experiments.

Response 1:

We respectfully disagree with the point raised. As shown in Tables 1 and 2, our proposed LEG-UNET and LEG-Cliport models outperform four strong baselines in 25 out of 26 testing scenarios, trained with either 1 or 10 demonstrations. Notably, our models trained with just a single demonstration can surpass the baselines trained with 10 demonstrations in the 'pick-tool' and 'pick-toy' tasks. Furthermore, they achieve a success rate above 90% in 6 out of 13 tasks. It is important to note that we trained our models exclusively on 'pick-random-v1' or 'pick-random-v2' and tested them on other V-1 or V-2 tasks.

Weakness 2:

Neural Descriptor Field [1] is an SE(3) equivariant grasping method with a similar design to the proposed method. Please cite and analyze the similarities and differences between this paper.

Response 2:

While NDF is commendable work, we believe that our design is not similar to it. Firstly, our focus is on language-conditioned grasping, learning directly from pixel space, whereas NDF does not incorporate language instructions and learns from point clouds. Secondly, our approach does not assume objectness, segmentation, or categorization, in contrast to NDF, which requires a segmentation module and learns a category-level pick-and-place policy. Thirdly, our method does not require any pretraining process, while NDF necessitates pretraining to acquire point descriptors. Although both methodologies learn from demonstrations and leverage task symmetries, it is intriguing why the reviewer perceives them as similar.

Weakness 3:

A large body of recent papers (to name a few, [2-4]) in language-conditioned grasping are not included for comparison and analysis.

Response 3:

In the revised related work section, we compare our proposed method with several recent works:

1. Unidexgrasp learns dexterous grasping from point cloud observations. Its approach to language-guided grasping involves projecting grasp proposals onto images and utilizing CLIP to compute the similarity between text and images. In contrast, our work is centered on language-conditioned grasping and proposes an end-to-end learning framework that effectively leverages task symmetries.
2. Language-embedded Radiance Fields for Zero-shot Task-oriented Grasping represents a very recent development. This method integrates language embeddings into the radiance field, highlighting regions according to language instructions. However, it uses GraspNet for grasp detection, and does not involve a learning process for grasping.
3. A Joint Network for Grasp Detection Conditioned on Natural Language Commands combines the visual vectors of sampled grasp regions with language vectors using element-wise products to regress the top-down grasp pose. Unlike our approach, this network is unable to generate a comprehensive distribution across the entire action space and cannot predict multiple optimal grasp candidates when they are available.

These comparisons highlight the unique aspects and strengths of our approach in the context of the evolving landscape of language-conditioned robotic grasping. Here we provide a table (Table R3) to illustrate the key differences between these methods and ours.

Thank you for the response. The authors have addressed most of my concerns. However, I still believe that the writing of this paper makes it more difficult to follow than it should be. Based on that, I am raising my rating to 5 and lowering my confidence to 2. I am lowering my confidence because the main reason that I give a negative rating is for its writing. I'd be happy to raise my rating to 6 if this paper were reader-friendly.

We thank the reviewer for an especially detailed and thorough review. We provide a point-by-point response to the comments and questions below.

Weakness and Response

Weakness 1:

The language part and equivariant part are not directly related to equivariance itself. I think what the paper tries to claim is finding a better way to condition and ground constrained networks here. However, starting with language conditioning can mislead the readers to focus on the wrong thing.

Response 1:

The primary focus of our work is to investigate efficient representations for grounding language in language-conditioned grasping tasks. As such, we place a strong emphasis on the language-conditioned steerable kernel, namely the language grounding aspect. Key points include:

1. Language-Conditioned Policy: Our policy is language-conditioned. This means that for a given image, the action distribution changes according to the language goal.
2. SE(2) Equivariance and Dynamic Steerable Kernel: The SE(2) equivariance, as stated in Equation 2, is achieved through a dynamic steerable kernel derived from the pre-trained language embedding. The process of generating this steerable kernel from the language embedding is a learnable aspect of our model.
3. Steerable Kernel Generation: To clarify, unlike previous approaches that may rely on E2CNN[1], our network does not constrain itself to achieve equivariance through such means. Instead, we generate the steerable kernel via lifting, a process detailed and validated in Appendix A.2.3.

Our approach demonstrates a novel method of integrating language grounding with action planning in robotic tasks, leveraging the unique capabilities of dynamic steerable kernels.

Weakness 2:

No equivariant baselines that simply takes in CLIP features by concatenation. If previous equivariant grasp models are not applicable here, please explain clearly why concatenation doesn't work.

Response 2:

Excellent question. There are several key points to consider regarding the integration of language features with visual features:

1. Local Symmetry: Merely attaching the same language feature to each pixel is not sufficient to achieve local symmetry. This approach overlooks the complexity of how language features interact with spatially varying visual features.
2. Inconsistency: The direct concatenation of the language feature (trivial representation which is no change if there is a rotation) with equivariant visual feature (commonly, a regular representation which permutes if there is a rotation on the image) is not consistent.
3. Data Augmentation: Our baseline model, FCGQ, employs the UNet architecture. In this model, the hidden visual feature is concatenated with the language feature in the bottleneck layer. Moreover, the model's training involves extensive data augmentation, aligning with the variation suggested by the reviewer. The results in Table 1&2 show that our method with steerable kernels outperforms the concatenation baseline by a large margin.

Thank you for the clarification.

I have carefully read through the comments by peer reviewers as well as your rebuttal. It seems that there are still shared concerns about baselines among all reviewers. Your response 2 to my weakness 2 still seems insufficient to convince me without additional experiments.

I'd maintain my score here.

We express our gratitude to the reviewer for their valuable feedback on our manuscript. Here is our detailed response to the concerns raised:

Weakness and Response

Weakness 1:

It is rather strange to see that rotation is applied to the language embedding, but not the image space...Need an explanation on this.

Response 1:

Excellent question. The direct way to apply rotation to image space is to feed a stack of rotated images to the network. It is computationally expensive compared with the use of a set of rotated kernels or steerable kernels, especially when the number of rotations is large. The language-conditioned steerable kernel is more manageable and efficient. Theoretically, we can derive from the commutative property of the convolution action $a*b=b*a$, where a, b denotes two feature maps and $*$ denotes convolution. Further, with group action $g \in SE(2)$, $(g\cdot a)*b = a*(g\cdot b)$.

Weakness 2:

Also, CLIP language features may not the best representation of actions such as "pick", "move" etc. Since the policy is grasping only it may not impose a bottleneck as of now.

Response 2:

Good observation. We used CLIP because its text encoder is trained to align with the image distribution, which makes it a motivation for some recent visual-text tasks in robotics [1][7]. However, as mentioned in section 3.2.2, our proposed framework is suitable to fit any pre-trained language encoder or VLM. Finding a better VLM model suitable for robotics tasks could be a potential future work. Moreover, since in this work the only robot action is “grasping”, if the language-conditioned kernel can learn to extract the object type from the CLIP text embedding, it is sufficient to perform correct grasping. Our experiments successfully demonstrate such learning ability of our language-conditioned kernel.

Weakness 3:

Need some discussions on why the whole policy, not just the steerable kernel, is also equivariant under the SE2 symmetric group. SE2 also includes a translation part, but I don't see that the steerable kernel is equivariant under translation.

Response 3:

Good question. The rotation symmetry is realized by the languaged-conditioned dynamic steerable kernel. The translational symmetry is realized by the cross-correlation action, which means that convolution actions are naturally equivariant with respect to translation before adding the additional SO(2)-equivariant property. Detailed explanations can be found in Appendix A.2.2.

Weakness 4:

In the paper it is not clear if there is a pre-training phase. It is hard to believe that the network weights (of $\phi$, and $\psi$ ) are properly learned with just one trajectory if they are always randomly initialized from the beginning.

Responses 4:

We do not engage in any pre-training. Our approach solely utilizes the pre-trained CLIP model. Specifically, for LEG-UNET, we utilize only the CLIP text encoder. In the case of LEG-Cliport, both the CLIP image encoder and text encoder are employed. Details regarding our training methodology and network specifications can be found in Section 4.3 and Appendix A.3, respectively.

For few-shot learning, equivariant networks are noted for their fast convergence and enhanced sample efficiency. This has been substantiated by multiple studies, as referenced in [1], [2], [3], [4], [5], and [6].

In our analysis, we focused on the symmetry inherent in the tasks and proposed a comprehensive framework that incorporates a dynamic steerable kernel. This framework is designed to exploit task symmetry, thereby achieving improved sample efficiency. We consider this ability to leverage symmetry as a significant strength of our work, rather than a limitation.

References

[1] Zhu, Xupeng, et al. "Sample Efficient Grasp Learning Using Equivariant Models." Robotics: Science and Systems. 2022.

[2] Huang, Haojie, et al. "Equivariant Transporter Network." Robotics: Science and Systems. 2022.

[3] Jia, Mingxi, et al. "Seil: Simulation-augmented equivariant imitation learning." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

[4] Wang, Dian, Robin Walters, and Robert Platt. "SO (2)-Equivariant Reinforcement Learning." International Conference on Learning Representations. 2022.

[5] Wang, Dian, et al. "Equivariant $q$ learning in spatial action spaces." Conference on Robot Learning. PMLR, 2022.

[6] Yang, Jingyun, et al. "EquivAct: SIM (3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulation." arXiv preprint arXiv:2310.16050 (2023).

[7] Tang, Chao, et al. "Task-Oriented Grasp Prediction with Visual-Language Inputs." arXiv preprint arXiv:2302.14355 (2023).

We express our gratitude to the reviewer for their valuable feedback on our manuscript. Here is our detailed response to the concerns raised:

Weakness and Response

Weakness 1:

The mainstream of grasping research is in SE(3), but this paper remains in SE(2). This paper seems to have conducted implicit pose estimation or simply learned the rotation angles of objects. However, it appears to lack novelty.

Response 1:

We respectfully disagreed with the reviewer that this is the weakness of our method.

1. The mainstream of SE(3) grasping does not indicate the lack of novelty of all SE(2) grasping currently. Language-conditioned grasping is being actively researched but still under-explored. As the other reviewers point out, there exist very recent publications such as [1][5] on SE(2) language-conditioned grasping. Most methods rely on segmentation [1, 2], cross-attention [5], or direct concatenation of language features and visual features [3, 4, 6]. To the best of our knowledge, our work is the first method that grounds language using language-condition steerable kernels, and we show its effectiveness compared with a strong baseline [6]. We are happy to address further concerns if the reviewer can kindly provide some related state-of-the-art work on language-conditioned grasping.
2. Our method, i.e., language-conditioned dynamic kernel, is straightforward to be extended to 3D to solve SE(3) grasping using 3D kernels, which is based on irreps of SO(3), i.e., Wigner-D matrices. This change is secondary compared to our approach on integrating language-conditioning.
3. To clarify, grasping is inherently a harder task compared to pose estimation. Previous work has demonstrated the complexity of the grasping problem [7, 8]. For example, a 6-Dof pose of a bowl does not necessarily provide information about where to grasp. And, object-level pose estimation methods cannot do our pick-part task.

Weakness 2:

Real-world experiments are limited to a single task (pick-by-part) and a small set of objects. Some of the experimental objects are not sufficient to demonstrate effectiveness, as they are mostly cylindrical in shape. As long as the object is between the grippers, closing the grippers will definitely be able to grasp it.

Response 2:

First, we chose the pick-part task for our real-world experiments because it represents a challenging yet intriguing aspect of robotic grasping. This task requires the model not only to pick the object but also to differentiate specific parts of it. And, we provide three metrics and experiments with different numbers of demonstrations to show the effectiveness of our method. We believe, given the difficulty of the task and various evaluations, our physical experiments are sufficient to demonstrate the effectiveness of the real-world applicability of our method.

Second, we respectfully disagree with the cylindrical grasping argument. We have many objects that are not cylindrical. Take the bowl as an example; it is not in a cylindrical shape. The model needs to find the rim and reason the right angle of picking at a certain point on the rim. Meanwhile, simply grasping the estimated pose of the object will lead to failed grasping.

Furthermore, beyond grasping, the model also needs to select the right object given language instruction. The model not only needs to pick the object but also needs to differentiate specific parts of it.

We express our gratitude to the reviewer for their valuable feedback on our manuscript. Here is our detailed response to the concerns raised:

Weakness and Response

Weakness 1:

The author fails to compare their method against some most recent work in language conditioned grasping. For example, this work[1] also has the task of semantic grasping(grasping some object by some particular parts). Moreover, this paper only focuses on 2D grasping with language instructions but some objects may not be graspable from just a 2d top-down grasp. Some work[2] have also shown their method on 4D grasping in cluttered scene with language instructions.

Response 1:

• We appreciate the suggestion and have updated the related work section to include a comprehensive comparison with all relevant studies known to us.
• We clarify that KITE[1] is a 3D policy network, primarily focusing on aspects other than grasping. The key pixel identification mechanism in KITE aligns closely with the attention module of Cliport[3], which serves as our primary baseline. The major difference between KITE and our Cliport-UNet is KITE's use of depth images projected into point clouds for learning in 3D space, whereas our Cliport-UNet decodes pixel features into 2D grasp orientations. Additionally, we provide some key differences between our method and KITE[1] as shown in Table R1.
• We acknowledge the recent work OCID-VLG/CROG[2], which shares a similar focus. We've thoroughly analyzed the differences between our method and theirs in the updated section. We also note that this is a very recent publication lacking available code. Moreover, the 4D grasp they refer to involves top-down grasping with gripper width prediction, differing from 6 DoF grasping in 3D space.

Weakness 2:

In the real world experiment, the tested objects and variations are very limited compared to the simulation. It is more convincing to show more objects in the real world experiments and potentially show the results of the different methods not just LEG-Unet in the real world.

Response 2:

• The critique about the limited variety of objects and variations in our real-world experiments compared to the simulations has been noted. We chose the pick-part task for our real-world experiments because it represents a challenging yet intriguing aspect of robotic grasping. This task requires the model not only to pick the object but also to differentiate specific parts of it.
• We evaluated performance using three metrics: grasping success rate, accuracy in selecting the correct object, and precision in grasping the intended part. The model was tested with varying numbers of demonstrations (1,5) and on unseen objects. The consistency of our real-world experiment's performance with the simulated pick-part experiments was established.
• Additionally, our manuscript's Tables 1 and 2 compare our method's performance with baselines across 13 tasks involving hundreds of objects. Thus, we believe our physical experiment sufficiently demonstrates the real-world applicability of our method.