Q6: How does your method perform compared to sampling-based methods that propose plausible actions and plan over them? Have you evaluated this as a baseline?

Thank you for your suggestion! We implemented a sampling-based method as an additional baseline. Specifically, the policy samples a macro action from a discrete set at each high-level time step and then plans to the sub-goal pose associated with the selected macro action.

For example, in the open-safe task, the macro action set is {grasp the door handle, grasp the knob, rotate the knob clockwise, rotate the knob counterclockwise, pull the door}. The sub-goal pose for each macro action is computed based on the object part pose annotation.

The quantitative results for this sampling-based method are shown in the following table. This method performs worse than AdaManip and other baselines. The main reason is that the sampling process lacks efficient way in leveraging prior or posterior learned from demonstrations.

Q7: What are the limitations of your adaptive policy learning approach when applied to more complex or real-world scenarios with more diverse mechanisms?

Currently, adaptive demonstrations are collected using a human-designed rule-based policy (in simulation) or through human teleoperation (in real-world experiments). While this approach is commonly leveraged in other works (e.g., [7], which also use rule-based policies for simulation data collection and human teleoperation for real-world data collection) and has successfully facilitated adaptive policy learning, these data collection pipelines require significant human effort for each task. As the complexity and diversity of mechanisms increase, designing such pipelines for each category may become impractical. A promising direction for future work would be to explore automated or streamlined methods for data collection.

Additionally, as the task complexity increases, the policy will need to retain a longer history context to accurately identify and recover from failures. This increases both the computational cost for training and inference and may also introduce instability during training due to the challenges of long-context modeling.

[7] Wang et al, GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy, CoRL 2024

Q4: How would the method handle combinations of the five proposed manipulation mechanisms?

To demonstrate the capability in handling mechanism combinations, we show how our method tackles the open-safe task, the most complicated task in the current version of the AdaManip environment, which incorporates the combinations of three of the five mechanisms: the Lock Mechanism, Random Rotation Direction, and Switch Contact Mechanism. In this task, the robot needs to determine whether the door is locked, the rotation direction of the knob lock, and the correct contact point to reach.

To handle this task, we designed a rule-based adaptive demonstration collection policy to cover all possible failure and success states, enabling the policy to adjust its actions based on feedback. Specifically, the demonstration policy follows these steps: 1. The robot first pulls the door handle. If successful, it proceeds to open the door. 2. If pulling the handle fails, the robot switches to rotating the knob. 3. It randomly chooses a rotation direction and switches to the other direction if the first attempt fails. 4. Once the knob is successfully rotated to unlock the door, the robot pulls the handle again to complete the task.

In the future, we plan to continually introduce more categories and mechanisms into the AdaManip environment. This will include more challenging tasks that combine multiple mechanisms, increasing complexity and ambiguity. Similarly, for those tasks, we can design above-described demo collection policy accordingly or collect adaptive demo by human teleoperation, and then learn policies covering mechanism combinations using the collected data.

Q5: The method relies heavily on the quality of the collected data, and the diffusion model learns everything simultaneously, which could lead to redundancy and potential overfitting to the training data.

The training data encompasses both failure and success states encountered during object manipulation, which prevents overfitting to specific successful trajectories. Additionally, it incorporates optimal recovery actions conditioned on failures, ensuring redundancy is avoided. The results in Table 4 further validate the efficiency of our scripted policy in minimizing redundant adaptive trials.

The diffusion model, trained on the adaptive and efficient demonstrations, captures the multi-modal prior distribution and develops the ability to adjust its posterior based on feedback. This training approach prevents overfitting to static trajectories and enables the model to dynamically adapt its behavior based on historical context.

For example, in the open-safe task, the policy initially predicts multiple possibilities, such as directly opening the door or rotating the knob to unlock it. If the robot attempts to pull the handle but fails to open the door, the policy updates its posterior based on this feedback, collapsing the multi-modal prediction into a single mode—rotating the knob. By learning from adaptive demonstrations and utilizing failure feedback, the AdaManip model effectively avoids overfitting to static trajectories and redundant trials.

Thank you for your valuable review! We would like to address your concerns in the following response.

Q1: The paper's main contributions focus on introducing a new manipulation environment and demonstrating real robot experiments, but it lacks significant innovation in the underlying methodology, raising questions about its fit for this particular conference.

We would like to emphasize that the contribution of our paper includes both the introduction of the articulated object manipulation simulation and dataset, and the proposed whole learning framework for adaptive policies based on the simulation. Adaptive articulated object manipulation represents a set of realistic and frequently encountered tasks for robots, while relatively underexplored due to the few attempts to build the simulation environment and dataset supporting the adaptive policy training. As recognized by Reviewer XbhV and eEiz, our work introduces the adaptive articulated object manipulation task along with the corresponding AdaManip environment and dataset. Moreover, we design an adaptive demonstration collection and policy learning pipeline, enabling the policy to model multi-modal trajectory distributions and update its posterior based on environmental feedback.

Through this work, we aim to pave the way for further research on real-world manipulation challenges, and highlight the importance of learning the efficient adaptive manipulation policy by collecting adaptive demonstrations over optimal demonstrations.

Regarding the fit for this conference, we believe our work strongly aligns with the accepted topic of "applications to robotics, autonomy, planning" at ICLR. Furthermore, recent ICLR papers, including ManiSkill2 [1] and DaxBench [2] (notable-top-5% at ICLR 2023), have focused on proposing realistic robot simulation environments without necessarily introducing significant innovations in learning algorithms. The contributions of our work are of a similar scope, providing a valuable resource for advancing robotics research, with a further designed learning framework in this novel environment.

Q2: Why action history was chosen over observation history, which is more common in reinforcement learning research.

As we mentioned in Sec 4.2, AdaManip policy "models the conditional distribution $P(A_t|O_t,\hat{A}_t)$. Here $A_t$ refers to the predicted action sequence, $O_t$ refers to the observation history, including 3D point clouds and proprioception states, and $\hat{A}_t$ refers to action history." Therefore, the policy is conditioned on both observation history and action history.

Q3: The generalization ability of this method.

The testing is conducted on the training shapes at the category level, with variations in the object positions, poses (orientations), and joint configurations in both simulation and real-world experiments. Such setting aligns with current imitation learning studies [3, 4] using few-shot demonstrations (in our work, we use only 20 demos per shape for training).

As for the generalization towards novel shapes, another branch of current works is trying to use large-scale data [5] or foundation models [6] to achieve such generalization capability, which is orthogonal to the contribution to our work: empowering imitation learning with the adaptive manipulation capability on diverse object mechanisms with our proposed environment, mechanisms and assets.

It is a promising direction for the following work to further study how to leverage the generalizable knowledge inherited from large models to enhance the cross-shape and cross-category generalization capability of adaptive manipulation on diverse object mechanisms.

[1] Gu et al, Maniskill2: A unified benchmark for generalizable manipulation skills, ICLR 2023

[2] Chen et al, Daxbench: Benchmarking deformable object manipulation with differentiable physics, ICLR 2023

[3] Chi et al, Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023

[4] Zhao et al, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware, RSS 2023

[5] Yu et al, RT-1: Robotics Transformer for real-world control at scale, ArXiv 2022

[6] Yan et al, DNAct: Diffusion Guided Multi-Task 3D Policy Learning, ArXiv, 2024

We sincerely thank you for your thoughtful review. As the discussion period draws to a close, we would appreciate your feedback to confirm whether our replies have addressed your concerns. If you have any remaining questions, we are happy to provide further clarification. If our responses have resolved your concerns, we would be deeply grateful if you could consider raising the score. Thank you again for your time and effort during the review process.

Thank you for your insightful feedback and for highlighting the importance of our work. We hope to address your concerns in the response below.

Q1: More comparison with additional baselines, including RL and traditional planning methods.

Thank you for your suggestion! In our imitation learning setting, the adaptive policy is trained using a limited number of demonstration trajectories, which aligns with the practical setting adopted by previous methods, such as Diffusion Policy and ALOHA. Therefore, it would not be a fair comparison to evaluate against RL methods that rely on online data collection beyond offline demonstrations.

We implemented a sampling-based method as an additional planning-based baseline. Specifically, the policy samples a macro action from a discrete set at each high-level time step and then plans to the sub-goal pose associated with the selected macro action.

For example, in the open-safe task, the macro action set is {grasp the door handle, grasp the knob, rotate the knob clockwise, rotate the knob counterclockwise, pull the door}. The sub-goal pose for each macro action is computed based on the object part pose annotation.

The quantitative results for this sampling-based method are shown in the following table. This method performs worse than AdaManip and other baselines. The main reason is that the sampling process lacks efficient way in leveraging prior or posterior learned from demonstrations.

Q2: More emphasis on how AdaManip differs from environments like PartNet-Mobility in complexity or adaptability would better explain the novelty of AdaManip. For example, you could talk about whether there are particular types of mechanisms or object categories that AdaManip includes that PartNet-Mobility does not.

Previous articulated object environments do not simulate the complex mechanisms implemented in AdaManip. For example, in PartNet-Mobility, opening a microwave door involves directly pulling it open. In contrast, our environment requires the robot to first push a button to unlock the door; otherwise, the door remains locked and cannot be opened. Similarly, in PartNet-Mobility, a bottle cap can be directly lifted off. In our environment, however, the robot must first rotate the cap to a specific degree before it can be lifted.

All nine categories in our dataset, along with their corresponding mechanisms, are not covered in previous datasets such as PartNet-Mobility. This highlights the complexity and adaptability offered by AdaManip, setting it apart from existing environments.

Q3: Is diffusion policy the only viable solution? Specifically, maybe discuss potential alternative approaches and explain why you chose diffusion policy over other methods.

In our imitation learning setting, the adaptive policy is trained using a limited number of demonstration trajectories, which aligns with the practical setting adopted by previous methods. Adaptive manipulation tasks inherently require the policy to address the ambiguity of manipulation trajectories under pure visual observation. Therefore, the model must excel at modeling multi-modal distributions, and diffusion models are well-known for their strong capability in this regard. The results from the additional ACT experiment further confirm that diffusion-based approaches outperform others in this context.

Exploring alternative settings, such as online learning methods (e.g., RL), foundation model based manipulation, or large pretrained visual-language-action (VLA) models, represents a different branch of research. Within the offline imitation learning framework, diffusion policy stands out as the most effective at modeling multi-modal distributions, which is why we selected it as the backbone for our method.

Q3: AdaAfford should also be able to solve the problem presented in this paper. Why is AdaAfford not so stable due to the capability limit of cVAE? Why is the method in this paper stable?

There are two main reasons why our method performs better than AdaAfford.

First, diffusion models are more capable of modeling multi-modal distributions, and the adaptive manipulation tasks we focus on naturally require the policy to capture the ambiguity of manipulation trajectories under visual observation. Since our method is based on the more advanced diffusion model compared to the cVAE used by AdaAfford, it is better at learning the adaptive policy.

Second, AdaAfford relies on learning the point-level affordance value distribution, which is trained on both positive and negative action labels obtained from a large number of trials. The point-level affordance space is more complex than the pose action space, and therefore requires more training sample data. However, in our imitation learning setting, we need to train the adaptive policy using limited demonstration trajectories, which is a practical setting adopted by previous methods, including Diffusion Policy [1] and ALOHA [2]. To ensure a fair comparison between the baselines and our method, we use the same quantity of training data. In this case, AdaAfford suffers from inadequate data to model the entire affordance distribution, leading to a significant performance loss.

[1] Chi et al, Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023

[2] Zhao et al, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware, RSS 2023

Thank you for your review! We would like to address your concerns regarding novelty, adaptive policy and baseline performance in the following response.

Q1: What is the novelty in this paper?

We would like to emphasize that adaptive articulated object manipulation, such as opening a bottle or a microwave, represents a set of realistic and frequently encountered tasks for robots. However, the lack of attempts to build the corresponding simulation environment and dataset makes this problem relatively underexplored. Existing simulation environments overlook the complex and realistic adaptive mechanisms inherent in the internal structures of such objects.

Therefore, the primary contribution of our paper lies in introducing the crucial and realistic adaptive articulated object manipulation task along with the corresponding novel AdaManip environment and dataset. Moreover, we design an adaptive demonstration collection and policy learning pipeline, enabling the policy to model multi-modal trajectory distributions and update its posterior based on environmental feedback.

As noted by Reviewer XbhV, our environment "addresses a gap in existing robotic manipulation environments." Similarly, Reviewer eEiz acknowledged the "significant contributions by providing a rich and diverse set of objects, enabling research on the important but under-explored adaptive manipulation problem."

The adaptive articulated object manipulation problem is both crucial and underexplored in robotic manipulation research. We believe our work addresses the lack of realistic adaptive articulated object manipulation environments and provides an essential resource for the robotics community. Through this work, we aim to pave the way for further research on real-world manipulation challenges and emphasize the importance of collecting adaptive demonstrations over optimal demonstrations for learning adaptive policies.

Q2: It seems that the algorithm/model itself does not guarantee adaptive. The only "adaptive" component stems from adaptive demonstration collection. How is this "adaptive" policy different from vanilla diffusion policies?

A policy dealing with adaptive manipulation mechanisms should be equipped with the ability to model multi-modal distributions, which is the strength of diffusion-based imitation learning. However, vanilla diffusion policy fails to ensure adaptiveness, as the policy is typically trained on static optimal success trajectories that do not account for failures during manipulation. We point out in this work that such optimal demo is not adequate for realistic articulated object manipulation policy learning, which is overlooked in existing literatures.

To address this, we developed the adaptive demonstration collection policy that captures recovery from failures, enabling adaptive policy learning. Initially, the policy prior, conditioned only on visual observations, predicts multi-modal trajectories. For example, as shown in Figure 1 in our paper, in the open-safe task, the AdaManip policy might predict both directly opening the door and rotating the knob to unlock it. After attempting to pull the handle and failing to open the door, the policy updates its posterior based on the feedback, collapsing the multi-modal prediction into a single mode—rotating the knob. By learning from adaptive demonstrations and conditioning on failure feedback, our AdaManip model ensures adaptive manipulation capabilities.

Table 3 demonstrates that policies trained with static optimal demonstrations perform significantly worse than those trained with adaptive demonstrations. Furthermore, Table 4 highlights the importance of avoiding redundant adaptive trials in the demonstrations, validating the effectiveness of our design.

We sincerely thank you for your thoughtful review. As the discussion period draws to a close, we would appreciate your feedback to confirm whether our replies have addressed your concerns. If you have any remaining questions, we are happy to provide further clarification. If our responses have resolved your concerns, we would be deeply grateful if you could consider raising the score. Thank you again for your time and effort during the review process.

Thank you for your further detailed reply! We greatly appreciate the time and effort you’ve taken to explain your concerns. We would like to take this opportunity to address some misunderstandings and clarify our work further.

First, when we refer to ManiSkill2 and DaxBench, our intention is not to directly compare our work with these works, but rather to emphasize that the dataset/simulation environment itself is a significant contribution that goes beyond algorithmic innovation. As we stated in our response, the adaptive manipulation problem is important yet under-explored, and our dataset aims to help close this gap. The data synthesis and simulation mechanisms were developed with considerable effort, as the objects were not simply borrowed from existing datasets but were manually crafted and annotated. Additionally, the environments themselves required significant coding to create, further emphasizing the level of effort involved. Our dataset addresses the lack of realistic adaptive articulated object manipulation environments, providing an essential resource for the robotics community that will support future research into real-world manipulation challenges. We believe this constitutes a meaningful contribution to the field.

Secondly, thank you once again for your detailed response. We now have a clearer understanding of the concerns you raised, and we appreciate the opportunity to clarify. You mention that "the introduction of adaptive demonstrations is not on algorithmic but on data aspect, and if it is only fully used by AdaManip and not by AdaAfford, it raises a concern because the comparison is not fair in terms of how the adaptive data is used." However, we want to clarify that AdaAfford is trained on the same data as AdaManip—the adaptive demonstrations collected by our designed policy. Therefore, we believe the comparison is indeed fair, as both methods use the same adaptive data for training.

The reason why we find the criticism in your previous response somewhat unclear is that we had difficulty understanding the connection between the statement "I don’t believe the explanation" and your subsequent comment, "Even if diffusion policy is better than cVAE, it has nothing to do with the argued novelty in this paper (i.e. 'adaptive' policy learning). So I don’t understand why such comparison is meaningful for this paper." In your initial review, you raised a concern about why the performance of AdaManip is better than AdaAfford, and we provided an explanation highlighting the differences between diffusion and cVAE policies, as well as the point-level training nature of AdaAfford, which requires additional training data. We believed that this explanation addressed your concerns about the performance gap. Therefore, we were uncertain about the reasoning behind the dismissal of our explanation, and we appreciate your further clarification that prompts us to explain that the data used by both methods are the same.

Regarding the meaning of the comparison: since both models are trained on the same data, we feel that comparing AdaManip and AdaAfford is valuable, as it highlights how different design choices in the model architecture and training pipeline can better support the training of adaptive policies for the tasks we propose. As noted in our first round response and recognized in your reply, diffusion policies alone do not ensure adaptiveness, which underscores the importance of adaptive demonstrations—one of the core contributions of our work. The ablation experiments presented in Table 3 are intended to evaluate the data aspect, and the comparison with AdaAfford helps illustrate the effectiveness of diffusion models and history-conditioned action imitation, as opposed to affordance learning.

We sincerely hope our response has addressed your concerns, and if there are any additional points you would like to discuss, we would be more than happy to respond to them. Once again, we truly appreciate the time you’ve taken to elaborate on your feedback and look forward to any further suggestions you may have.

Q7: How many history steps are needed for success across the tasks?

For the tasks in this paper, a history length of 4 is sufficient for the diffusion-based model to capture the necessary failure information and update the policy distribution effectively.

This relatively short context works because we employ trajectory sparsification, which removes intermediate states and retains only keyframes. If the history consists of a dense trajectory of the robot's end-effector poses, the policy would require a longer history context to capture previous failures. However, training a policy with a long history context is computationally expensive and less robust. To address this, we sparsify the trajectory by saving only the keyframes of the demonstration trajectories for imitation learning.

For example, in the open-safe task (the most complex task among all), the recorded history includes grasping poses and several manipulation poses while omitting most intermediate steps. The history condition for the policy is as follows: [grasp the handle, pull the door and fail to open]. Other intermediate poses during execution are excluded from the history. Based on this context, the robot predicts the next goal pose [unlock the key]. Once the goal pose is predicted, we apply inverse kinematics (IK) to plan the path for execution.

By abstracting the demonstration trajectories into keyframes, an observation history length of 4 is sufficient to capture all relevant information, ensuring computational efficiency and robust policy performance.

Q8: How is sim2real accomplished? Is a new dataset collected in real setting? Or is it transferred zero-shot.

We collect real-world demonstrations via human teleoperation to train the policy for real-world settings, and thus there is no sim-to-real gap.

Q9: Figure 1 is too dense, and the symbols unintuitive. More text should be incorporated into the diagram, it’s really hard to figure out what’s going on.

Thank you for the valuable suggestion! We agree that incorporating more text into Figure 1 could improve clarity and make the motivation easier to understand. We will revise the figure accordingly to better convey the key ideas and ensure it is more intuitive. Thank you again for your helpful feedback!

[1] Chi et al, Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023

[2] Zhao et al, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware, RSS 2023

[3] Yu et al, RT-1: Robotics Transformer for real-world control at scale, ArXiv 2022

[4] Yan et al, DNAct: Diffusion Guided Multi-Task 3D Policy Learning, ArXiv, 2024

[5] Tang et al, Automate: Specialist and generalist assembly policies over diverse geometries, RSS 2024

Q4: Generalization characterization and experimental protocol. How is success rate defined? It would be desirable to have a clear set of different benchmarks, e.g. to measure cross-instance generalization, generalization across initialization/viewpoint, cross-class generalization, etc.

The testing is conducted on the training shapes at the category level, with variations in the object positions, poses (orientations), and joint configurations in both simulation and real-world experiments. Such setting aligns with current imitation learning studies [1, 2] using few-shot demonstrations (in our work, we use only 20 demos per shape for training).

The success rate is defined as the total number of successful manipulation trials divided by the total number of objects $N$ times the number of test episodes $T$ for each object: $SR = \frac{\sum_i\sum_j I_{ij}}{N * T}$, where $i$,$j$ are indices for the object and test episode, respectively.

As for the generalization towards novel shapes and categories, another branch of current works is trying to use large-scale data [3] or foundation models [4] to achieve such generalization capability, which is orthogonal to the contribution to our work: empowering imitation learning with the adaptive manipulation capability on diverse object mechanisms with our proposed environment, mechanisms and assets.

We agree that incorporating different benchmark settings would enhance the comprehensiveness of the AdaManip environment and dataset. In the release of the AdaManip dataset, we promise to provide a clear set of different benchmarks including cross-instance/class/initialization/viewpoint generalization. It is a promising direction for the following work to further study how to leverage the generalizable knowledge inherited from large models to enhance the cross-shape and cross-category generalization capability of adaptive manipulation on diverse object mechanisms.

Thank you for the valuable comments!

Q5: Why wasn’t 3D Diffusion policy considered as a policy class?

Thank you for the suggestion! We conducted additional experiments to evaluate 3D Diffusion Policy as a baseline. The quantitative results in the table below demonstrate that 3D Diffusion Policy outperforms affordance-based methods, as the diffusion model is more effective at modeling multi-modal distributions with fewer demonstrations. However, it performs worse than AdaManip due to the absence of the adaptive demonstration collection pipeline and action history conditioning.

Q6: Why was the decision made to use a logical lock/mechanism for task progression instead of a physical mechanism?

The implementation of mechanism simulation combines high-level discrete conditions and physical simulation. We define the lock state as a discrete condition that updates the joint upper limit, which subsequently affects the physical simulation. As described in Section 5.3, we track the state of the key part’s joint to determine the lock state. When the lock state transitions to "unlock," the part joint limit is lifted, allowing the part to be opened.

Taking the case of a bottle for example, the cap must be rotated to a certain angle before it can be lifted. In the simulator, we track the cap's revolute joint angle, and when it exceeds the threshold, we set the lock_state to False and increase the prismatic joint's upper limit from 0 to a non-zero value. This change allows the cap to be lifted in the physical simulation.

We choose not to simulate complex lock collision shapes, as such simulations could introduce uncontrollable errors such as interpenetration (more details discussed in [5], Appendix C), leading to instability. Since our focus is on simulating realistic adaptive manipulation mechanisms, updating the joint limits based on discrete lock states provides sufficient realism while maintaining simulation stability. Therefore, we opted for the current implementation rather than using complex shape collision simulations.

Thank you for your valuable comments! We hope to address your concerns in the following response.

Q1: More details in the policy. The policy itself was not fully characterized. Not many details were provided as to the architecture, training procedure, etc.

AdaManip policy models the conditional distribution $P(A_t|O_t,\hat{A}_t)$. Here $A_t$ refers to the predicted action sequence, $O_t$ refers to the observation history, including 3D point clouds and proprioception states, and $\hat{A}_t$ refers to action history.

The AdaManip policy is composed of two models: a PointNet++ vision encoder and a U-Net-based noise prediction network. The PointNet++ encoder processes 3D point clouds into latent embeddings. These embeddings are concatenated with proprioception states and previous actions to form a comprehensive history condition vector, which is then used as input to the noise prediction network. The noise prediction network generates the noise conditioning on the history and denoising iteration k, for the denoising scheduler, ultimately producing the predicted action sequence.

The training procedure is one-stage. After collecting the demonstration data, it is processed into pairs of $(O_t,\hat{A}_t| A_t)$ to enable end-to-end training of both the PointNet++ encoder and the noise prediction network. The training follows the loss function specified in Equation 2 of Section 4.2:

$L(\theta) = E_{k \sim \mathcal{U}(0, 100), \epsilon^{k} \sim \mathcal{N}(0, \sigma_{k}^2 I)}\left[ \left\Vert \epsilon^{k} - \epsilon_{\theta}(A_t^0+\epsilon^k, O_t, \hat{A}_t, k) \right\Vert_2^2 \right]$

If you have further concerns regarding the policy architecture or training procedure, we would be happy to provide additional details and explanations.

Q2: The ablations don’t provide much insight into why the design decisions of the policy lead to improved performance over the baselines - “Ours” without history is much stronger than the baselines.

The ablation method is ours without adaptive demonstration data, maintaining the diffusion-based imitation learning. There are two main reasons why this ablation performs much better than the affordance-based baselines.

First, diffusion models are inherently more capable of modeling multi-modal distributions. The adaptive manipulation tasks we focus on naturally involve ambiguities in manipulation trajectories under visual observation. Although the ablation method is trained solely on static success trajectories and cannot efficiently recover from failures, it can still model the multiple possible successful manipulation trajectories of the mechanism. As a result, it can possibly sample corresponding trajectories based on the current object state. In contrast, affordance-based baselines use cVAE to model multi-modal distributions, which is less effective compared to diffusion models.

Second, affordance-based methods learn the point-level affordance value distribution, which is trained on both positive and negative action labels obtained from a large number of trials. The point-level affordance space is more complex than the pose action space, and therefore requires more training sample data. However, in our imitation learning setting, we need to train the adaptive policy using limited demonstration trajectories, which is a practical setting adopted by previous methods, including Diffusion Policy [1] and ALOHA [2]. To ensure a fair comparison between the baselines and our method, we use the same quantity of training data. In this case, the baselines suffer from inadequate data to model the entire affordance distribution, leading to a significant performance loss.

Q3: It was unclear whether the baselines were fairly conducted - were the VAT-MART and AdaAfford methods trained on the same data as “Ours”? Or were pretrained weights used?

The VAT-MART and AdaAfford were trained on the same data as our method, without using any pretrained weights. This approach ensures a fair comparison between the methods.

Thank you for the insightful comments! We will provide our point-by-point response and hope our response helps address your concerns.

Q1: The generalization ability of the method.

The testing is conducted on the training shapes at the category level, with variations in the object positions, poses (orientations), and joint configurations in both simulation and real-world experiments. Such setting aligns with current imitation learning studies [1, 2] using few-shot demonstrations (in our work, we use only 20 demos per shape for training).

As for the generalization towards novel shapes, another branch of current works is trying to use large-scale data [3] or foundation models [4] to achieve such generalization capability, which is orthogonal to the contribution to our work: empowering imitation learning with the adaptive manipulation capability on diverse object mechanisms with our proposed environment, mechanisms and assets.

It is a promising direction for the following work to further study how to leverage the generalizable knowledge inherited from large models to enhance the cross-shape and cross-category generalization capability of adaptive manipulation on diverse object mechanisms.

Q2: Figure 1 & 3 are confusing. A rule-based policy described as "adaptive" is employed to collect data. However, it appears that this policy does not incorporate any prior knowledge about the object’s mechanism for the learning phase. Could you clarify this?

This rule-based demo collection policy is designed to cover all the possible failure and success states (i.e., all possible manipulation rules) during the object manipulation, as well as the best recovery actions conditioning on failure.

For example, in the open-safe task, the demonstration policy follows these steps: 1. The robot first pulls the door handle. If successful, it proceeds to open the door. 2. If pulling the handle fails, the robot switches to rotating the knob. 3. It randomly chooses a rotation direction and switches to the other direction if the first attempt fails. 4. Once the knob is successfully rotated to unlock the door, the robot pulls the handle again to complete the task.

By training on the data generated by the rule-based demo collection policy, the AdaManip policy captures the multi-modal prior distribution and develops the ability to adjust its posterior based on feedback. For instance, at the start of the open-safe task, the policy predicts multiple possibilities, such as directly opening the door or rotating the knob to unlock it. If the robot attempts to pull the handle and fails, the policy updates its posterior based on the feedback, collapsing the multi-modal prediction into a single mode—rotating the knob.

Thank you for your suggestion regarding the figures! We agree that certain modifications and clarification would improve their clarity and will update them accordingly in the revised paper.

[1] Chi et al, Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023

[2] Zhao et al, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware, RSS 2023

[3] Yu et al, RT-1: Robotics Transformer for real-world control at scale, ArXiv 2022

[4] Yan et al, DNAct: Diffusion Guided Multi-Task 3D Policy Learning, ArXiv, 2024

Q3: How was the expert demonstration collected? Are 35 demonstrations sufficient to capture the task's multi-modal nature? Is the data collection method used in simulations scalable to real-world applications?

In the real-world experiments, we collect 35 adaptive expert demonstrations for each object by human teleoperation. The Franka Emika Panda Robot Arm is reset to the initial pose with randomization at the beginning of each trial.

35 demonstrations are sufficient to capture the multi-modal nature. For example, in the open-safe task, which is the most complex among all nine categories, the macro action set is {grasp the door handle, grasp the knob, rotate the knob clockwise, rotate the knob counterclockwise, pull the door}. All trajectories, including both direct successes and recoveries from failures, will not exceed 10 distinct possibilities. Therefore, 35 trajectories are adequate to cover all modalities.

In the simulation, we employed rule-based methods for data collection, which rely on object part pose annotations and human-designed policies. These methods are scalable to real-world settings, as similar policies can be designed and poses annotated manually. However, as the complexity and diversity of mechanisms increase, designing pipelines for each category may become impractical. A promising direction for future work is to explore methods to automate or streamline the data collection process.

Q4: More discussion on history information part. How did you incorporate history? would simply adding history cause overfitting problem?

The observation history includes 3D point clouds and proprioception states. The 3D point clouds are processed by a PointNet++ model and transformed into latent embeddings. These point cloud embeddings are then concatenated with proprioception states and previous actions to form a comprehensive history condition vector, which serves as the input to the noise prediction network of the diffusion model.

Adding history does not cause overfitting; rather, it helps prevent the model from fitting to certain static trajectories: The diffusion model trained on adaptive demonstrations learns to adjust its posterior dynamically based on observation and action history.

For example, in the open-safe task, the policy might initially predict multiple possibilities, such as directly opening the door or rotating the knob to unlock it. If the robot attempts to pull the handle and fails to open the door, the policy updates its posterior based on the feedback, collapsing the multi-modal prediction into a single mode—rotating the knob. By conditioning on failure history, the AdaManip model avoids overfitting particular static trajectories.

Q5: Why did you choose point cloud input rather than RGB?

RGB images lack depth information, which is crucial for accurate object pose estimation. Adaptive manipulation tasks require predicting action poses in 3D space, which necessitates 3D information. Point cloud inputs provide this essential depth information, making them more suitable for our approach.

Q6: What type of low-level controller was used in the real-world experiment?

We use catesian impedance controller for the Franka Emika Panda Robot Arm in real-world experiments.

Q7: What is the frequency of the policy? The frequency is about 50Hz.

We sincerely thank you for your thoughtful review. As the discussion period draws to a close, we would appreciate your feedback to confirm whether our replies have addressed your concerns. If you have any remaining questions, we are happy to provide further clarification. If our responses have resolved your concerns, we would be deeply grateful if you could consider raising the score. Thank you again for your time and effort during the review process.

Q4: How is the "unlock mechanism" and other similar mechanisms simulated? Is this based on a specific collision shape or on some high-level discrete conditions?

The implementation combines high-level discrete conditions and physical simulation. We define the lock state as a discrete condition that updates the joint upper limit, which subsequently affects the physical simulation. As described in Section 5.3, we track the state of the key part’s joint to determine the lock state. When the lock state transitions to "unlock," the part joint limit is lifted, allowing the part to be opened.

For example, consider the case of a bottle. The cap must be rotated to a certain angle before it can be lifted. In the simulator, we track the cap's revolute joint angle, and when it exceeds the threshold, we set the lock_state to False and increase the prismatic joint's upper limit from 0 to a non-zero value. This change allows the cap to be lifted in the physical simulation.

We choose not to simulate complex lock collision shapes, as such simulations could introduce uncontrollable errors such as interpenetration (more details discussed in [1], Appendix C), leading to instability. Since our focus is on simulating realistic adaptive manipulation mechanisms, updating the joint limits based on discrete lock states provides sufficient realism while maintaining simulation stability. Therefore, we opted for the current implementation rather than using complex shape collision simulations.

Similarly, for other mechanisms, the implementation also relies on a combination of high-level discrete conditions and physical simulation. For instance, in the Random Rotation Direction mechanism, we randomly sample the rotation direction (clockwise or counterclockwise) as a discrete condition. This condition is used to adjust the revolute joint limits, which then influence the physical simulation.

Q5: How is the real-world demonstration collected?

We collect 35 adaptive expert demonstrations for each object by human teleoperation. The Franka Emika Panda Robot Arm is reset to the initial pose with randomization at the beginning of each trial. During inference time, our policy takes the observed cropped point cloud and robot arm joint state and outputs the end-effector keyframe action.

Q6: Could you provide the detailed hyperparameter values for the diffusion policy, such as the action horizon and prediction horizon?

In our experiments, the prediction horizon is 4, and the action horizon is 2. We will include the detailed hyperparameters of the diffusion model in the appendix of the revised paper. Thank you for the helpful suggestion!

[1] Tang et al, Automate: Specialist and generalist assembly policies over diverse geometries RSS 2024

We sincerely appreciate your thoughtful feedback and acknowledgment of our work’s significance. Below, we address your concerns in detail.

Q1: Relying on rule-based expert policies for demonstration collection can limit the complexity of the learned behaviors. Exploring methods for learning these demonstrations or incorporating human demonstrations could lead to more robust and generalizable policies.

Currently, we rely on rule-based expert policies in the simulator. However, for real-world experiments, expert demonstrations are collected through human teleoperation.

Exploring learning-based methods to generate more diverse demonstrations is indeed an interesting direction, as it could enhance the robustness and generalizability of the learned policies. One potential approach is leveraging pretrained large visual-language models to generate manipulation trajectories. Adding random noise to the generated action sequences could simulate the jitter present in human manipulation, further improving the robustness of the policy.

Q2: Will the source code be released? Environmental specifications, including frame rate and API details, are not provided.

We will release the code in an anonymous repository soon. The environment operates at a frame rate of 60, and additional API details will be included in the accompanying documentation. We are committed to maintaining the project to ensure it is user-friendly and easy to follow.

Q3: The paper primarily compares against affordance-based methods. A comparison to other offline imitation learning methods (e.g., [1]) and online learning-from-demo methods (e.g., [2][3][4]) would provide a more complete picture of the method's performance.

Thank you for your suggestion! In our imitation learning setting, the adaptive policy is trained using a limited number of demonstration trajectories, which aligns with the practical setting adopted by previous methods, such as Diffusion Policy and ALOHA. Therefore, it would not be a fair comparison to evaluate against RL methods that rely on online data collection beyond offline demonstrations.

To address your feedback, we conducted additional experiments evaluating ACT (ALOHA) as another offline imitation learning baseline. The quantitative results are shown in the following table. While ACT outperforms affordance-based methods due to its ability to model distributions with fewer demonstrations, it performs worse than AdaManip. This is primarily because ACT lacks both the adaptive demonstration collection pipeline and the strong multi-modality modeling capability provided by diffusion models.