EnvBridge: Bridging Diverse Environments with Cross-Environment Knowledge Transfer for Embodied AI

Question 3: Metaworld results with CALVIN knowledge

We focused on RLBench as the source environment because MetaWorld’s standardized task set and simulated table robotic manipulation scenario are similar to the settings in RLBench. However, we sincerely agree with your suggestion to include CALVIN results, which is crucial for more comprehensive comparison. Here we have the updated results.

The new results show that knowledge transfer from CALVIN (31% average success rate) performs slightly worse than transfer from RLBench (37% average success rate) when applied to MetaWorld tasks. This could be explained by the factor that RLBench's environment and task structure might be more similar to MetaWorld compared to CALVIN, making knowledge transfer more effective.

Question 4: Section 4.3 source tasks

Thank you for pointing out the evaluation settings. We use RLBench as source tasks for CALVIN evaluation, and we will add this information to our rebuttal revision.

Question 5: Baseline choices

EnvBridge utilizes VoxPoser as its code generation module, and we compared these systems to evaluate the improvements achieved through our proposed Cross-Environment Knowledge Transfer. We chose VoxPoser over other methods such as Code as Policies (CaP) primarily because VoxPoser's hierarchical structure and higher-level planning modules are more suitable for EnvBridge.

Additionally, the previous study [1] has reported the following evaluation results of CaP and VoxPoser on RLBench tasks:

While direct comparisons with EnvBridge are not feasible due to differences in experimental conditions, existing comparative studies between VoxPoser and CaP have demonstrated VoxPoser's superior performance.

[1] J. Duan, W. Yuan, W. Pumacay, Y. R. Wang, K. Ehsani, D. Fox, and R. Krishna, “Manipulate-anything: Automating real-world robots using vision-language models,” arXiv preprint arXiv:2406.18915, 2024.

Question 6: Minor formatting issues

Thank you for pointing out the formatting issues, we will update accordingly in the rebuttal revision.

Thank you for your valuable feedback and for recognizing the creative potential of our approach. We particularly appreciate your positive feedback on using previously successful control code as in-context prompts for new tasks. This is indeed a key innovation of our work.

We address your concerns and questions below.

Weakness 1: Practical applicability to real-world robotic systems

Thank you very much for your insightful comment regarding the real-world applicability of our approach.

Existing methods such as CaP and VoxPoser have successfully demonstrated the feasibility of LLM-based robot code generation for real robots. Building upon VoxPoser's framework, our approach introduces Cross-Environment Knowledge Transfer techniques to enhance adaptability across various environments, thereby extending its potential for real-world applications.

While our current implementation relies on ground-truth observation data in simulation environments, we can extend our system to real-world applications by following VoxPoser's proven approach. Specifically, VoxPoser has demonstrated that Vision Language Models (VLMs) like OWL-ViT can effectively replace simulation-based observations with real visual information processing. Through this integration, our system can transition to real-world deployment without depending on simulation-specific APIs.

Question 1: Perception Choice

Thank you for your question regarding our system's input modalities.

In our current evaluation, we utilize ground-truth states as input. However, for real-world applications, our system is designed to work with camera inputs processed through Vision Language Models (VLMs), ensuring practical deployability beyond simulation environments.

Question 2: The experimental protocol in Section 4.1

Thank you for your question about our experimental protocol.

We selected 10 RLBench tasks (detailed in section A.4.1) and evaluated each task across 20 episodes with randomized instructions, objects, and placements. The success rate was calculated by averaging these 20 episodes per task, following the same evaluation protocol as VoxPoser.

Thank you very much for taking the time to review our paper and providing your feedback. We would like to address each of your comments in detail below.

Weakness 1: Introduction Section Redundancy

Thank you for your opinions. We understand that extraneous information can lead to confusion. Our introduction flows from the general landscape of LLMs and recent developments, followed by a transition to robotic manipulation as a use case that is gaining traction, and then to limitations of current works in the area before proposing our solution to tackle these limitations. We would appreciate it if you could let us know which points you found redundant.

Weakness 2: Clarity of the Article

We appreciate this feedback. We will consider revising the sentences to provide detailed and clear definitions.

Weakness 3: Quality of the Figures

We acknowledge that clear figures can greatly enhance comprehension. We would be grateful if you could let us know which aspects of the figures you found concerning.

Weakness 4: Disagreement on Generalization Issues

We sincerely thank the reviewer for raising this important point about generalization capabilities. We would like to respectfully offer some clarification about our specific findings:

4.1. Our paper identifies two specific limitations in current LLM-based approaches:

• Policy-based methods require environment-specific training, limiting cross-environment transfer
• Code generation methods struggle with fixed prompts in unseen environments

4.2. Our experiments systematically examined these limitations:

• The baseline tests show VoxPoser achieves 36.5% success on RLBench and 25% on MetaWorld tasks
• These results suggest there may still be room for improvement in handling unseen environments

4.3. Our proposed approach shows promising progress in addressing these challenges:

• Improving success rate to 69.0% on RLBench
• Achieving 56% success rate on MetaWorld

We would be very interested to learn about the specific LLM-based approaches the reviewer has in mind that demonstrate strong generalization abilities in robotic manipulation. This would be valuable for our research and help us provide more comprehensive comparisons. We are always eager to learn from and build upon successful approaches in the field.

Once again, thank you for your valuable feedback and time. To conduct better research, we would be grateful if you could provide more detailed advice, if possible.

Thank you for your thorough and constructive feedback! We are particularly encouraged by your recognition of our novel approach Cross-Environment Knowledge Transfer for robot code generation and its clear motivation from human behavior. We also appreciate your acknowledgment of our extensive experimental validation across multiple tasks and environments, which we indeed invested significant effort in to demonstrate the effectiveness of EnvBridge.

We address your concerns and questions below.

Weakness 1: Open-loop Trajectories

We appreciate your thoughtful observation about open-loop trajectories in dynamic environments. We would like to respectfully offer some additional context about our approach that may help clarify this aspect of our work.

• First, while EnvBridge does generate open-loop trajectories, it's important to note that these trajectories are generated separately for each decomposed subtask, not for the entire task at once. Our Planner and Composer break down complex tasks into simpler subtasks, each requiring only short, straightforward trajectories. This decomposition significantly reduces the risk of execution failures.
• Second, we sincerely agree that reactive control is crucial for robust operation in dynamic environments. However, EnvBridge's primary contribution is improving the agent's planning capabilities through cross-environment knowledge transfer, which is orthogonal to reactive control. We believe our approach is complementary to reactive control methods - the enhanced planning capabilities from cross-environment knowledge transfer could be combined with reactive control techniques to create more robust systems. We have added this discussion in our rebuttal revision to clarify the scope of our contribution and potential future directions.

Weakness 2: Addressing collision with environment

Thank you for pointing this out. The paper is inspired from Voxposer, where they use the concept of affordance and avoidance maps, for entity of interest and entities to be repulsed after which the trajectory is planned. This allows the system to consider potential collisions with the environment and plan trajectories that minimize such interactions. By leveraging these techniques, we aim to ensure safe and efficient robot behavior.

Regarding the concern about potential collisions involving other parts of the robot (e.g., arms or base) with the environment, we acknowledge this as a valid limitation. Addressing this challenge is an important direction for future work. We plan to explore and implement improved methods that account for the entire robot’s structure, incorporating advanced collision-avoidance techniques to enhance the system's overall safety and effectiveness in complex environments. Thank you for bringing this to our attention.

Weakness 3: Ensuring Avoidance of Non-Target Entities

Thank you for your comment. Currently we have considered the avoidance maps at composer level. For example, if the planner generates the following query from the perception received, the composer generates the relevant affordance and avoidance maps:

# Query: move to the back side of the table while staying at least 5cm from the blue block.
movable = parse_query_obj('gripper')
affordance_map = get_affordance_map('a point on the back side of the table')
avoidance_map = get_avoidance_map('5cm from the blue block')
execute(movable, affordance_map=affordance_map, avoidance_map=avoidance_map)

The composer generates maps to ensure the gripper interacts only with intended entities while avoiding unnecessary contact with others, such as the blue block. This method effectively minimizes unintended interactions, ensuring a safer and more efficient trajectory execution.

Weakness 4: Baseline comparison

Thank you for the suggestion. Comparing EnvBridge with other baselines is an interesting direction. We have compared our work to RVT—a multi-view transformer for 3D manipulation that utilizes RLBench training dataset with 100 expert demonstrations per task used to predict robot end-effector pose. EnvBridge achieves an average success rate of 69% on the RLBench benchmark compared to RVT’s 62.9%, highlighting the effectiveness of our approach. While we acknowledge that the tasks used in RVT’s evaluation are not identical to ours, this comparison highlights the efficiency of our approach. What sets EnvBridge apart is its domain-agnostic nature. Unlike RVT, which requires domain-specific training, EnvBridge is designed to generalize across diverse environments and tasks without extensive retraining or adaptation. This versatility makes EnvBridge both effective and adaptable, offering broader usability compared to baselines that are tied to specific domains. This distinction underscores the potential of our approach to address a wider range of applications in dynamic and varied settings.