IGOR: Image-GOal Representations are the Atomic Building Blocks for Next-Level Generalization in Embodied AI

We would like to thank the reviewer for valuable feedback and for the comments and questions for additional clarity. Below, we address the key comments and questions.

W1. Rationale and ablations on the architecture design

Thank you for your suggestion! To control training costs, we conducted ablation studies on a small subset of the training data and provided visualization results using FDM in the action tokenizer.

a) Latent Action Extraction.

We compared two approaches: (1) applying Conv2D on re-organized image patches, and (2) using readout tokens. With the same latent dimension, readout tokens ( shown in https://raw.githubusercontent.com/iclr-2025-4517/index/refs/heads/main/images/rebuttal_imgs/rebuttal_fig2.png ) achieved better reconstruction results than Conv2D ( https://raw.githubusercontent.com/iclr-2025-4517/index/refs/heads/main/images/rebuttal_imgs/rebuttal_fig1.png ), suggesting it help extract information more efficiently.

b) Leveraging Temporal Information.

We compared two methods: (1) using a vanilla ViT on each frame, and extracting information between the adjacent patches, and (2) adding an ST transformer after the ViT to fuse more history context. As shown in figure [ https://raw.githubusercontent.com/iclr-2025-4517/index/refs/heads/main/images/rebuttal_imgs/rebuttal_fig3.png ] the pure ViT version tended to directly copy the robot arm's position instead of learning relative changes. In contrast, adding historical information mitigated this issue. We hypothesize that incorporating historical context helps the action tokenizer to identify key objects changes. Additionally, applying random cropping, as discussed in Section 2.1, further improved the results.

We hope our findings and trials may bring some insights to the community!

W2. The training resources and inference frequency

For action tokenizer training, we used about 200 H100 GPU days. For policy and world model, we each used about 100 H100 GPU days to train.

Regarding inference frequency, we tested the policy's inference speed by performing 1,000 forward passes on a local A100 GPU. The process took approximately 40.2 seconds to complete, suggesting the policy itself can run at roughly 25Hz.

W3. Regarding the rotation information in latent actions

Thank you for your suggestion! While Fig. 6b shows that latent actions are more predictive of positions, we argue that they still retain sufficient information about rotations.

We acknowledge that the tasks in Simpler environments are limited. To address this, we propose an experiment where a video of a rotating and moving robot gripper is encoded into a latent action sequence. This sequence is then applied to the first frame and decoded back into a video through the world model.

If the latent actions contain rotation information, the reconstruction will accurately reflect both rotation and movement. Otherwise, the reconstruction will only show movement.

The results are available on “Latent Action for Rotation” part of the website [ https://iclr-2025-4517.github.io/index/rebuttal.html ], showing the original video on the left and the reconstructed video on the right. From the video, it is evident that the robot arm both rotates and moves. Though the gripper is slightly distorted when rotating, it completes the full trajectories with both rotation and movement, suggesting the latent action captures enough rotation information.

We sincerely appreciate the time you have taken to review our response. With our rebuttal now submitted, we are eager to know if it has adequately addressed your concerns. As the rebuttal phase draws to a close, we would be grateful for any additional feedback or clarification you might require.

We really appreciate reviewer dXPY for the positive and encouraging feedback!

The intuition on adding IDM and FDM.

We would love to share more intuition on the IDM and FDM for the latent action model. The key idea is based on the observation that we can easily identify the object correspondence between the image and goal (when they are not too far apart in time) and infer the motion of objects/arms solely from visual changes. Based on this key idea, the IDM takes a pair of image and goal (as well as historical frames) as input, allowing it to infer the visual changes by comparing the current image with the goal. The FDM then takes the current image and the latent action as input to reconstruct the goal as the training loss, which ensures that the latent action contains all the information about the visual changes. By compressing the latent action effectively, image-goal pairs with similar visual changes will have similar latent action embeddings, as demonstrated in the experiments and video demos presented in our paper.

We sincerely appreciate the time you have taken to review our response. With our rebuttal now submitted, we are eager to know if it has adequately addressed your concerns. As the rebuttal phase draws to a close, we would be grateful for any additional feedback or clarification you might require.

We would like to thank reviewer 68vo for recognizing the importance of our latent action approach. Below, we address the key comments and questions.

W1. Missing baselines: UniPi

Thank you for pointing out potential baselines. We implemented UniPi as our baseline. For the video generation model, we fine-tuned the pretrained OpenSora which is the same backbone as our world model. Additionally, we trained an inverse dynamics model to predict real robot action of google robot in Simpler environment.

The success rates are as follows, demonstrating that our method achieves higher success than UniPi rates across all tasks.

W2. Expect experiments on more tasks

We acknowledge that the tasks in Simpler environments are limited. Currently, we do not have real world robots for real world tasks. At this stage, our method focuses on verifying the feasibility of the latent action approach. In future work, we will explore its generalization across a broader range of tasks and real world robots.

W3. several claims that are not convincingly supported by the experiments

Thank you for your valuable suggestions regarding our wording. We greatly appreciate your feedback and have revised our claims to be more precise and rigorous. Based on your input, we have updated the title to: “IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI”. Here, "control units" emphasize the role of latent actions in enabling the learning of both foundation policy models and foundation world models from video data. The main paper will also be updated accordingly.

Regarding the claim about the latent action space versus the original image space, we have implemented UniPi as our baseline, as discussed in W1, and we hope this addition will support our argument.

By designing a unified latent action space, the IGOR framework facilitates the learning of foundation policy and world models using both robotic data and internet-scale videos. While the current dataset may not yet be large enough to serve as a foundation model, the method is fully scalable. We plan to expand the dataset to further improve the learning of foundation models in future work.

Q1. How is the data preprocessed? Why choose the optimal frame rate?

For data preprocessing, we employed a combination of automated and manual filtering and stabilization. Automated methods included a shaking-detection model and video stabilization software to enhance data quality, while manual inspection addressed edge cases.

The optimal frame rate was selected to ensure latent actions capture meaningful visual changes. We choose interval equals to 3 for OpenX dataset, and the time interval to be in [0.1, 0.5] for egocentric videos. A frame interval that is too short results in nearly identical frames, where latent actions represent minimal changes. Conversely, a frame interval that is too long may cause latent actions to capture excessive visual changes, particularly in egocentric videos.

Q2. What is the purpose of learning the world model? It is entirely not used for low-level control.

In this work, the world model primarily serves to assess the semantic consistency of the learned latent actions across humans and robots.

Besides, learning world models has long been a fundamental topic in embodied decision-making. By introducing a unified latent action space, the world model gains greater flexibility and potential. For future work, there is potential to extend the world model to synthesize data to address the data scarcity for policy learning.

We sincerely appreciate the time you have taken to review our response. With our rebuttal now submitted, we are eager to know if it has adequately addressed your concerns. As the rebuttal phase draws to a close, we would be grateful for any additional feedback or clarification you might require.

Q4: Missing citations and discussions to related works

We really appreciate your comment for pointing out the missing related works. We have cited all these related works in Appendix E of our updated paper, and discuss the relationship of IGOR and these papers as follows.

For latent action model and policy model related work: Any-Point Trajectory Modeling (ATM) and Tract2Act generates point tracking from the action-free videos for pretraining the policy to predict future trajectories of the tracked point, and learns a robot policy conditioned on generated trajectories. HOPMan generates future human plans as conditions for facilitating robot policy learning. Im2Flow2Act conditions the robot policy on complete generated object flows, which captures movement information only for the object, improving its cross-embodiment transfer capability.

Same as ATM, HOPMan, Im2Flow2Act, and Tract2Act, IGOR also pretrain a foundation policy on action-free videos that generalize across embodiments. There are two major differences: (1) IGOR uses an unsupervised way to learn and compress the visual changes, while existing work needs an additional pretrained video model for point tracking; (2) IGOR uses a compact latent representation for visual changes, while existing works uses an explicit representation for visual changes. The compact action representation enables IGOR to transfer human actions on robots directly, as shown in Figure 2 in our paper.

In the world model aspect, Gen2Act uses the VideoPoet model for text-to-video generation, and the generated human videos are used for downstream policy learning. Compared to existing world model works, IGOR can learn generalizable latent actions for fine-grained control on manipulation scenarios for both human and robots, while existing works use coarse grained text conditions for generating video predictions.

Q5: Add a figure for the method part (Section 2.1 - 2.3)

We really appreciate the comment, and have added a figure to describe our main method and network architecture in Appendix D of our updated paper.

We would like to thank reviewer WjFr for recognizing the novelty of our latent action approach, and for pointing out related works, such as point tracks or text as latent actions, which are alternatives of the latent action approach. We would love to address the reviewer’s questions in detail as follows.

Q1: Why is image goal (latent action) necessary given the alternative choices?

As mentioned in previous work that leverages actionless video data for robot learning, capturing motion in video is crucial, which is why there are numerous works on this topic. They employ different approaches such as point tracking, text, optical flow, and motion vectors from various research areas. While these methods are mature and deliver good performance, they all require additional prior knowledge from labeled information or labels generated by other models, falling under the category of supervised learning (SL) that leverages prior knowledge and structured data. In contrast, our latent action approach belongs to self-supervised learning (SSL), which has the potential to surpass supervised learning methods due to its ability to leverage large-scale unlabeled data.

Q2: Impact and utility of IGOR latent action model, and core contribution of IGOR

We propose the latent action approach as a self-supervised learning (SSL) alternative to existing methods like point tracking. Our efforts in this novel technical direction reveal the potential and possibilities of the latent action approach. Preliminary qualitative results demonstrate the semantic consistency of the learned unified action space across humans and robots using the world model. Additionally, preliminary quantitative results show the effectiveness of employing latent actions as sub-tasks within the policy model. Although these early results cannot yet match those achieved by more mature approaches in terms of task diversity and difficulty, we believe they demonstrate that the latent action approach is promising and merits further research.

Q3: Can IGOR represent actions that are more fine-grained (e.g. tasks demonstrated in the ALOHA paper) and long-horizon?

More fine-grained latent actions: We would like to provide more experimental results to show the expressiveness of IGOR latent action for capturing more fine-grained actions. We extract latent actions from out-of-distribution videos demonstrating various fine-grained control from Mobile ALOHA paper, and use the extracted latent actions and the initial frame to reconstruct the video by the IGOR world model. We provide the original video and generated video respectively in the “ALOHA Demonstration” part in the website [ https://iclr-2025-4517.github.io/index/rebuttal.html ]. The results indicate that IGOR can encode fine-grained robot controls in latent actions. We perform the same experiment with robot arm rotation, with the result shown in the “Latent Action for Rotation” part of the website [ https://iclr-2025-4517.github.io/index/rebuttal.html ]. The results show that IGOR can encode 3D robot arm rotations and movements in the latent actions.

Long-horizon: Our existing latent action model captures the visual changes between two frames. If the frames are too far apart, the changes might be too large to model accurately, since it might be hard to find the same objects or scenes between the two frames. As a result, it would be better to have a sequence of latent actions to capture long-horizon visual changes.

We sincerely appreciate the time you have taken to review our response. With our rebuttal now submitted, we are eager to know if it has adequately addressed your concerns. As the rebuttal phase draws to a close, we would be grateful for any additional feedback or clarification you might require.

I understand the difference between SL and SSL outlined by the authors. I'd like to use the line of work that learns robotics knowledge from human videos (HOPMan, Track2Act, Gen2Act, and more recently ZeroMimic https://zeromimic.github.io/) as context for my discussion. It is true that these works use labels in the form of segmentation masks, point tracks, or hand reconstruction. But these labels are automatically generated by running existing off-the-shelf vision foundation models on unlabled datasets. Ultimately, the input and output of their systems are the same as this work: they also use unlabeled videos as input, and produce some "intermediate action representation" as output, with no extra information needed. I argue that the choice of producing this "intermediate action representation" using an off-the-shelf foundation that everyone has access to is a method choice and detail, and again does not change the input and output of the system. Therefore, I respectfully disagree with the authors' statement that the work "has the potential to surpass supervised learning methods due to its ability to leverage large-scale unlabeled data", given that these "supervised learning methods" in fact uses unlabeled datasets too.

Each work chooses a different representation of action as their version of the "intermediate action": mask, point tracks, and in this work, some latent vector. Which action representation is better? This work provides no experimental evidence to show that their representation is more advantageous, only that it is "new", which I acknowledge. In fact, I would argue that the action representations in prior works have two obvious advantages over this work: 1. they are more interpretable, and 2. they are more scalable since they only need to train on human videos (note that in all the listed prior works, robot data is only used for downstream policy learning but not for training intermediate action generation), and also significantly less human videos data than this work, whereas this work requires training on both robot and human videos, making it less scalable.

Finally, a note about using text labels. All of the datasets IGOR trains on actually have text labels, but it was the authors' choice not to use them. I would argue that using text to specify the task is more advantageous: it is much more intuitive and less cumbersome than using a goal image. Furthermore, using image goal to specify task is also not a new idea, and it has been explored in H2R (https://sites.google.com/view/human-0shot-robot), HOPMan, ZeroMimic, just to name a few.

First, I'd like to apologize for my late response.

Image goal representations are not a novel idea. As listed in my previous comment, a line of prior works (HOPMan, H2R, ZeroMimic, etc.) have explored generating intermediate actions from image goal representation by training on unlabeled datasets, same as IGOR. Latent actions as the intermediate action is indeed new, but there's no justification that this latent action is superior to other types of intermediate actions, therefore I don't believe this paper merits acceptance.

Ultimately, I believe the best evaluation of an embodied AI method, whether it has potential and is promising, is with embodied AI performance itself, especially in comparison to prior methods. I am not convinced by the authors that IGOR has potential for embodied AI given its lackluster robot results and insufficient baseline comparisons.

And therefore, I would like to keep my original score.