ALP: Action-Aware Embodied Learning for Perception

We sincerely thank you for helpful feedback and insightful comments. We appreciate that our paper is recognized for several positive aspects: (1) our proposed framework ALP is interesting and novel in that we actively learn visual representations only using action aware training objectives; (2) our experiments and ablations are extensive and thorough, covering aspects related to effectiveness of our framework, and results outperform baselines; (3) paper is well written and organized. We address your comments and questions below:

---

Results from multiple random seeds

We already include experimental results of multiple random seeds for RND-ALP in Table 11 in Appendix. Given the same pre-trained representation and the same task samples dataset, our 3 runs of results achieve similar performance.

We additionally include downstream performance of RND-ALP over 2 runs with different runs for both pre-training and fine-tuning stages below. We generally found that performance in unseen scenarios contains slightly higher variance than that in training environments across multiple runs; overall our method could achieve consistent performance.

---

Data collection baselines

We report experimental results of fine-tuning different visual representations in downstream labeled data using random policy in Table 8 of Appendix C. We also present further comparisons with ANS method in the same Table, which tends to achieve stronger performance than frontier-based exploration. We generally found that models trained on passive samples from random policy underperform models trained on active data in both evaluation settings. This demonstrates the importance of active data collection in improving visual representation learning and training better downstream task models.

---

Explanation of RND-ALP performance

For SimCLR (FrImgNet) in Table 2, we initialize with ImageNet SimCLR pre-trained weights, and continue to learn representation from all actively explored frames by RND-ALP using contrastive loss with data augmentation. Our intuition is that ImageNet SimCLR provides a stronger initialization since it sees much more diverse curated data, and together with actively explored diverse samples from the simulator this boosts the representation quality in training environments, while at the same time we generally found that contrastive loss slightly hurts performance in unseen test scenarios. However, we found that ALP outperforms other visual representations trained from-scratch in simulator (Table 2 and 7); this shows action aware training objectives result in performance gains than self-supervised contrastive learning in embodied settings.

---

Compare to COCO pre-trained models

We provide experiment results of fine-tuning COCO pre-trained models below. All of these models are trained on the same task samples dataset collected from RND-ALP. We found that in our settings, initializing with ImageNet supervised pre-training backbone and COCO pre-trained Mask-RCNN generally achieved similar performance.

Again our goal is not to outperform the performance of these models here, since their pre-training has access to a significant amount of supervisions from semantic class labels or annotations and constructing these curated datasets require huge manual efforts.

Visual representation learning baselines

We compare to image-level visual pre-training baselines on fixed and static datasets such as ImageNet, since we propose to incorporate action signals from active embodied movements into learning visual representation in our ALP framework, which is not available when pre-training on fixed and static datasets. Given that contrastive loss is a popular self-supervised learning approach and can be integrated with a wide range of model architectures, we mainly compare to ImageNet SimCLR as the representation learning baseline model pre-trained on large-scale curated datasets. We additionally compare to CRL [1] as the previous embodied learning baseline method, and refer to ImageNet supervised pre-training in Table 3. R3M [3] utilizes language annotations of egocentric videos as another data modality that are not available in our settings, and focuses on improving success of robotics manipulation tasks, differently from our goals; thus we did not include it as a baseline in our paper draft.

Importantly, we want to show that by actively exploring in embodied environments the agent can effectively learn visual representations from action information. We are not trying to present a better pre-trained visual representation model at this point; instead we want to deliver our key insight that leveraging learning signals from active embodied movements can provide substantial gains in learning visual perception, as also shown in Table 2 and 7.

Given that more recent pre-trained visual representations are proposed, we provide reference to performance of these suggested models, including MAE [4], VC-1 [5], and R3M [3], below. We initialize the backbone encoder of Mask-RCNN models with each of pre-trained visual representations, and fine-tune on the same downstream task dataset collected from RND-ALP. We use default training configurations as ViTDet in detectron2 library [6].

However, our goal is not to outperform these methods here. Datasets used to train these representations, such as ImageNet and Ego4D, require large amounts of human effort for collections and annotations. This is different from our framework, where pre-training and fine-tuning datasets are autonomously collected from an active exploration agent moving with intrinsic motivations. Specifically, R3M is trained with a video-language alignment loss on egocentric manipulation videos, which may not be especially suitable for visual semantic tasks; although MAE and VC-1 achieves much better performance, it is commonly known that ViTDet significantly outperforms on COCO benchmark [6, 7]. Again these models are trained on curated internet-scale datasets that are manually collected and annotated with huge effort. Thus, we don’t think missing comparisons to these visual representations in our paper draft would be a weakness of our framework.

---

Revisions in manuscripts

We really appreciate your detailed feedback in improving our writing. We have updated the paper draft (marked in blue), in the Introduction section to better describe the novelty and contribution of our work and in the Table 2 caption to more precisely summarize experimental results. Please let us know if this helps clarify or answer your questions!

---

We would like to thank you again for your efforts and time in reviewing our manuscripts and providing substantial feedback. In the meantime please let us know whether our responses answer your questions and address your concerns and whether there are additional clarifications we could provide!

---

References

[1] Du, Yilun, Chuang Gan, and Phillip Isola. "Curious representation learning for embodied intelligence." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[2] Chaplot, Devendra Singh, et al. "Seal: Self-supervised embodied active learning using exploration and 3d consistency." Advances in neural information processing systems 34 (2021): 13086-13098.

[3] Nair, Suraj, et al. "R3m: A universal visual representation for robot manipulation." arXiv preprint arXiv:2203.12601 (2022).

[4] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[5] Majumdar, Arjun, et al. "Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?." arXiv preprint arXiv:2303.18240 (2023).

[6] https://github.com/facebookresearch/detectron2/tree/main/projects/ViTDet

[7] https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md

Dear Reviewer e8e8,

Thank you for your time and efforts in reviewing our paper draft.

Based on your suggestions and feedback, we provide clarifying responses and additional experiments to answer your questions. We sincerely hope that this addresses your concerns.

We would like to kindly remind that the discussion period will end soon. We wonder whether there are additional clarifications that we could further provide to address your concerns (if there are any).

Thank you very much!

Best, Authors

We would like to thank you again for your efforts and time in reviewing our manuscripts and providing substantial feedback. In the meantime please let us know whether our responses answer your questions and address your concerns and whether there are additional clarifications we could provide!

---

References

[1] Wijmans, Erik, et al. "Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames." arXiv preprint arXiv:1911.00357 (2019).

[2] Chaplot, Devendra Singh, et al. "Seal: Self-supervised embodied active learning using exploration and 3d consistency." Advances in neural information processing systems 34 (2021): 13086-13098.

[3] Du, Yilun, Chuang Gan, and Phillip Isola. "Curious representation learning for embodied intelligence." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[4] Ye, Joel, et al. "Auxiliary tasks speed up learning point goal navigation." Conference on Robot Learning. PMLR, 2021.

[5] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

[6] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[7] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

We sincerely thank you for helpful feedback and insightful comments. We appreciate that our paper is recognized for several positive aspects: (1) the idea of using reinforcement learning objective for training exploration policy as part of visual representation learning approach is interesting; (2) experiments show improvements in both visual representation quality and data quality from better exploration. We address your comments and questions below:

---

Novelty

We would like to highlight that the contributions of our framework come in two perspectives.

First, we train an active exploration agent from intrinsic motivation to collect diverse and informative samples for both representation learning and downstream task models. This is different from previous works, where CRL [3] only actively pre-trains visual representations or SEAL [2] only fine-tunes a detection segmentation model using actively collected labeled samples. Our experiments demonstrate that ALP benefits from both actively learned visual representations (Table 2, 3) and actively collected task samples (Table 4), showing the importance of actively collecting samples for training vision models.

Second, given that the agent actively explores in visual environments, we propose a representation learning approach that only uses action information from embodied movements, which optimizes a reinforcement learning objective and an inverse dynamics objective. While prior works adopt similar ideas of “shared backbone” in supervised end-to-end RL tasks [4], we demonstrate that leveraging learning signals from actions can provide substantial gains in several vision tasks. This is different from popular visual representation learning methods on fixed or static datasets, such as contrastive loss [5, 6] or reconstruction error [7]. Importantly, we want to show that by actively exploring in environments the agent can effectively learn visual representations from embodied movements, as good as or better than baselines (Table 3). Our experiments demonstrate that ALP improves visual representation learning compared to self-supervised contrastive learning from the simulator (Table 2, 7) and each component contributes to better performance (Table 5).

We hope this could bring attention to the importance of actively collecting data for learning visual perception, both for pre-training visual representations and for fine-tuning task-specific models. This provides a different perspective from popular large-scale curated datasets, such as ImageNet and COCO, that require large amounts of human effort for data collection and annotations; instead our exploration agent is trained actively and autonomously from intrinsic motivation. Additionally, we only leverage action information in embodied settings as learning signals for visual representations, which is different from popular methods trained on fixed and static datasets. Through extensive experiments and ablations on a wide range of tasks we demonstrate benefits and gains from both perspectives.

---

Results from multiple runs

We already include experimental results of multiple random seeds for RND-ALP in Table 11 in Appendix. Given the same pre-trained representation and the same task samples dataset, our 3 runs of results achieve similar performance.

We additionally include downstream performance of RND-ALP over 2 runs with different runs for both pre-training and fine-tuning stages below. We generally found that performance in unseen scenarios contains slightly higher variance than that in training environments across multiple runs; overall our method could achieve consistent performance.

---

Difficulties in training instability

We mostly follow prior work [1] when running policy training experiments. We use the distributed version of policy training algorithms and parallelize more environments, in order to gather diverse samples and reduce training instabilities. We generally found that collecting more diverse rollout samples and thus using a larger batch size could help with training instability issues in our experiments. Given our available compute resources we try to parallel as many environments as possible until saturating memory constraints of machines.

---

Revisions in manuscripts

We have updated the paper draft (marked in blue), in the Introduction section to better describe the novelty and contribution of our work and in the Table 2 caption to more precisely summarize experimental results. Please let us know if this helps clarify or answer your questions!

Dear Reviewer DHKz,

Thank you for your time and efforts in reviewing our paper draft.

Based on your suggestions and feedback, we provide clarifying responses and additional experiments to answer your questions. We sincerely hope that this addresses your concerns.

We would like to kindly remind that the discussion period will end soon. We wonder whether there are additional clarifications that we could further provide to address your concerns (if there are any).

Thank you very much!

Best, Authors

Visual representation learning baselines

We compare to image-level visual pre-training baselines on fixed and static datasets such as ImageNet, since we propose to incorporate action signals from active embodied movements into learning visual representation in our ALP framework, which is not available when pre-training on fixed and static datasets. Given that contrastive loss is a popular self-supervised learning approach and can be integrated with a wide range of model architectures, we mainly compare to ImageNet SimCLR as the representation learning baseline model pre-trained on large-scale curated datasets. We additionally compare to CRL [2] as the previous embodied learning baseline method, and refer to ImageNet supervised pre-training in Table 3.

Importantly, we want to show that by actively exploring in embodied environments the agent can effectively learn visual representations from action information. We are not trying to present a better pre-trained visual representation model at this point; instead we want to deliver our key insight that leveraging learning signals from active embodied movements can provide substantial gains in learning visual perception, as also shown in Table 2 and 7.

Given that more recent pre-trained visual representations are proposed, we provide reference to performance of these suggested models, including MAE [6], below. We initialize the backbone encoder of Mask-RCNN models with each of pre-trained visual representations, and fine-tune on the same downstream task dataset collected from RND-ALP. We use default training configurations as ViTDet in detectron2 library [7].

However, our goal is not to outperform these methods here. Datasets used to train these representations, such as ImageNet, require large amounts of human effort for collections and annotations. This is different from our framework, where pre-training and fine-tuning datasets are autonomously collected from an active exploration agent moving with intrinsic motivations. Specifically, although MAE achieves much better performance, it is commonly known that ViTDet significantly outperforms on COCO benchmark [7, 8]. Again these models are trained on curated internet-scale datasets that are manually collected and annotated with huge effort. Thus, we don’t think missing comparisons to these visual representations in our paper draft would be a weakness of our framework.

---

Revisions in manuscripts

We have updated the paper draft (marked in blue), in the Introduction section to better describe the novelty and contribution of our work and in the Table 2 caption to more precisely summarize experimental results. Please let us know if this helps clarify or answer your questions!

---

We would like to thank you again for your efforts and time in reviewing our manuscripts and providing substantial feedback. In the meantime please let us know whether our responses answer your questions and address your concerns and whether there are additional clarifications we could provide!

---

References

[1] Chaplot, Devendra Singh, et al. "Seal: Self-supervised embodied active learning using exploration and 3d consistency." Advances in neural information processing systems 34 (2021): 13086-13098.

[2] Du, Yilun, Chuang Gan, and Phillip Isola. "Curious representation learning for embodied intelligence." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[3] Ye, Joel, et al. "Auxiliary tasks speed up learning point goal navigation." Conference on Robot Learning. PMLR, 2021.

[4] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

[5] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[6] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[7] https://github.com/facebookresearch/detectron2/tree/main/projects/ViTDet

[8] https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md

We sincerely thank you for helpful feedback and insightful comments. We appreciate that our paper is recognized for several positive aspects: (1) our experimental results outperform baselines in multiple tasks; (2) our ablation experiments effectively strengthen our framework. We address your comments and questions below:

---

Novelty

We would like to highlight that the contributions of our framework come in two perspectives.

First, we train an active exploration agent from intrinsic motivation to collect diverse and informative samples for both representation learning and downstream task models. This is different from previous works, where CRL [2] only actively pre-trains visual representations or SEAL [1] only fine-tunes a detection segmentation model using actively collected labeled samples. Our experiments demonstrate that ALP benefits from both actively learned visual representations (Table 2, 3) and actively collected task samples (Table 4), showing the importance of actively collecting samples for training vision models.

Second, given that the agent actively explores in visual environments, we propose a representation learning approach that only uses action information from embodied movements, which optimizes a reinforcement learning objective and an inverse dynamics objective. While prior works adopt similar ideas of “shared backbone” in supervised end-to-end RL tasks [3], we demonstrate that leveraging learning signals from actions can provide substantial gains in several vision tasks. This is different from popular visual representation learning methods on fixed or static datasets, such as contrastive loss [4, 5] or reconstruction error [6]. Importantly, we want to show that by actively exploring in environments the agent can effectively learn visual representations from embodied movements, as good as or better than baselines (Table 3). Our experiments demonstrate that ALP improves visual representation learning compared to self-supervised contrastive learning from the simulator (Table 2, 7) and each component contributes to better performance (Table 5).

We hope this could bring attention to the importance of actively collecting data for learning visual perception, both for pre-training visual representations and for fine-tuning task-specific models. This provides a different perspective from popular large-scale curated datasets, such as ImageNet and COCO, that require large amounts of human effort for data collection and annotations; instead our exploration agent is trained actively and autonomously from intrinsic motivation. Additionally, we only leverage action information in embodied settings as learning signals for visual representations, which is different from popular methods trained on fixed and static datasets. Through extensive experiments and ablations on a wide range of tasks we demonstrate benefits and gains from both perspectives.

---

Inverse dynamics prediction as training objective

While reinforcement learning objective provides a learning signal to associate visual frames to action, we additionally make an observation that consecutive frames encode agent’s movements in interactive settings. We design our inverse dynamics model to predict sequences of actions given observations, as shown in Figure 1. In this case, we directly infuse action information into visual representation learning, in addition to policy and reward information from reinforcement learning objectives. Our inverse dynamics model also provides temporal learning signals since it is trained on sequence of trajectories over multiple steps, different from RL policy that associate observation and action at every single step. Our ablation experiments in Table 5 clearly show that both of these components contribute to improved performance.

Dear Reviewer h74j,

Thank you for your time and efforts in reviewing our paper draft.

Based on your suggestions and feedback, we provide clarifying responses and additional experiments to answer your questions. We sincerely hope that this addresses your concerns.

We would like to kindly remind that the discussion period will end soon. We wonder whether there are additional clarifications that we could further provide to address your concerns (if there are any).

Thank you very much!

Best, Authors

Results from multiple runs

In addition to Table 11, we include downstream performance of RND-ALP over 2 runs with different runs for both pre-training and fine-tuning stages below. We generally found that performance in unseen scenarios contains slightly higher variance than that in training environments across multiple runs; overall our method could achieve consistent performance.

---

Number of pre-training steps

We try to follow prior work [3] to pretrain for 10M steps, but given available compute resources we reduce the number of parallel environments than in their implementation and our exploration policy is trained for around 8M steps. We fix this parameter value across experiments to compare effects of other components of our framework on final performance.

---

Revisions in manuscripts

We have updated the paper draft (marked in blue), in the Introduction section to better describe the novelty and contribution of our work and in the Table 2 caption to more precisely summarize experimental results. Please let us know if this helps clarify or answer your questions!

---

We would like to thank you again for your efforts and time in reviewing our manuscripts and providing substantial feedback. In the meantime please let us know whether our responses answer your questions and address your concerns and whether there are additional clarifications we could provide!

---

References

[1] Xia, Fei, et al. "Gibson env: Real-world perception for embodied agents." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[2] Chaplot, Devendra Singh, et al. "Seal: Self-supervised embodied active learning using exploration and 3d consistency." Advances in neural information processing systems 34 (2021): 13086-13098.

[3] Du, Yilun, Chuang Gan, and Phillip Isola. "Curious representation learning for embodied intelligence." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[4] Savva, Manolis, et al. "Habitat: A platform for embodied ai research." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[5] Ye, Joel, et al. "Auxiliary tasks speed up learning point goal navigation." Conference on Robot Learning. PMLR, 2021.

[6] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

[7] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[8] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

We sincerely thank you for helpful feedback and insightful comments. We appreciate that our paper is recognized for several positive aspects: (1) the paper is well-written and easy to follow; (2) our framework has strong generality and can be integrated with any active exploration method; (3) we provide extensive experimental comparisons to baseline methods and empirical evidence is clear. We address your comments and questions below:

---

Limited scope of environments

While we perform all experiments in the Gibson dataset [1], different scenes in the Gibson dataset contain some extent of diversity. We follow prior work [2] in splitting training and unseen test scenes, details provided in Appendix B2. Our evaluations on unseen test scenarios could indicate some generalization capability of our framework, since these environments are not available during either pre-training or fine-tuning stages. We found it a little challenging to directly evaluate on another dataset such as Replica or Matterport3D, since these datasets capture different styles of scenes or buildings and distributions of object categories are different, which could add further complications to evaluations. As discussed in the last part, we would like to leave further extensions to other simulator datasets and/or real world environments as part of future work.

---

Simple action space

We define the action space in our experiments same as in prior works [2]. We render the Gibson dataset in the Habitat simulator [4] and train a reinforcement learning policy of a navigation agent. We tried looking up and down as possible actions similar to CRL [3], but we found that this collects weaker samples, e.g. the agent might get many observations of ceilings and floors without valid semantic annotations of objects. We thus use the same action space as SEAL [2]. We are not aware of different choices of embodied agents as you described under our setup; could you further clarify this part? Thanks! Although we currently utilize a simple action space in our experiments, we already observe performance gains, indicating effectiveness of leveraging action information for visual representation learning. As discussed in the last part, we would like to leave further investigations on more complex action space as part of future work.

---

Novelty

We would like to highlight that the contributions of our framework come in two perspectives.

First, we train an active exploration agent from intrinsic motivation to collect diverse and informative samples for both representation learning and downstream task models. This is different from previous works, where CRL [3] only actively pre-trains visual representations or SEAL [2] only fine-tunes a detection segmentation model using actively collected labeled samples. Our experiments demonstrate that ALP benefits from both actively learned visual representations (Table 2, 3) and actively collected task samples (Table 4), showing the importance of actively collecting samples for training vision models.

Second, given that the agent actively explores in visual environments, we propose a representation learning approach that only uses action information from embodied movements, which optimizes a reinforcement learning objective and an inverse dynamics objective. While prior works adopt similar ideas of “shared backbone” in supervised end-to-end RL tasks [5], we demonstrate that leveraging learning signals from actions can provide substantial gains in several vision tasks. This is different from popular visual representation learning methods on fixed or static datasets, such as contrastive loss [6, 7] or reconstruction error [8]. Importantly, we want to show that by actively exploring in environments the agent can effectively learn visual representations from embodied movements, as good as or better than baselines (Table 3). Our experiments demonstrate that ALP improves visual representation learning compared to self-supervised contrastive learning from the simulator (Table 2, 7) and each component contributes to better performance (Table 5).

We hope this could bring attention to the importance of actively collecting data for learning visual perception, both for pre-training visual representations and for fine-tuning task-specific models. This provides a different perspective from popular large-scale curated datasets, such as ImageNet and COCO, that require large amounts of human effort for data collection and annotations; instead our exploration agent is trained actively and autonomously from intrinsic motivation. Additionally, we only leverage action information in embodied settings as learning signals for visual representations, which is different from popular methods trained on fixed and static datasets. Through extensive experiments and ablations on a wide range of tasks we demonstrate benefits and gains from both perspectives.

Dear Reviewer mMb6,

Thank you for your time and efforts in reviewing our paper draft.

Based on your suggestions and feedback, we provide clarifying responses and additional experiments to answer your questions. We sincerely hope that this addresses your concerns.

We would like to kindly remind that the discussion period will end soon. We wonder whether there are additional clarifications that we could further provide to address your concerns (if there are any).

Thank you very much!

Best, Authors