Table 2: Navigation Performance of IG-Net compared with NoMaD, ViNT, and GNM.

To Reviewer UHPS: Thanks for your valuable feedback! We provide the response to each of your questions as follows.

Q1: Related work section does not provide a clear discussion of why the proposed model (IG-Net) is different from previously published models

A1: IG-Net stands out from previous models because it builds an implicit map comprehension through training the model with the several proposed auxiliary tasks: relative and absolute position prediction, distance prediction and navigation path prediction. Specifically, we show the implicit map understanding is especially helpful in large game environments like ShooterGame as compared to VGM, ViNT, and other baselines.

---

Q2: This is not the first model to propose to solve the navigation problem with offline data; Missing related work for offline RL

A2: We thank your comment for mentioning those related works which we don’t fully cover, and we will add these works in the discussions of our related work part. In the meantime, our method mainly focuses on designing auxiliary tasks for implicit map understanding that benefits the performance for behavior cloning methods, which is different from the main contributions of offline RL algorithms. Also, our method does not leverage reward information, which is in a different setting compared with offline RL methods.

---

Q3: Missing entire model architecture in Section 4.2 - 4.4

A3: We add the network architecture in Section 4.3 in our revision. Briefly, IG-Net uses MAE-pretrained ViT to separately encode the observation image and goal image, and feed the embedding into four additional transformer layers, and concat the final joint embedding to feed into four different encoders for corresponding tasks.

---

Q4: How the number of trajectories impacts the performance of the model

A4: We agree that it is important to test the scaling-up ability of our dataset and algorithm, which will be an important part of study in our future work. We also believe that 200 trajectories are sufficient to validate the effectiveness of our proposed approach as our model can successfully reduce all losses on both training and validation sets.

---

Q5: More baselines are needed

A5: We have added 4 baselines to the experiments including ViNT, GNM, NoMaD, and NoMaD-EMA, shown in Table 2. These baselines all use RGB observations and can be trained without RL. Results show that IG-Net outperforms all the baselines in ShooterGame. We believe the added baselines suffer significant performance loss due to lack of a z-axis, lack of long-horizon training, and lack of a low-level navigator.

---

Q6: How does the proposed approach perform in standard embodied AI navigation benchmarks?

A6: We have added Gibson experiments where IG-Net is evaluated on different seen and unseen floorplans. Since Gibson environments are much smaller (Table 1), IG-Net doesn’t have an advantage compared to previous approaches. This experiment validates IG-Net has generalization ability to novel environments and the implicit training is especially important in larger game experiments.

---

Q7: No stop action, and performance is not aligned with embodied AI literatures

A7: In the above experiments on Gibson, we have added the Stop action for both IG-Net and baseline method, and measure the success of navigation only when the agent calls the stop action within the distance threshold of the goal position. We didn’t add Stop action for ShooterGame because this may add an additional factor to the evaluation (the strategy for calling stop action, etc.). However, we will add the Stop action in the future experiments to align with existing literature.

---

Table 1: Generalization capability of IG-Net on Gibson environment on 14 unseen environments.

To Reviewer rCRb: Thanks for your valuable feedback! We provide the response to each of your questions as follows.

Q1: Although being larger, ShooterGame is more empty. Discuss other metrics for navigation difficulty.

A1: Other works use metrics including specific surface area and navigation complexity to show the complexity of the environment (see Table 1 in the paper of Gibson dataset https://arxiv.org/pdf/1808.10654.pdf). We believe that these two metrics overlook the scale of the environment which greatly affects the navigation difficulty in the environment. We think that the minimum angle of turns alongside the navigation episode is a great metric. We will add more metrics for comparing navigation complexity in our future work.

---

Q2: The paper doesn’t provide network architecture for IG-Net, and input/output for the network

A2: We add the network architecture in Section 4.3 in our revision. Briefly, IG-Net uses MAE-pretrained ViT to separately encode the observation image and goal image, and feed the embedding into four additional transformer layers, and concat the final joint embedding to feed into four different encoders for corresponding tasks.

---

Q3: IG-Net trains and tests in the same environment. Should demonstrate results in novel environments

A3: Please refer to our new experiments on Gibson, where the agent is evaluated on 14 unseen environments for short horizon navigation. Results show that IG-Net can navigate well in unseen environments and outperforms no auxiliary task baselines, showing the generalizability of IG-Net.

---

Q4: The definition of “No position”, and why removing two auxiliary tasks at the same time

A4: “No position” ablation study simultaneously removes both relative position prediction and absolute position prediction. We remove two auxiliary tasks at the same time to verify the joint effect of position loss as a whole. We have realized that removing each single auxiliary task for ablation study is better and will perform the experiment in our future work.

---

Q5: Whether historical states serve as inputs for IG-Net. Historical states are crucial to searching behaviors

A5: We really appreciate the comment. Currently, IG-Net only uses one current image and one goal image as inputs, which we also observe failure searching behavior around the corner of the map and hope to try adding historical states in the inputs. In the future work, we will carefully design experiments to verify the design for historical states.

To Reviewer Sxb4: Thanks for your valuable feedback! We provide the response to each of your questions as follows.

Q1: Compare with state-of-the-art methods on ImageNav

A1: We have included comparison between ViNT, GNM, and NoMaD baselines, shown in Table 2. We thank the reviewers for suggesting SOTA methods such as FGPrompt and OVRL. However, to keep the comparison fair and due to the restrictiveness of our environment in simulation speed and perception domains, we have to focus on non-RL algorithms that use RGB observations only.

---

Q2: The experimental setup is not testing generalization to new maps in the ShooterGame environment

A2: We perform additional experiments of IG-Net on new maps in Gibson experiments, where the agent is trained on 396 environments and evaluated on 14 novel ones, shown in Table 1. Results show that IG-Net can navigate well in the easy setting and outperforms no auxiliary task baselines, showing the generalizability of IG-Net.

---

Q3: Unclear architecture for policy network

A3: We add the network architecture in Section 4.3 in our revision. Briefly, IG-Net uses MAE-pretrained ViT to separately encode the observation image and goal image, and feed the embedding into four additional transformer layers, and concat the final joint embedding to feed into four different encoders for corresponding tasks.

---

Q4: Ablate each auxiliary task separately

A4: We really appreciate the comment, and will perform the experiment for ablating each auxiliary task in our future revision of this work.

---

Q5: Why it is not possible to get online data for RL approach

A5: We thank the reviews for suggesting the RL approach. Currently, our simulator runs at a maximum of 5 FPS to wait until an action is executed through keyboard simulation. This is too slow for RL training compared to 62.5 FPS in MuJoCo [1] and 8000-10000 FPS in Habitat-Sim [3][4].

---

[1] MuJoCo Official Documentation: https://mujoco.readthedocs.io/en/stable/programming/samples.html?highlight=fps#sabasic

[2] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033, 2012. doi: 10.1109/IROS.2012.6386109.

[3] Habitat-Sim Official Documentation: https://aihabitat.org/#:~:text=Habitat%2DSim%20simulates%20a%20Fetch,body%20dynamics%20for%201%2F30sec.

[4] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habi-tat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.

Table 1: Generalization capability of IG-Net on Gibson environment on 14 unseen environments.

Table 2: Navigation Performance of IG-Net compared with NoMaD, ViNT, and GNM.

To Reviewer m9pY: Thanks for your valuable feedback! We provide the response to each of your questions as follows.

Q1: Boundary walls are distant so exploration is easy compared to indoor environments

A1: Although the boundary walls are relatively distant, the small width of the corridors compared to the large game map still poses great challenges for exploration.

---

Q2: SPL is not good compared to SR

A2: We mention the reason for this in Section 5.2. We use Euclidean distance between state and goal position since geodesic distance is not available in ShooterGame, which makes SPL smaller compared to normal cases. We should more clearly show this fact in the table. We may also use expert path length as comparison for SPL calculation to alleviate the problem.

---

Q3: Network architecture should be more clear

A3: We add the network architecture in Section 4.3 in our revision. Briefly, IG-Net uses MAE-pretrained ViT to separately encode the observation image and goal image, and feed the embedding into four additional transformer layers, and concat the final joint embedding to feed into four different encoders for corresponding tasks.

---

Q4: How much is the policy suitable for FPS games compared to real world robots

A4: Our challenge setting mimics the real-world human player perception in games. By not requiring a panoramic camera, depth camera, or global pose, we showcase the robustness of our navigation algorithm. We do agree that there is a domain gap between game simulators and real-world when humans and other agents are involved, which is a different type of setting than this paper.

---

Q5: Clearly mention the action space

A5: Our action is discrete and similar to how humans play games using a keyboard. Specifically, the actions are {Forward, Turn Left, Turn Right} with key-pressing time being {0.4, 0.2, 0.2} seconds, respectively. We use UnrealCV as the APIs for executing the actions in the unreal engine simulator.

To Reviewer JFQf: Thanks for your valuable feedback! We provide the response to each of your questions as follows.

Q1: Experiments with a single model learned for multiple environments

A1: We perform additional experiments of IG-Net on new maps in Gibson experiments, where the agent is trained on 396 environments and evaluated on 14 novel ones, shown in Table 1. Results show that IG-Net can navigate well in the easy setting and outperforms no auxiliary task baselines, showing the generalizability of IG-Net.

---

Q2: Performance when observation and actuation noise exists, which is common in the real world

A2: Actuation noise is a very important problem for real world applications, and it can be simulated in game engines by adding action noise. Exploring how IG-Net can handle actuation noise is a good research direction and we will explore it in future work. Observation noise requires specific design in the camera, and we will test the performance in the future work.

---

Q3: Show train-time and test-time state-goal distribution

A3: We verify the difference of test-time state-goal pair and training state-goal pair by defining a metric of minimum state-goal distance. For each evaluation state-goal pair $(s,g)$, the minimum state-goal distance is defined to be $\min_{i} \|s_i - s\|_2 + \|g_i - g\|_2$, where $(s_1, g_1), …, (s_N, g_N)$ are $N=200$ state-goal pairs for navigation training data. Minimum state-goal distance describes the lowest difference in the training dataset of an evaluation state-goal pair. The mean value of minimum state-goal distance on the whole evaluation data is $2130$ on $D<4000$ difficulty, $2260$ on the $4000<D<8000$ difficulty, and $2375$ on the $D>8000$ difficulty, which is much larger than the distance threshold for success judgment ($800$). This demonstrates that the evaluation state-goal pairs have a small overlap with the training dataset.

---

Q4: Experiment for probing the implicit understanding of the map, e.g. predicting shortest path

A4: This is a great direction for testing the internal scene understanding in the IG-Net. We will perform detailed analysis on designing probing tasks for pretrained IG-Net to verify this in the future work.

---

Q5: How does IG-Net perform action predictions

A5: Actions are categorical {Forward, Left, Right} and are learned through Cross-Entropy loss between expert training data. We have added the description for action prediction in Section 4.2 in our revisions.

---

Table 1: Generalization capability of IG-Net on Gibson environment on 14 unseen environments.