Learning with a Mole: Transferable latent spatial representations for navigation without reconstruction

The number of real-world experiments is rather low (11 episodes). While I can see how three baselines + the method itself add up in terms of workload, with only 11 runs I cannot tell if the sim2real results are statistically significant.

We agree that the more real-world experiments, the better. As rightfully noted by the reviewer, these experiments are time-consuming, and benchmarking four different methods leads to considerable engineering effort. In this paper we conducted 44 robot experiments, and we would like to stress that most papers in this space present significantly fewer experiments, for example:

• “Object Goal Navigation using Goal-Oriented Semantic Exploration”. NeurIPS 2020: 20 robot experiments.
• “Success weighted by completion time: A dynamics-aware evaluation criteria for embodied navigation”. IROS 2021: 12 runs, i.e. 2 episodes, each one repeated 3 times, for 2 methods.
• “Navigating to Objects Specified by Images”. ICCV 2023: 8 experiments.

The paper is currently missing information about the complexity of the environments and the length of the evaluated navigation runs in them. [...] I am primarily concerned about this because navigation success metrics depend very much on a) the geodesic length of the navigation trajectory and b) how maze-like the layout of the scene is. Revealing the complexity of the considered navigation tasks will make the paper stronger.

Thanks, the revised manuscript features a top-down map of the real environment with the ground-truth test trajectories in Table 1 (previously Table 2, we swapped Table 1 and Table 2 for readability). We also include the average geodesic length of the test episodes, which is 8.9 meters - for comparison Gibson val episodes have an average length of 5.9 meters. One room of the test environment is shown in Fig. 1 and features thick carpets, uneven floor and large windows that disturb the robot sensing.

I am not certain if the blindness property [...] is as useful compared to data augmentation: [...] I am not convinced that the removal of observations in the auxiliary policy matters a lot. Simple behavior cloning + the subgoal data augmentation ((c) in Table 1) performs very well both in Sim and in NoisySim, better than plain navigability training ((d) in Table 1).

Differences between methods (c) and (d) are indeed small in simulation (Table 2 in the revised manuscript, previously Table 1). However we argue that the navigability loss improves sim2real transfer because it relies on non-visual navigability information that has reduced sim2real gap compared to RGB-D, used with data augmentation. Thus it is not surprising that performance in simulation is similar for the two approaches, while we observe significant gains in real-world experiments, Table 2 of the revised manuscript (previously Table 1), where Navigability improves Success Rate and SPL with respect to data augmentation, model (c), by about 10%.

I understand that the number of RGB-D observations with the proposed method is reduced, but this is a choice. The number of env interactions remains the same, so the comparison is on even terms.

This is indeed true, however the navigability loss brings a significant computational advantage at training because the simulator does not have to render RGB-D observations, allowing to speed-up training and reduce hardware and energy requirements.

In the sim2real experiment, was the BC baseline ((c) in Table2) trained with the data augmentation regime, or regular? I believe sim2real results with BC + subgoal augmentation compared against the proposed method would have been ideal here.

Indeed, the BC baseline in the sim2real experiment (Table 1 in the revised manuscript, previously Table 2) is the one with subgoal augmentation. We modified the notation on the table to "(c) BC (L) (S)" to clarify this.

The main text should specify the exact data fed into the networks clearly, this would make it much easier to assess the applicability of the solution. If I understand appendix A.2 correctly, the representation GRU takes both RGB-D images (not just RGB) and positional data (GPS + compass orientation), whereas the main text only talks about visual observations.

The inputs are indeed RGB-D and Goal vector, we clarify this in the revised manuscript.

The probing network is not directly related to the policy, so I am skeptical about the interpretation of the probing results at the very end of sec. 5. I think the paper would be better off by highlighting that this investigation is speculative, and should be taken with a grain of salt.

This was a design choice: the probing experiments are not directly related to the policy, they are designed to probe the representation built in Phase 1 and gain intuitions of what is captured at that stage of training. We agree that we can only speculate about these experiments, this is added to the paper. We also tone down claims related to these experiments.

Thanks for responding! The top-down map figure and the BC (L) + (S) clarification in the new Table 1 were much needed improvements from my perspective, thanks for addressing them. The rest of the clarifications also help.

My original assessment was already leaning positive, the above raises my confidence in that assessment. My current score would be between 6 and 7.

Thank you for the constructive and encouraging remarks!

Sec. 4 reasoning makes sense for GRUs & LSTMs since the gating mechanism in GRUs & LSTMs explicitly encourages the specified behavior. Would this also hold for other architectures, eg. transformer? It'd be useful to provide some insights into this.

The reasoning in section 4 is not limited to any specific form of representing the history of observations of the agent. It can be compressed into a recurrent memory, as in our experiments (GRU or LSTM), or be self-attention over time, like in a decision transformer.

Sec. 4 argues that the training will prioritize learning r1 and neglect r2 and provides two reasons. Is there any empirical evidence for this in the context of navigation?

While compression mechanisms in recurrent memory have been studied in the past (cf. the references we provide in section 4.), up to our knowledge, we are the first to investigate them in the context of navigation.

On the other hand, short-cut learning is a well established concept in the literature in a very general context, establishing that, if the training data contains spurious correlations, trained models tend to learn simpler (mostly unwanted) decision strategies:

• Leon Bottou. “From machine learning to machine reasoning”. Machine learning, 94(2):133–149, 2014
• Geirhos et al. “Shortcut learning in deep neural networks”. Nature Machine Intelligence, pages 665–673, 2020.

In terms of empirical work specifically on navigation, we can refer to the well known “copy-cat” behavior in autonomous driving, where agents with memory tend to behave less well than agents taking only the current observation, e.g. as in

• Wen et al., Fighting Copycat Agents in Behavioral Cloning from Observation Histories, NeurIPS 2020.

In the following work it has been empirically shown that an agent attends to the relevant latent memory slot when a searched (and previously seen) object is not in view, and that attention to memory disappears once the object is in view. While this has only been qualitatively shown (Figure 4 and associated explanations) and not quantitatively evaluated, it provides further evidence for the behavior:

• Beeching et al., EgoMap: Projective mapping and structured egocentric memory for Deep RL. ECML-PKDD, 2020.

For BC(L)(S)(data augm.) baseline in Table 1, when training on the short routes, is the hidden state copied from the long trajectory or initialized separately?

This is variant (c.3), “always continue”, where the hidden state is copied from the long trajectory.

How are the c.1 & c.3 variants in Fig. 6 different from Table 3? The c.3 variant in Fig. 6 is worse than other baselines but it works the best in Table 3. It seems like Always continue works well with navigability but not with BC. It'd be helpful to clarify why the performances are so different.

The reasoning requirements for agents in these scenarios are different. We can only speculate on the reasons: We expect the “always continue” variant to perform less well because of the inconsistency of the hidden state at the sub goals, and this is corroborated by the experiments in Figure 6, which only uses the spatial representation, per construction. However, the full agent presented in Table 3 uses the spatial representation but also reactive components from visual observations, which can learn short-cuts in reasoning. This agent will largely benefit from having more information integrated in its hidden memory, which in this case is not reset at each sub goal.

These are conjectures and it is hard to derive conclusive insights from these probing experiments. We stress this in the revised paper and we will tone down the claims related to the probing experiments.

What does sym-SPL represent? why is the minimum of the two ratios taken?

sym-SPL is a symmetric version of SPL that we introduce in paragraph “Probing the representations”, page 9: $*Sym-SPL* = \sum_{i=1}^D \sum_{n=1}^N S_{i,n} \min \left (\frac{\ell_{i,n}}{\ell_{i,n}^*},\frac{\ell_{i,n}^*}{\ell_{i,n}} \right ).$ The reason to use this measure is that the reconstructed map used by the agent to navigate might erroneously indicate non-navigable space as navigable, leading to SPL larger than 1. For this reason, besides computing the ratio between the predicted path $\ell_{i,n}$ and ground-truth shortest path $\ell_{i,n}^*$ as in standard SPL, we also compute $\frac{\ell_{i,n}^*}{\ell_{i,n}}$ and take the minimum to penalize planned trajectories that go through non navigable spaces.

In other words, and to give an intuitive explanation, compared to standard SPL, to be optimal, not only needs the agent’s path approach the ground truth (GT) “from below”, it should also not “overshoot”.

Thanks for the constructive remarks!

I feel the prediction of future labels without reconstruction is not a very novel architecture. I am also not sure if the improvement in results is because of the “blindness property” in the navigability loss (equation 5) or simply the addition of training data from generated sub-goals.

The main difference between adding the navigability loss and standard data augmentation comes from the fact that the auxiliary agent is blind: the steps on short episodes improve the representation integrated over visual observations on long episodes, whereas classical data augmentation would generate new samples and train them with the same loss. We see this as generating a new learning signal for the existing samples (i.e. on long episodes) using privileged navigability information from the simulator. The experiments with the real robot also clearly show the advantage of this method compared to data augmentation.

There is an experimental discussion section addressing this concern (“Is navigability reduced to data augmentation?”, page 8) where they compare against a behavior cloning agent with added sub-goals. However, the difference in performance ((c) versus (e) in Table 1) seems relatively small (about 1% for success rate and even smaller point margins for SPL) and there is no reported confidence intervals or variance for these results over repeated seeds or trials. Thus, I am unsure how convincing these results are.

Our hypothesis is that the navigability loss improves sim2real transfer because it relies on non-visual navigability information that has reduced sim2real gap compared to RGB-D, used with data augmentation. Therefore it is not surprising that performance in simulation is similar for the two approaches (Table 2 of the revised manuscript - previously Table 1), while we observe significant gains in real-world experiments (Table 1 of the revised manuscript - previously Table 2), where Navigability (e) improves Success Rate and SPL with respect to (c) by about 10%.

Table 2 seemed to be discussed in detail before Table 1 in the experimental section which I found a bit disorienting for readability purposes.

Thanks a lot for the comment, in the revised manuscript we swapped Table 1 and Table 2 to improve readability.

I believe the methods relation to a mole is only explained in appendix A.2. (The auxiliary agent ρ (a.k.a. the “mole”)) and so I was initially a bit confused by the title.

The name “mole” was chosen because moles are blind, as our auxiliary agent is, so we found it to be an evocative image.

On page 8, “Impact of navigability”, the text reads: “We see that PPO (a) outperforms classical BC (b), which corroborates known findings in the literature.” However, the results in Table 1 appear to indicate the opposite, with BC (b) having a higher success rate. Is this a typo or a misunderstanding on my part?

Thanks a lot for the remark, this was indeed a typo and it is corrected in the revised version of the manuscript.

In Table 3, the “c.3. Always continue” variant performs best. However, in Figure 6, now c.1 outperforms c.3. Is there any intuition behind this result?

Table 3 shows results for the complete agents on the PointGoal navigation task, e.g. navigation in 3D photorealistic environments, whereas Figure 6 reports results for the experiments probing the representations after Phase 1 of training, e.g. “navigation” in estimated 2D maps. The reasoning requirements for agents in these scenarios are different. We can only speculate on the reasons: We expect the “always continue” variant to perform less well because of the inconsistency of the hidden state at the sub goals, and this is corroborated by the experiments in Figure 6, which only uses the spatial representation, per construction. However, the full agent presented in Table 3 uses the spatial representation but also reactive components from visual observations, which can learn short-cuts in reasoning not related to the spatial structure. This agent will largely benefit from having more information integrated in its hidden memory, which in this case is not reset at each sub goal.

These are conjectures and it is hard to derive conclusive insights from these probing experiments. We stress this in the revised paper and we will tone down the claims related to the probing experiments.