Vision-Language Navigation with Energy-Based Policy

Q1: The improvement of ENP is lower than that of semantic grid maps.
A1: First, respectfully, the improvement of 1-2% on both SR and SPL is non-trival in VLN. This point was acknowledged by the other reviewers, such as "work pretty well" (Reviewer UeCa) and "achieving obvious improvements" (Reviewer Ngjf). Second, although we agree that semantic maps [54-57] are powerful tools, ENP improves the performance from different perspectives. ENP learns an unbiased expert policy by exploiting information from the training data (Eq. 3), while semantic maps introduce additional scene information, e.g., [55] encodes spatial priors by grid maps. Third, the below table shows ENP requires significantly less GPU memory than [55] (DUET-based [6] framework).

Therefore, it is unfair to compare ENP directly against the semantic maps. More importantly, ENP is a flexible solution and can be seamlessly integrated into existing methods without extra inference time. The following table indicates that ENP is complementary to [55] and further enhance the performance.

Q2: "... when ENP is better than BC ..."
A2: First, in Tables 1-4, ENP can achieve consistent performance gains across different datasets. Second, we explore ENP and BC across different trajectory lengths in Fig. S1 of the "global" response PDF. In Fig. S1 (a) and (b), ENP exceeds BC by 6% and 9% on R2R val seen and val unseen (16-20m). Third, we further validate the effectiveness of ENP on R4R [ref-3] with the longer paths in the following table (also in Fig. S1 (c) and (d)). ENP surpasses BC by 6.40% and 9.03% on R4R val seen and val unseen (>20m), and the gains in unseen environments are relatively more significant.

We will add these figures and tables to the Appendix.

[ref-3] Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation. ACL 2019.

Q3: A benchmark with longer trajectories.
A3: We clarify that we have conducted experiments on R2R-CE [32] and RxR-CE [33] in Tables 3 and 4 (L283-292), which involve significantly longer action trajectories (56 vs. 5 steps in R2R). In addition, per your request, we report the results on R4R [ref-3] with longer paths. CLS [ref-3] measures how closely the trajectory aligns with the ground truth. ENP achieves 3.52% and 3.11% gains on SR and CLS of val unseen.

Q4: A detailed analysis of success/failure cases.
A4: Thanks for your feedback. We include more success/failure cases in the PDF material of "global" response (Fig. S2). In Fig. S2 (a), our agent with ENP successfully reaches the final location, whereas DUET fails due to the accumulation of errors over long paths. In (b), both our agent and DUET struggle to locate the target due to the lack of a specific landmark recognition module like [50,51]. This figure will be added to Fig. 3.

Q5: Training time for ENP.
A5: We report the training time for ENP with different SGLD iterations I on a NVIDIA RTX 4090. In practice, we find that $I_{ft}=5$ SGLD loops per training iteration is good enough for model convergence (Algorithm 1), introducing a minor computational overhead, approximately 4 minutes delay for 1K training iterations.

Q6: The comparison with recent methods.
A6: Sure, we will reorganize Tables 1-3 and highlight the comparison with the most recent methods. The complete tables will be included in the Appendix.

Q7: "... the DUET paths not shown in Figure 3b ..."
A7: To clearly present the comparison, we only include two navigation trajectories in each figure for better understanding (Fig. 3). We will include the comparison results of these methods in the Appendix and provide further analysis.

Q8: The standard deviation ($\pm$) in tables.
A8: Our apologies. "$\pm$" in Tables 1-4 denotes the standard deviation after repeating the experiments with different random seeds. Unfortunately, the corresponding standard deviations for other methods were not provided in their original papers.

Q9: "... increasing $I_{pre}$ and $I_{ft}$ results in a lower SR/SPL (Table 7)"
A9: Since there is a domain gap between seen and unseen environments [3,9], increasing SGLD iterations ($I_{ft}$) may lead to overfitting on val seen, resulting a lower SR/SPL on val unseen. Additional results on val seen for Table 7 are provided in the following.

Q10: More discussion about the limitations.
A10: One limitation of ENP is that the optimization through the SGLD inner loop at each training iteration (Algorithm 1). It would reduce the training efficiency (about 9% delay for 5 SGLD iterations). In practice, ENP balances performance and training time with a limited number of SGLD iterations. We will provide more analysis in the Discussion.

Q1: Evaluation on ScaleVLN.
A1: Per your request, we conduct additional experiments on ScaleVLN [49] using the original hyperparameters of ENP (L226-228).

As seen, ENP outperforms DUET with the same training data from ScaleVLN, e.g., 1.40% and 2.25% SR gains over val unseen and test unseen, respectively.

Q2: "IL/BC is much more friendly and stable than ENP to scale with data."
A2: First, we have validated the effectiveness of ENP on large-scale training data in Q1. We agree that classic behaviour cloning (BC) is straightforward and easy to train for sequential decision-making in VLN. However, as mentioned in many studies [13,16,29], BC suffers from accumulation errors during both training and inference due to covariate shift (L25-35). Second, ENP utilizes persistent contrastive divergence (PCD) [73] to train the energy-based models (L182-190). The stability of PCD has been widely demonstrated in various tasks, such as classification [76], generation [23], reinforcement learning [77,78], and natural language processing [ref-1]. Third, the training curves provided in the PDF of "global" response (Fig. S3) clearly show that our ENP maintains a stable training process on both the original R2R data (14K) and ScaleVLN (4.9M). We will clarify this in the Experiment of the final version.

[ref-1] Residual energy-based models for text generation. ICLR 2020.

Q3: "It is unclear if the method will still benefit larger models ..."
A3: It is interesting to investigate ENP on larger models. We noted that NaviLLM [62], which is built updon a large language model (Vicuna-7B [ref-2]), uses the similar pre-training and finetuning paradigm [6] with BC, so ENP can be seamlessly integrated into this framework. Due to the limited time, we incorporate ENP ($I=15$, $\epsilon=1.3$, $Var(\xi)=0.01$) during the finetuning stage. From the following results, we observe that ENP gives 2.43% and 2.78% improvements on SR of R2R val unseen and SR of REVERIE val unseen, respectively. More results will be provided in the Experiment of the final version.

[ref-2] Instruction tuning with GPT-4.

Q4: More analysis on successful/failure cases.
A4: We provide more analysis of ENP vs. DUET on success/failure cases in the PDF material of "global" response (Fig. S2). As illustrated in Fig. S2 (a), our agent with ENP successfully reaches the final location, whereas DUET fails due to the accumulation of errors over long paths. In Fig. (b), both our agent and DUET struggle to find the target landmark due to the lack of a specific landmark recognition module like [50,51]. We will incorporate this figure into Fig. 3 in the final revision.

Q5: "... in what scenarios an agent trained with ENP shows an advantage."
A5: First, in Table 3 and 4, ENP achieves consistent performance gains in both seen and unseen environments across different datasets (R2R-CE [33] and RxR-CE[37]). Second, the gains in unseen are relatively more significant than that in seen. For example, ENP achieves 5.18% gains of SR on R2R val seen (>10m) and 9.35% gains on val unseen (>10m) as follows. Third, we further observe that our ENP gives more improvements on challenging cases with long trajectories as follows. ENP demonstrates its capability on R4R [ref-3], where the average path length is about twice that of R2R. The average success rate of ENP exceeds that of BC by 6.40% on val seen (>20m). The performance gain can also significantly reach up to 9.03% on R4R val unseen (>20m).

This indicates that ENP alleviates the cumulative errors in sequential decision-making (L200-210). In the PDF of "global" response, we provide a detailed analysis of success rate across different path lengths (Fig. S1). We will include it in the Appendix of the final version.

[ref-3] Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation. ACL 2019.

Q6: More explanation about energy-based models (EBMs).
A6: Here, we analyze the EBMs from the perspective of sequential prediction [ref-4]. Existing VLN models are based on locally normalized models which optimize the prediction through maximum likelihood at each step. To multigate the exposure bias [ref-4], EBMs define a global energy function at the sequence level, where lower energy values correspond to higher probabilities [21]. Then the normalized probability density can be obtained through dividing by the partition function (Eq. 4). Thus, estimating the partition function is crucial for training EBMs. There is a long history in machine learning [20] for training EBMs. We adopt persistent contrastive divergence [22,73] to sample the partition function and optimize the EBMs (L182-190). We hope it will help you understand more clearly. We will add a detailed background of the EBMs in the Related Work.

[ref-4] Sequence level training with recurrent neural networks. ICLR2016.

Thanks to the authors for providing a clear and solid response to my concerns. I am happy with the additional results that justify the data and model scalability and successful/failure cases (questions raised by more than one reviewer), given the limited rebuttal time. I decided to raise the rating from Weak Accept (6) to Accept (7). I sincerely hope that the authors can further extend and incorporate all responses into the final version of their paper.

Dear Reviewer Ngjf,

Thank you very much for your valuable time and thoughtful review. We are pleased to hear that our rebuttal addressed your concerns, prompting a rise in the rating. We will revise our paper according to your comments and incorporate all suggestions into the final version.

Thanks again,
Authors

Q1: "... whether the proposed approach also benefits VLN agents when having large-scale data."
A1: Thank you for your comments. The proposed ENP is a general framework to mitigate the accumulation of errors in the sequential decision-making process of VLN. To demonstrate the effectiveness of our ENP on large-scale data, we conduct experiments on ScaleVLN [49], without image augmentation [48] as follows.

As seen, ENP gives consistent improvement with large-scale training data (i.e., 1.40% and 2.25% SR on val unseen and test unseen, respectively). Please note that all the experiments above utilize the original hyperparameters of ENP (L226-228). We will update this result in Table 1 of the final version.

Q2: "The approach might be sensitive to hyperparameters as suggested in Table 6 and Table 7 ..."
A2: We clarify the robustness of ENP to hyperparameters based on the following aspects. First, as shown in Table 6 and Table 7, ENP achieves superior performance with consistent hyperparameters for the same model across different tasks (e.g., R2R [1] and R2R-CE [32]). Second, ENP is insensitive to parameter changes within adjacent ranges. Due to limited space in the paper, we reported only a subset of the results in Tables 6 and 7. For Table 6, we provide detailed results of $\epsilon$ as follows. We find that it has minimal impact on the performance when $\epsilon$ varies between 1.3 and 1.9.

In addition, more experimental results related to Table 7 are presented below. Obviously, when the hyperparameter $I$ is between 15 and 19, the performance of ENP-VLNBERT remains stable.

We will definitely add these detailed tables to the Appendix in the final version.