Test-Time Adaptation for Online Vision-Language Navigation with Feedback-based Reinforcement Learning

Thank you for your positive evaluation of our work. We are committed to enhancing the quality of the figures in the final version upon acceptance. If there are any further questions or suggestions during the discussion period, we would greatly appreciate the opportunity to address them and further refine our work towards clear acceptance.

Thank you for the detailed feedback. Below, we provide our responses to each comment and hope they contribute to a better evaluation of our work.

Q1. Computational feasibility of FeedTTA

A : We first clarify that FeedTTA does not require high-end server-grade GPUs and can be efficiently deployed on practical hardware (e.g., GTX 1080). The trained DUET policy consists of 181.08M parameters, whereas FeedTTA trains only 78.67M parameters for adaptation—just 43% of the total—making adaptation highly efficient. Furthermore, FeedTTA increases memory usage by only 0.67%, requiring 4.42 GB compared to the 4.39 GB used by baseline VLN models during inference. Lastly, while FeedTTA introduces some latency due to backpropagation, its episodic updates do not impact real-time navigation performance, making it a practical and feasible solution for real-world deployment.

Q2. Can we leverage LLMs to provide more detailed feedback instead of simple binary feedback?

A : As discussed in Section 5.3, LLMs achieve at most 73% accuracy in predicting simple navigational outcomes. While this is sufficient to guide baseline policies, errors can accumulate, hindering stable adaptation. In our response to Q3 of reviewer vESv, we showed that dense step-wise rewards, though less efficient than sparse goal-based feedback at test time, still significantly improve performance. As LLMs advance in navigational reasoning, their potential for richer feedback remains an exciting research direction.

Q3. Could LLM-generated counterfactual evaluations replace SGR?

A : The counterfactual reasoning of SGR is a regularization technique applied on a limited number of parameters, which means that the large portion of parameters should be updated based on proper feedback for intended functionality. Furthermore, while LLMs can indeed reason counterfactual scenarios, their reliability on predicting navigation outcomes itself still remains as a challenge, making them unsuitable as a direct replacement for SGR.

Q4. Can FeedTTA be applied to VN tasks?

A : Yes, FeedTTA can be applied to VN tasks even in the absence of complex language instructions, as it only requires determining success or failure within the navigation system. To identify the dominant modality influencing navigation outcomes, we analyze navigation consistency for each trajectory in the REVERIE dataset, where each trajectory is paired with multiple language instructions. Specifically, we compute the average success rate across different instructions for each trajectory. We then identify trajectories with consistent outcomes—defined as those with a high (> 0.8) or low (< 0.2) average success rate—and calculate their proportion within the validation set. Our experiment yields a ratio of 0.72, suggesting that visual observations are a key factor not only in VN tasks but also in VLN, where they play a more decisive role compared to language variations.

Q5. How does SGR prevent catastrophic forgetting?

A : The expected absolute value (EAV) of the gradients quantifies the deviation from the case where neither forgetting nor adaptation occurs, indicating the extent of policy forgetting and adaptation. For brevity, we omit the dimension index $m$ in subsequent derivations.

In a standard gradient update, the EAV is given by: $\sum \mathbb{E}[|\nabla_{\theta} J(\theta)|] = \sum |g_{\theta}|.$

For small $\alpha$ and $p$ such that $|\alpha| < p$, the EAV for the SGR-modified gradients is: $\sum \mathbb{E}\left[ \left\|\nabla_{\theta} J(\theta)^{\prime} \right\| \right] = \sum \left[ p \left\|\alpha g_{\theta} \right\| + (1 - p) \left\| \frac{g_{\theta}}{\alpha p + (1 - p)} \right\| \right].$

Using a first-order approximation: $\sum \mathbb{E}\left[ \left\|\nabla_{\theta} J(\theta)^{\prime} \right\| \right] \approx \sum \left[ p \left\|\alpha g_{\theta} \right\| + (1 - p) \left\| (1 + p) g_{\theta} \right\| \right] = \sum (1 - p^2 - \alpha p) \left\| g_{\theta} \right\|.$

Applying SGR scales the EAV by a factor of $(1 - p^2 - \alpha p) \leq 1$, reducing the gradient magnitude compared to the standard gradient update.

• $\alpha = 0$ (Gradient Dropout):
  The scaling factor is fixed at $1 - p^2$.
• $\alpha > 0$ (Gradient Scaling):
  The scaling factor is controlled by $\alpha$, but remains bounded above: $1 - p^2 - \alpha p < 1 - p^2.$
• $\alpha < 0$ (Gradient Reversion):
  The scaling factor is controlled by $\alpha$, bounded both above and below: $1 - p^2 < 1 - p^2 - \alpha p \leq 1.$

This result demonstrates that reversing a subset of gradients as proposed in SGR provides a strategic way to balance plasticity and stability in adapting to unseen environments.

Thank you for your positive evaluation of our work. We hope our response fully addresses all concerns and demonstrates the strength of our contributions.

Q1. How does different sequence orderings affect adaptation?

We agree that online learning is sequence-dependent. However, we show that the benefits of FeedTTA is invariant to sequence ordering through the following experiments with three different configurations. We use the 'validation unseen' split of the REVERIE dataset and compare with the DUET policy. For all configurations, the reported numbers of FeedTTA are the average of the results from 3 different seeds, with standard deviation reported in brackets.

1. General TTA

• In this configuration, all episodes are randomly ordered regardless of scene IDs, which corresponds to the experimental setting reported in Table 1 of our paper.

2. Per-Scene TTA

• Here, we analyze the effects of random episode orders for each scene ID. Note that in this setting, the adaptation is performed per-scene, and not throughout the entire validation set. The results below are in the form of (DUET / +FeedTTA)

3. Continual TTA

• For this configuration, we fix the episode orders for each scene ID, and set the adaptation sequence based on mixed scene ID orders, evaluating continual adaptation performances across different scenes.

These experiments confirm that sequence ordering does influence navigation outcomes; however, the benefits of FeedTTA remain consistent, as evidenced by superior performances with low variations across different seeds.

Q2. Does increased trajectory length represent beneficial exploration?

A : We justify that the increased trajectory length (TL) indicates beneficial exploration by empirically testing the hypothesis: “The overall increase in TL primarily results from episodes that would have failed in the original navigation but succeeded after applying FeedTTA”. In the table below, we compare the increase in TLs for the successful navigation episodes after adaptation, categorized based on the pre-tested results before applying FeedTTA. For this experiment, we use the 'validation unseen' split of the REVERIE dataset with DUET as the base policy. Here, we discover that the average TL increase is significantly larger for fail-to-success cases than success-to-success cases. This clearly highlights the role of FeedTTA in overcoming failure cases through extended exploration in unseen environment.

Q3. How does FeedTTA compare to approaches that use more informative feedback (beyond binary signals)?

A : The rationale behind choosing a simple binary episodic feedback mechanism stems from the practical limitations of the online test-time navigation environment:

• Human involvement should be minimal, as following every navigation steps to provide rewards is infeasible in real-world environment.
• Reward systems used in offline learning (e.g. step-wise distance-based rewards ) are infeasible at test-time, as we assume no access to ground-truth goal position or pre-defined maps.

We empirically evaluate the efficiency of the feedback system by comparing our method with the step-wise distance-based reward system used in HAMT, where the feedback is defined as the reduction in distance to the target at each step. Additionally, if the agents successfully arrives at the goal positions, 2 is given as a success signal and otherwise -2 as a penalty. As we observe from the table below, our binary episodic feedback surpasses the distance-based dense reward system, even without access to ground-truth information. This clearly demonstrates that the proposed feedback mechanism appears to be simple, yet efficient and effective in improving navigation performance.