Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning

We appreciate the insightful and helpful comments from Reviewer vY56. Please find our responses to your concerns and questions below.

Suggestion 1: The tasks are related to diffusion models for long-term planning.

Answer:

Thank you for noting the connection to diffusion-based planning. As references [1,2] were missing, we reviewed related works [r3, r4]. Though differing in approach—trajectory generation vs. step-wise planning—both address long-horizon dependencies in complex tasks. We will revise the manuscript to reflect this connection and consider future integration.

Additionally, if the reviewer can provide [1,2], we will be happy to cite and discuss them in our final version.

[r3] Janner M. et al. Planning with diffusion for flexible behavior synthesis. ICML 2022.

[r4] Mishra U.A. et al. Generative skill chaining: Long-horizon skill planning with diffusion models. CORL 2023.

Weakness 1: More discussion on methodology.

Answer:

We agree that the methodology section could be clearer and will revise it for the final version. In particular, we agree with your proposal to lean more on illustrative figures in the paper. Currently, Figure 6(e) visualizes the learned dynamic latent transition kernel of DT-VIN across several states. We plan to integrate this into Figure 2 and include a side-by-side comparison with an invariant transition kernel to better highlight the advantages of the dynamic approach.

Weakness 2: More discussion on trade-offs between performance gains and the additional computational costs introduced by using deeper networks.

Answer:

We already partially address this in Appendix G, where we found that:

• At the same depth, DT-VIN achieves better performance with a similar or lower GPU memory and training time (Appendix G.1, Tables 14–15).

• DT-VIN’s depth scales almost linearly with planning steps, ensuring manageable complexity as tasks grow harder (Appendix G.2).

To more specifically address your concern, we have produced Tables R1 and R2 (shown below), which explicitly analyze the trade-off between computational cost and performance across different network depths $N$. While deeper networks naturally incur higher computational costs, DT-VIN demonstrates substantial performance gains as depth increases. Specifically, it achieves +81.23% and +99.77% success rate improvements at $N=300$ and $N=600$ over $N=100$, respectively. In contrast, VIN, GPPN, and Highway VIN show little to no improvement despite incurring similar or even higher computational costs.

We will incorporate these new tables into the revised manuscript to clarify the performance–efficiency trade-offs of DT-VIN.

Table R1: GPU memory (GB) and GPU hours (h) during training, on $35\times 35$ maze tasks for varying network depths $N$.

GPU memory (GB):

GPU hours (h):

Table R2: Success rates of different models at varying depths on $35\times 35$ mazes with shortest paths in [200, 300]. See Appendix Fig. 9 for plots.

Q1: Whether planning in continuous action mazes is essentially the same as in discrete ones. May the main difference is the controller.

Answer:

You're generally right—the high-level planning in continuous control tasks is conceptually similar to that in discrete mazes (all planning in reinforcement learning is essentially about producing a sequence of actions—discrete or continuous—that will lead to a desired state or sequence of states).

However, DT-VIN is not simply a stack of a high-level planner and a low-level controller. Instead, it is trained end-to-end to map observations directly to actions, learning high-level planning implicitly from expert demonstrations without requiring high-level planning supervision. This simplifies training and improves robustness.

For example, in D4RL tasks, we supervise using expert control actions (e.g., from a 2-DOF ball or a 8-DOF quadruped robot). While D4RL uses the same $9\times 12$ maze layout for all training samples (varying only the start and goal positions), we evaluate on much larger, unseen mazes ($35\times35$, $100\times100$), making the task significantly more challenging.

We sincerely thank reviewer TBA2 for the valuable comments.

Concern 1: Clarification on state/observation-dynamic in Sec 3.1.

Answer:

To clarify: the observation refers to the maze image input, which is mapped by the model to a latent MDP. Each latent state corresponds to a specific node within this latent MDP.

We distinguish two types of dynamics:

• Latent state-dynamic: As the reviewer correctly noted, in this setup each latent state $(i,j)$ has its own transition kernel $\overline{\mathsf{T}}_{i,j} \in \mathbb{R}^{|\overline{\mathcal{A}}| \times F \times F}$.

• Observation-dynamic: The transition kernel is generated from the input observation $x$ via a learned mapping function: $\overline{\mathsf{T}} = f^{\overline{\mathsf{T}}}(x)$.

These two notions are orthogonal, defining independent aspects of variation in the transition kernel. They give rise to four configurations: latent state-dynamic/invariant and observation-dynamic/invariant (see Table R1). Our ablation study (Sec. 4.4) shows that both components are essential—removing either significantly degrades DT-VIN performance.

We will clarify this distinction in the final version.

Table R1: Four configurations of latent transition kernels based on whether they are dynamic with respect to latent state and/or observation.

Q1: How is suboptimality measured when the success rate is less than 100%?

Answer:

As defined in the paper (Line 220), suboptimality is measured relative to the best solution across all models, including the expert. This is applicable in the 2D Maze Navigation task, where the expert solution can be computed using Dijkstra’s algorithm with access to an underlying binary representation of the maze. For tasks without expert access (e.g., continuous control), only the success rate is reported.

Q2: Why does highway VIN get a better optimality rate with increasing shortest path lengths?

Answer:

Thank you for the thoughtful question. One possible explanation is that longer planning tasks offer a more structured search space, allowing Highway VIN to better propagate informative signals over multi-steps. In contrast, shorter mazes may contain more local optima or distractors, leading to suboptimal choices despite a shorter horizon. As Highway VIN is not our focus, we leave deeper analysis to future work.

Q3: line 260-261 - we examine depths of $N = 600, 5000$ what do you mean? Do you mean in range $[600, 5000]$?

Answer:

Thank you for pointing this out. We meant that we examine depths at two specific values, $N = 600$ and $N = 5000$, rather than a continuous range. We will clarify this in the final version.

Q4: Why is pretraining required in Sec 4.2?

Answer:

Pretraining is necessary because the training dataset for the continuous control task includes only a single $9\times12$ maze layout, differing solely in start and goal positions. However, during evaluation, the model encounters significantly larger ($35\times35$ and $100\times100$) and previously unseen maze layouts, substantially increasing task complexity and requiring strong generalization abilities.

To address this challenge, we leverage pretraining on the diverse and easily accessible 2D maze navigation dataset. This approach helps the model acquire generalizable planning skills, resulting in strong performance with success rates of 98% on the Point Maze and 93% on the Ant Maze tasks (refer to Table 1 in Sec. 4.2 for further details).

While it is feasibly possible for it to solve the task without pretraining, the sample complexity would be high—a property shared with VIN (and VIN variants) here.

Suggestion 1: Minor comments on figure readability and typos.

Answer:

Thank you for bringing this to our attention. We will correct this for the camera-ready version.

We sincerely appreciate Reviewer HD2P's valuable time and constructive comments.

To improve clarity and conciseness, we have reorganized the reviewers’ comments by grouping similar points.

Q1: Experiments are limited to 2D path planning. Evaluating more realistic tasks, integrating visual perception, and discussing broader applications.

Answer:

We thank the reviewer for their thoughtful suggestions and for pointing out [r1]; we will cite and discuss it in the final version. However, we believe there may be a misunderstanding—our submission already includes tasks aligned with [r1] and extends them in several ways:

• 3D Visual Maze Navigation (Sec. 4.1, Additional Experiments, Appendix C.3): Our setup is identical to that of [r1], with both following established protocols from prior work [r2]. As the reviewer noted, this involves “visual navigation with perception input using additional network heads.” Agents must infer maze layouts from noisy, ambiguous first-person visual views, making the task especially challenging.

• Continuous Control Tasks (Sec. 4.2): These involve long-horizon planning with low-level torque control to actuate movement of the agent. While [r1] uses 2-DOF manipulation tasks, we evaluate on a 2-DOF ball and an 8-DOF quadruped robot—sharing the same fundamental task but introducing greater control complexity due to higher dimensionality.

• Rover Navigation (Sec. 4.3): This more realistic task, inspired by the original VIN paper but absent in [r1], involves planning over noisy, incomplete aerial terrain images—adding complexity beyond synthetic 2D gridworlds.

• Scale: Our benchmarks feature much larger mazes (up to 100×100 vs. up to 49×49 in [r1]), posing harder long-horizon challenges.

These tasks—spanning visual navigation, control, and realistic planning—are already described throughout the original submission (Abstract, Introduction, Experiments). Nonetheless, we will revise the final version and cite [r1] to more clearly highlight their generality and realism.

Extending the VIN architecture to broader applications is a compelling direction for future work—previously barred by the limited planning capacity of VIN-based architectures. However, this work aims to focus on the first step in this of demonstrating the performance gains of DT-VIN over related methods, which we view as a necessary precursor to deployment in more applied domains. Indeed, while we are looking into more applied domains currently, such an investigation is of the scale that would necessitate a separate paper.

[r1] Zhao et al., Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation, ICLR 2023

[r2] Lee et al., Gated Path Planning Networks, ICML 2018

Q2: Concern over the impact of relaxed dynamic kernel weight sharing on translation equivariance and performance.

Answer:

Thank you for the thoughtful and constructive feedback.

While our dynamic transition kernel relaxes the weight sharing, it preserves translation equivariance—i.e., shifting the input (e.g., maze image) results in a corresponding shift in the output. This may seem counterintuitive but is straightforward upon closer inspection. This holds because the kernel $\overline{\mathsf{T}}^{\rm dyn} = f^{\overline{\mathsf{T}}}(x)$ is generated by a CNN, which is itself translation equivariant. Consequently, the downstream value iteration process retains this property (see Eq. 2).

To verify this, we conducted a small experiment comparing DT-VIN, a standard CNN, and VIN on 5,000 maze images with translations of 1 to 10 pixels. All models use stride and pooling of (1,1). We measured the L2 error between the output on a translated input and the translated output of the original. Results in Table R1 show perfect translation equivariance across all models.

Additionally, ablation studies in the original submission (Figs. 6a and 6b) compare dynamic vs. invariant kernels. Dynamic kernels consistently outperform their invariant versions, particularly in long-horizon and obstacle-rich settings—highlighting both practical advantages and robustness.

Table R1: Translation Equivariance — L2 Error

Q3: Benefits and limitations of adaptive highway loss.

Answer:

Thank you for the suggestion. The adaptive highway loss avoids the vanishing gradient problem in very deep networks by adding skip connections from intermediate layers to the final loss, guided by the planning trajectory length (computed from training data). This improves gradient flow and stabilizes training (Sec. 4.4, adaptive highway loss).

A potential limitation is computational overhead, which we mitigate by applying the loss every $J$ layers (Eq. 3). This balances efficiency and performance; e.g., $J=10$ significantly accelerating training with minimal performance loss (Appendix F.1).

We will clarify this in the revised manuscript.