TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies

Thank Reviewer qxt2 for the insightful feedback and for acknowledging TraceVLA's technical novelty and empirical contributions. We address your questions and concerns in detail below.

The choice of active points depends on a threshold. Does this threshold need to be chosen differently for each robot or dataset?

We use a consistent threshold in CoTracker to track motions across different robot datasets during training, ensuring effective tracking without requiring dataset-specific tuning or embodiment-specific adjustments during inference. The threshold is chosen to strike a balance between precision and recall, ensuring that the selected active points are most indicative of motions relevant to manipulation tasks.

In practice, setting this threshold is straightforward, as the movement magnitudes of active objects and static backgrounds generally differ significantly for the majority of trajectories. Any value within this range reliably separates points on the robot end-effector and active objects, making the approach robust across diverse datasets and embodiments.

The authors mention that a longer trace length can obscure key information. However, in the method, they state that they mitigate this by inputting both the original and overlay images. In that case, a longer trace length should not lead to worse results. This explanation seems self-contradictory.

Thank you for this insightful question. Since we use 2D traces in the 2D image inputs, longer trajectories not only will obscure more observations, but more importantly may partially overlap previous motion traces of robots in the trace overlaid images, which could make the visual trace distracting and non-informative.

For fine-tuning OpenVLA with history images, six history frames work best for TraceVLA, but this might be too redundant when using image. Would using three or two history frames be better when using history images as input?

We have conducted additional experiments fine-tuning OpenVLA with fewer history frames (2 or 3 frames), which we include in Figure 11 of Appendix B.2. While setting step=2 results in a slight performance gain (0.8%) over baseline OpenVLA, TraceVLA significantly outperforms these variants, demonstrating the effectiveness and efficiency of our trace representation in capturing temporal and spatial information.

Does the dataset used for fine-tuning OpenVLA contain scenes similar to those in simplerEnv? That is, are all the testing scenes entirely unseen in the training dataset?

It is an important question about dataset composition. The tasks in SimplerEnv are not present in the fine-tuning dataset, but use the same robot embodiments (WidowX and Google Robot). We fine-tuned both models using the same real-robot dataset, since differences in the physical setup (like backgrounds, camera angles, and lighting conditions) make it difficult for any model to work immediately without training (as stated in line 338-343). Even with identical fine-tuning conditions, our model demonstrates stronger generalization capacity compared to baselines.

To further evaluate generalization, we conducted additional real-robot experiments with four unseen tasks, specifically designed to test adaptation to novel objects, goals, language instructions, and motion scenarios. As shown in Figure 12, TraceVLA significantly outperforms baselines in these challenging scenarios, exhibiting stronger language grounding capability and generalizes beyond spurious correlations.

We appreciate the constructive feedback that has helped improve our paper's quality. Additional results will be included in the Appendix. We sincerely hope our responses have adequately addressed your concerns and look forward to any further discussion.

Thank Reviewer 4XPZ for the thoughtful feedback and valuable suggestions. We appreciate Reviewer 4XPZ's acknowledge of TraceVLA’s technical and empirical contributions. Below, we provide detailed responses to each concern and question.

It would be better to evaluate the proposed method in more generalization settings, including novel backgrounds and more novel tasks.

Thank you for this valuable suggestion regarding generalization evaluation. In our real-robot experiments, we have already added randomized background distracting objects for each trial to test model's robustness under novel background. Additionally, to test more novel tasks, we have expanded our experiments in two key aspects:

• We conducted additional real-robot experiments with four unseen tasks designed to test generalization across novel objects, goals, language instructions, and motion scenarios in Appendix C. As shown in Figure 12, TraceVLA demonstrates significantly better generalization compared to OpenVLA when handling these novel elements.
• We added qualitative visualizations in Appendix D to provide detailed insights into our model's behavior across different generalization scenarios. These results further validate the effectiveness of our visual trace prompting technique.

RT1-X, despite having a much small number of parameters, outperforms TraceVLA-Phi3 with 3B parameters in SimplerEnv. The advantage of TraceVLA with 7B parameters is also not substantial. Is it possible to provide more discussions on potential reasons? Additionally, it would be beneficial to include RT1-X in real-robot experiments for comparison.

Thank you for this insightful question. We conducted a detailed evaluation of RT1-X by fine-tuning it on our real-robot dataset using the officially released RT1-X training code.

In our real-robot experiments, RT1-X exhibited the following performance: for in-domain tasks like Fold Cloth, RT1-X was able to detect and track objects but struggled to complete the full task successfully. For out-of-domain tasks, it failed entirely, resulting in a zero success rate across all tested tasks. We attribute RT1-X's weak generalization capability in real-world scenarios to two key technical limitations:

• Weaker Language Understanding: RT1-X employs the Universal Sentence Encoder for language embeddings, which offers limited language comprehension. This often led to failures in correctly identifying target objects. For instance, in a pick-and-place task involving a banana, RT1-X misinterpreted the language instruction and attempted to grasp the plate instead of the banana.
• Limited Visual Feature Extraction: Compared with TraceVLA and OpenVLA which uses a fused DINO-SIGLIP vision encoder, RT1-X uses an EfficientNet-based vision encoder, which demonstrates suboptimal visual feature extraction capabilities in standard computer vision benchmarks. This resulted in imprecise grasping and repetitive basic pick motions without task completion.

These findings underscore the superior generalization of VLM-based approaches like TraceVLA, which leverage robust language and visual encoders for handling complex real-world scenarios.

The proposed method leverages CoTracker to provide visual traces. Were any failure cases due to inaccurate tracking observed during experiments? How robust is TraceVLA in handling these inaccuracies?

Indeed during our experiments, we a few failure cases due to tracking inaccuracies, particularly in scenarios where the background and objects or the robot share similar colors, or in cases of partial occlusions. However, our method incorporates multiple mechanisms to handle such challenges:

• Dual-input design: TraceVLA leverages both original images and trace overlays. This ensures that critical visual details from the original image remain intact, allowing the model to disregard irrelevant or inconsistent traces caused by tracking inaccuracies.
• Multi-point tracking: By tracking multiple points simultaneously, we maximize the likelihood of maintaining robust tracking for the robot, even when some points on the background are mistakenly included.
• Periodic reinitialization: We periodically restart tracking to ensure traces' quality and robustness, allowing the tracking to recover from errors caused by background motion.

Thank you for your insightful feedback and review! We are encouraged that you think our visual trace prompting technique is novel and our ablation study across various design choices is thorough. Below, we address your concerns and questions in detail.

Sec 4.1 is a bit hard to read. Mixing figures, tables, and your main context together with reduced blanks is not a good idea to present your experiment results clearly.

Thank you for your suggestion. We have reorganized the figures, tables and texts in Section 4.1 for better readability.

Lack of details in experiments. For example, how do you define success, especially in real robot experiments? When you add noise to the environment, do you have a limit over related parameters? Is there any case study that qualitatively shows the advantage of applying visual tracking prompts? Please add them to your revision, which is also good for reproducibility.

We have added comprehensive experimental details in Appendix A, including: success criteria for each task in real-robot experiments, environmental initializations and description of distracting objects, and detailed evaluation protocols for both simulation and real-world settings. Additionally, we have added a qualitative visualizations of TraceVLA against baseline OpenVLA in Appendix C.

How does this method generalize under camera orientation changes? By nature it helps the model against illumination/background changes, but how can VLA remain effective when the camera moves, given only 2D tracking result? If the change is small, then to what degree do you vary?

We would like to clarify that, in this paper, we follow the settings of existing robot generalist policies such as OpenVLA, Octo, and RT1/RT2-X. Specifically, we use a third-person camera view for observations, which is mounted and remains stationary throughout each episode.

However, we would like to highlight that our visual trace prompting technique is versatile and can be extended to handle egocentric views. To address camera motion in such cases, a homography transformation can be applied to align historical traces with the current observed frames. This involves calculating a 3×3 homography transformation matrix between points in a historical frame (e.g., at time $t-k$) and the current frame (time $t$). Using this matrix, we can transform historical points to align with the current frame, thus enabling the generation of visual traces for moving robot arms and/or objects while compensating for camera motion.

A concrete example demonstrating this approach is available here, where we apply homography transformation to an egocentric video from the EpicKitchen dataset.

Extending our visual trace prompting to egocentric videos, and potentially leveraging large-scale action-free human videos, is indeed an exciting direction for future work. However, this exploration falls beyond the scope of the current paper and is left as a future research direction.

You mentioned in the real robot section that finetuning is needed due to domain shift, but not for the simulation. Is that also due to the domain gap being small in simulated experiments?

Regarding the need for finetuning in real robot experiments but not in simulation, this difference stems from broader domain gaps beyond just camera variations - real-world scenarios introduce additional challenges like lighting variations, sensor noise, and imperfect actuation that aren't present in simulation. The simulation experiments primarily test geometric viewpoint variations in a controlled environment, making the domain gap relatively smaller and manageable without additional finetuning, while real-world deployment requires bridging multiple domain differences simultaneously.

Thank you for your thoughtful and detailed review! We are encouraged that you think our visual trace prompting technique is novel and our experimental result is solid with extensive empirical evaluations. Below we address you questions in details.

While the author presented results highlighting the importance of historical information from the trace, there were no results demonstrating spatial reasoning capabilities as claimed in the "spatial-temporal" contribution. One potential approach could involve fine-tuning a Vision-Language Model (VLM) for line-drawing tasks, connecting it to low-level policy, and leveraging the VLM independently for spatial reasoning, similar to the RoboPoint framework.

Thank you for your insightful question. We believe the spatial reasoning capability of our approach is demonstrated through the visual grounding of the VLA model, which bridges 2D visual trace prompting with the 7-dimensional 3D action execution. This capability is quantitatively validated in the success rates reported for both simulated and real-world robotics benchmarks.

We appreciate your suggestion of RoboPoint as relevant related work and have incorporated it into our related work section. While RoboPoint fine-tunes a Vision-Language Model (VLM) to predict 2D keypoints and uses depth maps and motion planners to execute actions, our method, directly fine-tunes the VLA model to produce action tokens. The core focus of our method lies in efficiently encoding historical movements to enhance spatial-temporal awareness, particularly in leveraging past visual and action information for improved decision-making. This emphasis distinguishes our approach from RoboPoint, whose primary goal is keypoint prediction combined with motion planning.

While our current implementation does not explicitly include keypoint prediction, integrating such auxiliary tasks with our VLA framework could be an intriguing direction for future exploration. This could further enrich the spatial reasoning aspect of our approach by combining explicit keypoint reasoning with direct action token prediction.

The author relied entirely on real-world data to generate the trace, which may have introduced variability from environmental factors such as lighting. Generating trace data in simulation could mitigate these issues, potentially minimizing the sim-to-real gap.

In this work, our primary focus is on learning a generalist robot policy directly from real-world data, where diversity and variability in the training process are viewed as beneficial for improving robustness. The use of SimplerEnv Eval serves solely to validate the performance of our trained policy on real-world data, as the exact physical setup of the Google Robot dataset cannot be replicated. Addressing the sim-to-real gap is not within the scope of this work, as our approach exclusively trains policies on real-world data, with SimplerEnv functioning as a controlled evaluation metric rather than a simulation-to-real bridging tool.