Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agents

Thank you for your thoughtful feedback! Below, we provide point-by-point responses to each of your questions.

W1. Weak connection between theoretical proofs and practical guarantees.

Thanks for your suggestions. Our theoretical analysis aims to offer insight into how our self-supervised objectives induce ordered and continuous representations, moving beyond traditional goal-reaching assumptions. While these proofs do not provide direct guarantees on model behavior in practical deployments, our empirical results demonstrate that the theoretical properties are indeed reflected in real-world performance.

Specifically, Theorems 1 and 2 establish that the learned feature space is ordered and continuous, which directly supports real-world imitation learning by enabling more accurate and data-efficient behavior cloning in continuous action spaces. This structure also supports generating dense progress rewards that can effectively identify task boundaries, benefiting real-world reinforcement learning. Additionally, Theorem 3 guarantees robustness to linguistic perturbations, an important property for real-world robotic applications involving natural language instructions. We will further clarify these connections and provide additional discussions in the revised version.

W2. Scalability to long or hierarchical sequences.

Thanks for raising this meaningful concern. The global temporal coherence in AcTOL is primarily maintained through the Vision-Language Ordering (VLO) loss applied during pretraining. This loss samples frames randomly from each video within a batch and encourages consistent temporal ordering. For very long videos, it is true that more batches may be required to cover most frame pairs. However, it is neither necessary nor efficient to optimize over every possible frame pair. Instead, training should be viewed as optimizing the expected VLO loss across all possible random batches. Once this expectation is sufficiently minimized, Markov's inequality ensures that the loss for any batches will also be low enough with high probability. This statistical property allows AcTOL to scale gracefully even as sequence length grows.

Regarding hierarchical or multi-task videos, such sequences can be naturally decomposed into shorter sub-trajectories using a high-level planner, such as LLMs/VLMs. Each sub-trajectory can then be modeled effectively by AcTOL using the local continuity and ordering constraints. This modular design makes AcTOL compatible with hierarchical planning and scalable to longer video sequences in practice.

Q1. Limited discussion of broader applicability.

Thanks for the suggestion. While AcTOL is primarily demonstrated on manipulation tasks, its vision-language-aligned representations are general and could support other embodied AI domains such as navigation and planning. To apply AcTOL in these settings, the model would need to be pretrained on video-instruction pairs that are specific to navigation behaviors or planning scenarios. This ensures that the learned representations capture the appropriate domain-specific semantics and temporal patterns.

For navigation, AcTOL could be trained on egocentric video sequences paired with spatial instructions such as “go to the sink” or “walk to the hallway.” For planning, AcTOL can support high-level reasoning by evaluating the semantic progress of proposed visual plans or subgoals. We will discuss such potentials in revised version and view this as a promising direction for future exploration.

Thank you for your thoughtful feedback! Below, we provide point-by-point responses to each of your questions.

W1. Unclear justification for ordering and continuity as effective supervision.

Thank you for raising this point. We attribute AcTOL’s ability to provide meaningful visual reward signals to two key factors: the pretrained CLIP embeddings and the self-supervised nature of our objectives. As shown in Figure 5, CLIP features already encode meaningful distinctions between different states in a visual trajectory, especially the initial and goal states. This is due to large-scale vision-language pretraining. However, these features are not naturally ordered or continuous throughout the entire trajectory.

By enforcing temporal ordering and continuity, our self-supervised objectives structure the CLIP feature space to better reflect the progression of task execution. These constraints guide the representations to unfold in an ordered and smooth manner between the already well-separated initial and goal states，without relying on strong goal-conditioned supervision as in prior methods. As a result, the learned representation aligns well with task progression and can serve as a dense visual reward signal for downstream control tasks. We will clarify this point more thoroughly in the revised version of the paper.

W2. Lack of quantitative support for action-instruction alignment.

Thank you for the helpful suggestion. To quantitatively assess whether AcTOL retains CLIP’s ability to distinguish between actions associated with different instructions, we evaluate the clustering quality of visual features on the EPIC-Kitchens dataset. Specifically, we selected videos from 50 different action classes, each associated with a unique instruction. For every video, we extracted frame-level features using both CLIP and AcTOL, and aggregated them using temporal mean pooling to obtain a single feature per video. Then we adopt two widely used unsupervised clustering metrics: the Davies–Bouldin score (lower is better) and the Calinski–Harabasz score (higher is better). As shown below, AcTOL achieves better scores than CLIP on both metrics, indicating improved intra-class compactness and inter-class separation:

These results suggest that AcTOL preserves the discriminative structure of CLIP representations with respect to action-instruction alignment.

W3. Missing zero-shot visualization and analysis on main benchmarks.

Thank you for the helpful suggestion. Since we are currently limited to text responses during the rebuttal phase, we will include more visualizations in the revised version. To partially address this concern here, we provide a representative table below that shows the zero-shot visual reward produced by AcTOL on our real-world robot data.

In this experiment, we evaluate the temporal evolution of visual reward for the "open first drawer" task, using two instructions: one correct ("Open first drawer") and one distractor ("Open second drawer"). The normalized reward is measured at different timepoints (20% to 100% of the trajectory).

As seen in the table, the reward for the correct instruction (1st row) steadily increases as the task progresses, peaking near completion. In contrast, the reward for distractor instruction (2nd row) initially grow due to visual similarity in the early motion but diverges in later stages, as the action clearly corresponds to a different drawer. This trend highlights AcTOL’s ability to assign fine-grained, temporally-aware rewards that distinguish between correct and incorrect tasks over time, even in a zero-shot setting without fine-tuning.

W4. Limited complexity in real-world evaluation.

AcTOL is theoretically capable of handling more complex tasks; however, doing so requires significantly more demonstrations with high quality. As we describe in Appendix B.2 (line 591), collecting demonstrations with our D1 arm is expensive because it currently relies solely on remote control. To address this limitation, we are exploring low-cost teleoperation methods to enable more scalable data collection, which would allow us to explore more diverse and complex tasks in future work.

It's also worth noting that our current experiments require fine-grained understanding of the scence layout. For example, in the "open [X] drawer" task, we use a drawer setup with a complex layout. We collect 60 demonstrations in total, 15 for each drawer where X ∈ {"first", "second", "third", "fourth"}. During evaluation, the model is given a randomly sampled drawer instruction and must roll out accordingly. This setup demands fine-grained semantic understanding of the target drawer, linking language (e.g., "third drawer") to the corresponding visual context.

Additionally, we have already added experiments under visual domain shift conditions in the Franka Kitchen environment, please refer to our response to Reviewer h3Cm’s W1 for details. We hope this can partially address your concern.

Thank you for your response. It has addressed all my concerns. I have no further questions. I raise my rating to 4.

Thank you for your thoughtful feedback! Below, we provide point-by-point responses to each of your questions.

W1. Limitation with cyclic or repetitive actions.

Our method relies on the assumption that actions progress in a temporally ordered manner. As we have mentioned in Limitations (line 736), repetitive, such as dishwashing (e.g., scrub, rinse, scrub), may partially violate this assumption and could pose challenges for our current formulation.

However, most real-world tasks, particularly in household robotics, tend to follow a primarily ordered structure. Our approach is therefore well-suited for such scenarios, as demonstrated by the empirical results. We appreciate the suggestion and consider handling repetitive or cyclic behaviors an important direction for future work.

W2. Spatial perception and fine-grained affordance issues.

We agree that spatial perception plays a significant role in geometrically complex tasks. While CLIP-based encoders offer strong semantic priors, they may lack the fine-grained spatial precision needed for tasks like cup-picking.

However, our work primarily focuses on improving the temporal ordering and continuity of visual-language representation, rather than object- or pixel-level spatial understanding. That said, enhancing spatial perception is a promising direction to complement our approach. For instance, some feature-level super-resolution techniques like Featup [1] could be explored to improve the spatial granularity of CLIP embeddings, combining our method to achieve temporal and spatial optimization.

[1] FeatUp: A Model-Agnostic Framework for Features at Any Resolution, ICLR'24

Q1. Scalability to long-horizon, multi-subgoal tasks.

AcTOL is not specifically designed for long-horizon tasks with multiple subgoals, but it can be naturally extended to such settings by leveraging a Large Language Model (LLM) as a high-level planner, like [2]. The LLM can decompose a complex instruction (e.g., "make coffee") into a sequence of shorter sub-tasks (e.g., "pick up kettle" → "fill water" → "boil" ...), each of which can then be independently handled by AcTOL.

Regarding the scalability of the Brownian Bridge loss, Figure 4 demonstrates that AcTOL maintains smooth transitions even between distinct sub-tasks. This continuity property is expected to generalize to longer sequences of consecutively executed tasks. We will include additional visualizations in the revised version to further support this point.

[2] Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents, ICML'22

Q2. Hyperparameter sensitivity analysis ($\lambda$).

Due to space constraints, we included the sensitivity analysis of the hyperparameter $\lambda$ in Appendix A (line 570). As shown in the results, our strategy is verified agnostic to the choices of $\lambda$ varying from ${0.01}$ to ${1}$. We chose $0.1$ to ensure a reasonable scaling between the ordering and continuity losses.

Q3. Visual domain gaps between human videos and real-world deployment.

Thank you for raising this important point. There is indeed a visual domain gap between human-centric videos and third-person robot observations. AcTOL can addresses this challenge in belowing ways.

• AcTOL learns a general inductive bias of temporal ordering and continuity from vision-language pretraining. This bias focuses on the progression of visual states over time, rather than on specific visual details. As a result, the learned features can generalize across different embodiments. For example, although a robot gripper may look different from a human hand, both follow similar patterns of interaction when performing tasks like opening a drawer or picking up an object. This generalization ability is demonstrated in Figures 4 and 9, where AcTOL successfully produces meaningful reward signals from real-world robot videos, even when trained on human videos.
• During downstream policy learning such as behavior cloning, we find only fine-tune the model using a small number of robot demonstrations can largely mitigate the domain gap. To prove this, we first take 25 in-domain demonstrations (5 per task) in Franka Kitchen to fine-tune the pre-trained encoders using the AcTOL objective. Then, as before, we freeze the fine-tuned encoders and train policies on top using behavior cloning. We report the comparisons when using 15 demos for policy training. As shown in the table below, the success rate improvement also demonstrate that the learned temporal inductive bias can be effectively adapted to the robot domain with limited supervision.

Thank you for your thoughtful feedback! Below, we provide point-by-point responses to each of your questions.

W1/Q2. Robustness under visual shifts.

We appreciate the reviewer’s concern regarding the limited diversity in our real-world experiments. Due to limited time, we were unable to collect additional data and complete a full evaluation of visual shift robustness in the real-world setting. To partially address this concern, we have conducted additional experiments in the Franka Kitchen environment by introducing visual distribution shifts following the setup in reference [1]. In particular, we evaluate AcTOL and the strongest baseline method, DecisionNCE, under various visual changes that are not present in the training data. These changes include:

• Object distractors of increasing difficulty: easy, medium and hard level, corresponding to scenes containing 1, 3, and 9 distracting objects from YCB object set, respectively.
• Texture variations in the background: marble hinge texture and metal slide texture.

We use 15 demonstrations per task for policy training. The success rates averaged over 5 tasks under each setting are presented in the following table:

While performance drops under visual shifts, which is expected, AcTOL continues to outperform DecisionNCE in all available test conditions. This suggests that the learned representation maintains useful generalization ability even without any specific adaptation for visual domain shift.

For future work, we plan to improve visual robustness by incorporating stronger image backbones such as Vision Transformers, applying more extensive data augmentation during pretraining, and exploring domain randomization techniques to enhance performance under visual out-of-distribution conditions.

[1] WHAT MAKES PRE-TRAINED VISUAL REPRESENTATIONS SUCCESSFUL FOR ROBUST MANIPULATION? Arxiv'23

W2/Q1. Domain gap between egocentric pretraining and third-person deployment.

Thank you for raising this important point. There is indeed a visual domain gap between human-centric videos and third-person robot observations. AcTOL can addresses this challenge in belowing ways.

• AcTOL learns a general inductive bias of temporal ordering and continuity from vision-language pretraining. This bias focuses on the progression of visual states over time, rather than on specific visual details. As a result, the learned features can generalize across different embodiments. For example, although a robot gripper may look different from a human hand, both follow similar patterns of interaction when performing tasks like opening a drawer or picking up an object. This generalization ability is demonstrated in Figures 4 and 9, where AcTOL successfully produces meaningful reward signals from real-world robot videos, even when trained on human videos.
• During downstream policy learning such as behavior cloning, we find only fine-tune the model using a small number of robot demonstrations can largely mitigate the domain gap. To prove this, we first take 25 in-domain demonstrations (5 per task) in Franka Kitchen to fine-tune the pre-trained encoders using the AcTOL objective. Then, as before, we freeze the fine-tuned encoders and train policy networks on top using behavior cloning. We report the comparisons when using 15 demos for policy training. As shown in the table below, the success rate improvement also demonstrate that the learned temporal inductive bias can be effectively adapted to the robot domain with limited supervision.

W3. Narrow language perturbation coverage.

We appreciate the reviewer’s concern regarding the evaluation of language diversity. As detailed in Appendix B.4 (line 609), we have included an experiment specifically designed to assess this aspect. In this setting, we provide four language variations per task, including those generated by GPT-4o. These instructions are approximately 15 words long and feature diverse verbs, richer noun phrases, and extended contexts. For example:

"Mind pushing open the right cupboard cabinet door? I need to grab the cups inside."

Despite the increased linguistic complexity, AcTOL experiences only a modest 3% drop in success rate (see Appendix B.4) while still outperforming baseline methods. This experiment is intended to emulate the variability of natural household instructions, and we believe it offers a meaningful measure of robustness under diverse language conditions.

Q3. Hyperparameter sensitivity analysis ($\lambda$).

Due to space constraints, we included the sensitivity analysis of the hyperparameter $\lambda$ in Appendix A (line 570). As shown in the results, our strategy is verified agnostic to the choices of $\lambda$ varying from ${0.01}$ to ${1}$. We chose $0.1$ to ensure a reasonable scaling between the ordering and continuity losses.

Q4. Brownian Bridge might oversmooth critical semantic boundaries.

While real-world tasks often consist of multiple discrete semantic stages (e.g., “grasp → lift → place”), these stages typically unfold in a continuous and temporally ordered manner. As a result, visual features near stage boundaries usually exhibit gradual transitions rather than abrupt changes. As such, visual features near stage boundaries tend to exhibit gradual transitions rather than abrupt discontinuities.

To capture this structure, our approach employs two complementary components: VLO captures long-term visual-semantic ordering and helps recognize discrete semantic boundaries for multi-stage actions, while Brownian bridge act at a short-term frame level, promoting smooth local transitions in visual representations. This division of roles ensures that high-level semantic stages are preserved, while still allowing dense visual reward learning at a finer timescale.

As illustrated in Figure 4, AcTOL successfully distinguishes between different action phases while maintaining temporal continuity. This continuity is beneficial for producing dense reward signals that vary smoothly with task progress, leading to more effective learning in complex scenarios.

Q5. Limitations of VLO loss with repetitive actions.

Thanks. The datasets we use for pretraining, such as EPIC-KITCHEN, consist of clips that focus on a single activity segment, and therefore do not contain cyclic or repetitive actions like repeatedly opening and closing a drawer.

However, when using noisy web videos that may include cyclic behaviors, we acknowledge that temporally distant frames can occasionally share similar semantics, potentially affecting the training performance as we have mentioned in Limitations (line 736).

In practice, this issue is somewhat alleviated since chance of sampling semantically overlapping frames as negatives is relatively low.

To further reduce this risk, we propose a few simple but effective strategies:

• Similarity filtering: discard candidate negatives that are too similar to the positive.
• Action-boundary segmentation: segment actions before training to avoid sampling within the same repeated behavior.

For future work, we plan to explore how noisy videos with cyclic or repetitive actions can be better incorporated into AcTOL pretraining.

Thank you for your thoughtful feedback!

Q1. Assumption of temporal progression in videos.

Yes, our method assumes that input videos are temporally progressive. If temporally inconsistent or reversed videos are introduced during pretraining, they could potentially degrade the model’s learning, since the visual transitions would no longer reflect meaningful action progressions. However, we believe such cases are rare in naturally collected human demonstration videos.

At evaluation time, our trained model is capable of identifying inconsistencies between the video content and the given instruction. For example, when prompted with an instruction like "open the drawer", if a reversed video (i.e., a drawer being closed) is provided, the model tends to assign a decreasing reward, as shown in Figure 4. If a video contains unrelated content (e.g., washing dishes) or shuffled frames, AcTOL produces fluctuating reward curves that reflect poor alignment with the intended task. We appreciate this suggestion and will include additional examples of such cases in the revised version of the paper to further illustrate the model's behavior.