DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making

We appreciate Reviewer mTuA’s recognition of the novelty. Please find our responses to each comment below.

Seen manipulation tasks during training;lacks evaluation on unseen tasks for generalization.

• Randomized initializations within seen tasks are a standard evaluation protocol. In Table 1, tasks are fixed across train/test, but random initial states introduce diverse, unseen configurations, allowing evaluation of generalization to state and visual variations. Baselines follow the same setup.
• Our paper included evaluations on unseen tasks.
  • Instruction generalization(Table 2): Testing on paraphrased, unseen commands.
  • Cross-task transfer(Table 4): Testing on more complex, unseen tasks.
  • Compositional generalization(Table 6): Testing on novel task combinations.

2D navigation limitation.

To complement this, we extended experiments to iTHOR, which features realistic indoor navigation scenes. Results for other methods (AVDC, GVMA) are taken from [1];both belong to the line of learning to act from video.

[1] Grounding Video Models to Actions through Goal Conditioned Exploration.ICLR2025

Related work clarity.

We will clarify these connections in the revision. Unlike methods that model the environment or generate future frames, our approach predicts video dynamics, making it more suitable for long-horizon tasks. It also avoids large-scale pretraining, learning goal-conditioned representations for control.

The claim that "a single language instruction can correspond to multiple videos" motivates abstracted features from video, but video generation models can model the randomness of video given the text condition. The benefit of using abstracted features need further justification.

• Our goal is to bridge the gap between abstract language and detailed video, which becomes more pronounced when one language instruction maps to multiple executions. We focus on mitigating the modality gap by learning compact dynamic representations that capture what matters most in a video, making ours fundamentally different from modeling trajectory diversity like video generation models.
• As for the benefit of video abstraction, a concurrent vision-language study[2] shows that modality gaps—like the asymmetry between image and text in CLIP—negatively impact downstream performance. While their work is analytical, we build on similar insights and offer a practical solution in the embodied setting. [2]Two Effects,One Trigger.ICLR2025.

Abstraction vs. Transformer attention.

• Explicit Inductive Bias vs. Data-Driven Attention. Unlike data-driven attention, our method introduces an explicit bias toward semantically relevant and visually salient frames.
• Lightweight Structure. FrameScorer is a simple 2-layer MLP, lighter than Transformer-based alternatives and better suited for long-horizon tasks.
• Empirical Validation.

Inference-time video input and use of history frames.

Our method uses an online inference setup, where only current and past frames are available. During testing, frames are collected incrementally for real-time decision-making, consistent with standard practice in prior work.

The designed consistency loss for the abstraction directly use cosine similarity(CS) of the image features and text features. How to ensure they are aligned in the same feature space?

• We clarify that the consistency loss is computed between abstracted video dynamics and language embeddings, rather than between raw image and text features. Instead of forcing alignment between inherently mismatched modalities, we introduce an abstraction module as a bridge.
• While CS is a standard metric for cross-modal alignment, we understand the reviewer’s concern. Therefore, we estimate MI to capture global statistical dependency in the complex, long-horizon Franka Kitchen. The higher MI between abstracted dynamics and language, compared to raw pairs, further supports our approach. -|MI -|- Dynamic↔Language|0.058 Video↔Language|0.011

Use of pretrained model.

We uses a frozen DistilBERT for language and no large-scale pre-trained visual features.

Clarification on higher similarity indicate higher importance.

We understand the reviewer’s concern, and we clarify that our method does not assume that higher similarity directly indicates frame importance. The semantic consistency loss is applied over the entire dynamic representation sequence and the language embedding, encouraging global task relevance rather than relying on per-frame similarity. Importantly, we avoid strong frame-level assumptions. Instead, frame importance is inferred contextually, guided by consistency and saliency losses, enabling the model to focus on what truly matters—not just what appears similar.

Missing period.

We’ve added the missing period.

We sincerely thank Reviewer fGaZ for the valuable feedback. Below, we respond to each comment and will revise the paper accordingly.

Works like LAPO, LAPA, and IGOR use video to learn latent actions for decision-making. PIDM relates to future prediction.

These works, like ours, target video-based decision-making. We briefly summarize the differences here and will provide a detailed comparison in the revised version. LAPO [ICLR2024] enables learning latent-action policies from raw video via consistency objectives, which can be quickly fine-tuned into expert-level policies. LAPA [ICLR2025] extends this by incorporating language. In contrast, our method avoids pretraining and latent action modeling, and instead learns video dynamics to address the modality gap through end-to-end training. IGOR [arXiv:2411.00785] learns a shared latent space for cross-embodiment transfer, while our method models dynamics within a single embodiment. PIDM [ICLR2025] predicts future states and feeds them into an inverse dynamics model to couple perception and control. Our method instead focuses on “what matters” rather than pixel-level predictions.

Comparation with MotoGPT,PIDM

MotoGPT [arXiv:2412.04445] and PIDM are relevant as both leverage video and use Transformer-based architectures for policy learning. MotoGPT follows a three-stage pipeline: latent motion token modeling, generative model pretraining, and collaborative fine-tuning. PIDM adopts an end-to-end approach that predicts actions from forecasted states, coupling perception and control. Given the structural similarity and its end-to-end design like ours, we compare with PIDM on the Franka Kitchen. For fairness, we use the same image and language encoders trained from scratch. A detailed comparison with MotoGPT will be included in the final version.

Training details in appendix

We will include training details such as the image encoder, learning rate, epochs, batch size, and hyperparameters.

How FrameScorer learns to find keyframes from language with limited data? Can it extend with larger training datasets and pretrained visual representations?

Reasons for finding key frames with limited data

• Sub-task repetition: Many instructions share similar sub-tasks, providing multiple observations of the same semantic goal across different trajectories.
• Multiple forms of weak supervision: FrameScorer is guided by two information-rich signals: focusing on semantically relevant frames and visually salient ones.

Regarding the scale of Franka Kitchen dataset

In our main experiments, we trained the model with 25 trajectories per instruction (~300 video-action pairs each), constrained by computational resources. We anticipated the impact of dataset size and included an ablation study (Appendix, Fig.12) showing that our method scales well. Increased trajectory diversity helps the model better capture video-language patterns and improves performance.

Regarding pre-trained visual representations

We had similar thoughts with replacing our lightweight visual encoder with the large-scale pretrained R3M(frozen during training; Appendix Table 7), but observed a performance drop. We attribute this to (1)modality misalignment—R3M’s CLIP-style modeling struggles to bridge language–vision gaps, and (2)domain gap—R3M is trained on human egocentric videos, which differ from our robotic tasks. Nonetheless, we see strong potential in pretrained models and are exploring adapter-based fine-tuning for better integration.

Common frameworks typically use LLMs to decompose tasks or letting the VLA predict the next sub-task instruction? Which is better compared to yours?

The methods you described often follow a language-centric paradigm, whereas our approach is video-centric, offering a complementary perspective. Language-centric methods provide modularity and interpretability but depend on accurate sub-task decomposition, which can cause cascading errors in open-ended or ambiguous tasks. Our method models task progression implicitly by capturing temporal patterns in video conditioned on the language instruction. We see combining both approaches as a promising direction for future work.

It is possible to split the language instruction into sub-instructions, and learn the Framescorer by matching them to dynamic features?

Interestingly, this suggestion overlaps with a direction we explored by integrating our method with LISA, which decomposes language instructions into sub-instructions, corresponding to skills and matched to video dynamics (Fig.8). However, the integration did not improve performance. Mutual information analysis showed consistently low correlation between the decomposed language and video features by LISA, suggesting that the decomposition introduced noise or mismatches. Nonetheless, we believe this remains a promising direction and plan to explore stronger decomposition models or structured task planners in future.

We thank Reviewer VsTD for the feedback. To ensure clarity, some responses are stated directly—we appreciate your understanding.

The fixed window size is less flexible

• Fixed window sizes are standard practice, used in baselines like LISA and SkillDiffuser, and in some video understanding work.
• Performance is stable under moderate window changes (Fig.7a), suggesting robustness.
• We acknowledged the limitation(lines 886–888) and plan to explore adaptive horizons.

What is the image encoder—pretrained from SkillDiffuser or from scratch?

The image encoder is a CNN trained from scratch, not from SkillDiffuser or any pretrained source, ensuring that improvements reflect our own contributions.

Eq.4&5 compute similarity between abstraction of the short clip and instructions of a whole task—seems questionable.

We do not compute similarity between a short clip and language. Instead, as noted in lines 170–173 and 184–188, it is computed between the full dynamic representation of the entire trajectory and the task-level language instruction.

Inputs to Video Dynamic Reasoning differ between training and inference may impact performance.

We adopt a closed-loop process to reduce the train-test gap, a common strategy in decision-making. Both stages use dynamic abstracted from real observations, mitigating error accumulation. Specifically, during training, inputs come from actual frames; during inference, predicted dynamics guide actions, and real observations are appended iteratively.

The source of the goal token $g_t$

As noted in lines 271–274(left) and 220–229(right), during training, $g_t$ is gradually shifted from ground-truth future frame to predicted dynamics to stabilize learning. At inference, $g_t$ is taken from the predicted dynamics.

Confused why actions are $a_{t-C+1:t-1}$ but the input are $e_{t-C+1:t-1}$—shouldn't actions be autoregressively generated?

The model is auto-regressive, generating actions step by step during inference. Following common practice in sequence modeling, we apply parallel supervision over the action sequence during training to improve efficiency and stability.

A pseudocode might make training clear

We’ve prepared detailed pseudocode but couldn’t include it due to space limits. It will be added in the final version.

Fig.3 shows a sharp drop from 0.4(2 tasks) to 0.1(3 tasks)

The sharp drop in performance with more tasks reflects the challenge of long-horizon task, especially under constrained settings (lightweight model, limited data). As shown in Fig.3, all baselines struggle, while ours performs better.

Tab.2 uses paraphrased instructions for generalization. With T5-XXL, this is less of an issue due to embedding similarity.

• We use the same lightweight pretrained language encoder (DistilBERT) as our baselines, yet achieve better instruction generalization(Tab.2). This indicates that success relies not just on the language model, but on the ability to connect language with videos.
• Our method is resource-efficient and complementary to LLMs(11B for T5-XXL). While LLMs reduce linguistic variation, ours bridges the language–vision gap.

Fig.6:Why does FrameScorer help more in Kitchen than in BabyAI?

• The difference stems from environment: FrameScorer helps in Kitchen with rich visuals and redundancy, while BabyAI’s simplicity reduces the need for abstraction.
• FrameScorer is one part of our method;overall performance gains in BabyAI reflects the effectiveness of other modules.

Fig.7:Explain why D&LG performs poorly than dynamic-guided

D&LG underperforms due to a mismatch: language encodes long-term goals, while dynamics reflect short-term cues. Their fusion introduces redundant or conflicting signals, hindering decisions—supported by low mutual information in Fig.8(bottom).

Fig.8:LISA’s mutual information is nearly zero—inconsistent with the original paper

Due to environment differences: we compute MI in the more complex Franka Kitchen (lines 406–407), while LISA uses BabyAI. Higher diversity in Kitchen leads to lower MI.

Supplementary Material lacks details

We will include details to ensure reproducibility.

Not convinced that SkillDiffuser underperforms DynaMind on simple tasks.

• SkillDiffuser results are directly taken from the original paper, without any modification.
• SkillDiffuser underperforms on simple tasks, where skill reuse is limited. It suits long-horizon tasks but adds unnecessary complexity to simpler ones.
• The two methods are complementary, not competing, and each has its strengths. Our goal is fair comparison under a unified protocol, not to claim superiority.

Text2video is expensive

Unlike T2V methods that require full trajectory generation, our lightweight model predicts compact dynamics without heavy computation. On Lorel(A800 GPU, batch size 64), it achieves better SR with comparable or lower cost.

We thank Reviewer kRnT for recognizing the novelty and presentation of our work. We address each concern below and will revise the paper accordingly.

Comparison with more recent imitation learning methods from the past two years

We agree that including comparisons with more recent imitation learning methods strengthens the overall evaluation. In response, we add 3 recent methods:

• Diffusion Policy(DP)[IJRR2023]: a diffusion-based imitation learning method.
• LCSD[ICAPS2024]: an extension of DP that introduces a language-conditioned skill learning module.
• PIDM[ICLR2025]: an end-to-end imitation learning approach that unifies vision and action by predicting actions from forecasted visual states.

We compare these methods with ours under 2 benchmarks:

Lorel Sawyer

Franka Kitchen

• Lorel results are reported from LCSD.
• Kitchen results are from our re-implementation using their public codebases.

Are there any directly comparable methods?

To the best of our knowledge, we are the first to address video-language modality imbalance from a video-centric perspective for language-conditioned decision-making. We compared with the most relevant works—LISA and SkillDiffuser—which approach the problem from the language side. Notably, a concurrent study [1] identifies similar modality imbalance in CLIP-style models, caused by information asymmetry between image and text. It further shows that smaller modality gaps lead to better performance, and that embedding dimensions contribute unequally. Though focused on a different task, it highlights the general importance of the problem and supports our motivation. [1] Two Effects,One Trigger.ICLR2025

In Table 1, some tasks (e.g.,move black mug right) show a large performance gap.

The short-horizon, low-complexity tasks are well-suited for vanilla imitation learning, which excels at fitting simple, deterministic behaviors. Interestingly, both our method and Text2Video approaches like SkillDiffuser underperform in these cases(line 282), likely due to indirect objectives introducing unnecessary complexity. This reflects a common but often overlooked limitation, which we plan to address for better adaptability across task complexities. In contrast, vanilla imitation methods struggle on more complex tasks with long-term dependencies or greater generalization demands (Table 1,2,6).

How does Hybrid Assignment handle early training and implement the transition?

To mitigate the impact of early-stage instability in the dynamic reasoning module on action prediction, we initially use ground-truth goal features (future frame representations) for stable supervision, then gradually shift to predicted dynamics to enable end-to-end learning while maintaining training stability. This transition is controlled by a linear annealing schedule with sampling probability $p_n=\frac{n}{N}$, where $n$ is the current epoch and $N$ is the total epochs.

How are historical frames, actions, and predicted dynamics fed into the Action Transformer and fused?

The Action Transformer takes three inputs: 1) historical frame features, 2) historical actions (embedded into the same latent space), and 3) a goal token, as described in the last question. The fusion process includes: adding timestep embeddings to encode temporal order, interleaving state and action tokens by timestep, prepending the goal token for global conditioning, and applying LayerNorm and an attention mask before feeding the sequence into the Transformer.

Results of integrating with language-centric methods

This is a thoughtful insight—it happens to align with a direction we’ve explored. Specifically, we integrated DynaMind with the language decomposition module from LISA (Figure 8), but it did not improve performance in the Franka Kitchen. To further investigate, we analyzed the mutual information during training(Fig.8, bottom). The results show that DynaMind progressively increases the mutual information between modalities, while LISA does not, suggesting that its language decomposition may lead to semantic information loss. Nevertheless, we consider our method orthogonal and potentially complementary to language-based approaches. In future work, we plan to explore integration with more advanced language decomposition methods.

Training cost of this method

Our method is designed to be lightweight, using a small visual encoder and shallow Transformer blocks for dynamic reasoning and action prediction. To assess training cost, we compare with LISA (Transformer) and SkillDiffuser (Diffusion) under identical A800 GPU settings. We report parameter count and GPU memory usage (batch size 64, Lorel Sawyer). As shown in the table, Ours balances computational cost and success rate.