Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination

We thank the reviewer for their engagement with our work and their feedback.

First, we clarify a technical misunderstanding highlighted in the reviewer’s summary: inner speech generation employs a Conditional Variational Autoencoder (CVAE) rather than a language model. The CVAE is trained using representations derived from vision-language models, enabling stochastic behavioral mediation through learned latent spaces.

[1] Presentation issues

We have worked to give a clear exposition in the paper. While other reviewers commended the writing of our exposition and awarded us high scores on clarity, we appreciate that this reviewer found certain sections unclear. To address this, we provide here a systematic summary, with explicit references to figures, equations, and empirical results, to clarify concepts and methods:

The paper introduces MIMIC, operationalizing Vygotsky's cognitive mediation theory within imitation learning (Section 1, lines 54-65). The core proposition (Proposition 1, lines 61-65) establishes that agents should learn $\pi_{\theta}(a|s,m)$ rather than $\pi_{\theta}(a|s)$, where m represents linguistically-encoded inner speech mediating between perception and action. This paradigm shift from behaviorist state-action mapping to the cognitive framework of state→inner speech→action (Section 3.1, lines 143-183) addresses fundamental limitations in capturing human behavioral diversity while enabling linguistic controllability.

The MIMIC architecture (Figure 1, page 5) integrates three components in a unified pipeline. During training, a pre-trained VLM generates behavioral descriptions from demonstration GIFs using the prompt specified in Section 3.2.2 (lines 221-225), which are then encoded via CLIP embeddings (visualized in Figure 2's t-SNE plot). A CVAE learns to generate CLIP-encoded inner speech representations by minimizing Equation 4 (line 238), conditioned on W-length observation windows. The diffusion policy (DDPM-T architecture) generates the actions, incorporating the inner speech through classifier-free guidance (Equation 3, line 197). During simulation (Algorithm 2, page 20), the agent generates inner speech every W timesteps starting at $t_0$, with these polling parameters critically balancing behavioral stability against adaptability (Figures 3a-b show sensitivity analysis for $W\in$ {1,10,20,50,100} and $t_0 \in$ {0,1,W/2,W-1}).

Empirical validation demonstrates 20-30% success rate improvements across D3IL benchmarks (Table 1, page 7) with behavioral entropy increases up to 40%. Wasserstein distance metrics (Tables 2 and 6) confirm high-fidelity imitation. In human-AI coordination tasks (Overcooked (Table 3, page 8)), agents trained with MIMIC achieve 25% higher collective rewards when paired with human proxies compared to diffusion behavior cloning. Figure 3's ablation studies reveal: (c) MIMIC outperforms random/k-means inner speech alternatives, validating linguistic grounding; (d) GPT-4o marginally outperforms o4-mini/4o-mini VLMs. Conditional generation experiments (Table 4, Figure 7) demonstrates zero-shot behavioral steering through designer-specified language.

In regard to Figure 3, we acknowledge the missing details, especially in the caption. We will improve the figure to include additional details, which we also outline below:

(a) Initial Step Sensitivity ($t_0$ parameter): This subplot examines the impact of delaying the first inner speech generation. The x-axis represents $t_0$ values {0, 1, 25, 50}, where $t_0=0$ indicates immediate inner speech generation upon trajectory initiation. Three metrics are evaluated: Success Rate (blue, 0-1 scale), measuring task completion frequency; Entropy (green, measured in bits), quantifying behavioral diversity through categorical distribution analysis; and Distance (red, L2 norm), indicating final position error from target alignment. The declining performance with increasing $t_0$ (success rate drops from ~0.8 to ~0.65) demonstrates that early cognitive mediation critically enhances both task performance and behavioral fidelity.

(b) Update Window Analysis (W parameter): This investigation reveals optimal polling frequencies for inner speech regeneration. The x-axis displays $W\in$ {1, 10, 20, 50} representing timesteps between updates. The non-monotonic relationship observed—peak success at $W=20$ with $0.74$ success rate—indicates a fundamental trade-off: frequent updates $(W=1)$ induce behavioral instability through excessive re-planning, while sparse updates $(W=50)$ permit accumulation of distributional drift. Entropy increases monotonically with $W$, suggesting that less frequent cognitive intervention enables greater behavioral exploration.

(c) Ablation Study on Inner Speech Representations: This comparative analysis validates our linguistic grounding hypothesis. Four conditions are evaluated: "None" (baseline BC without inner speech), "Random" (stochastic vector representations), "K-means" (clustered behavioral prototypes), and "Inner Speech" (MIMIC). The substantial performance gap—MIMIC achieving 0.72 success rate versus 0.66 for baseline—demonstrates that linguistically-structured representations provide superior behavioral mediation compared to arbitrary or cluster-based alternatives.

(d) Vision-Language Model Comparison: This ablation examines the impact of the selection of a VLM on behavioral description quality. Two GPT models (o4-mini, 4o) are used to generate inner speech during training, with performance differences reflecting CLIP embedding space alignment. GPT-4o's marginal superiority (0.72 success rate) over alternatives suggests that description quality influences but does not dominate performance, validating our framework's robustness to VLM variations. At this point, we welcome the reviewer to raise any outstanding concerns or clarification questions, and we will be happy to address them.

At this point, we welcome the reviewer to raise any outstanding concerns or clarification questions regarding the presentation, and we will be happy to address them.

[2] Cognitive grounding and technical support

We acknowledge a crucial epistemological distinction raised by the reviewer that warrants clarification. Our invocation of Vygotskian cognitive theory serves as a conceptual scaffold rather than a claim of biological fidelity. We explicitly position our work within the tradition of cognitively-inspired AI (Section 2.1, Line 115) —where psychological theories inform computational architectures without asserting neurobiological correspondence.

The technical operationalization substantiates three core properties of inner speech identified in the cognitive literature: (1) semantic condensation through CVAE's latent compression, (2) stochastic generation operationalized via diffusion policy and (3) temporal persistence via periodic polling mechanisms. These architectural choices translate abstract cognitive principles into concrete computational mechanisms.

Productive AI research often draws inspiration from cognitive science without requiring neural fidelity. Just as convolutional networks leverage principles of visual processing without mimicking V1 neurons, our framework seeks to extract functional principles from inner speech theory. To address the reviewer’s concerns, we will revise our exposition to more precisely delineate between: (a) cognitive theories providing conceptual motivation, (b) computational mechanisms inspired by but not replicating these theories, and (c) empirical validation of behavioral outcomes.

We emphasize that our computational approach is firmly grounded in cognitive theory and performs direct modeling of “inner speech as behavior guides,” building on Sokolov's neurophysiological model and Vygotsky's mediational theory. The theoretical discussion in Section 3.1 captures the mediational aspect and can be enhanced to include the theoretical specificity required for a complete Sokolovian instantiation. We will augment Section 3.1 to include further enhancements to the equations to clearly map out the cognitive tools we use to the components of our architecture discussed in Section 3.2.

[3] Strong Goal Conditioned Baseline

We thank the reviewer for this suggestion, which will help to further highlight the contributions. We have conducted new experiments on two additional behavior cloning approaches to compare their performance against MIMIC: (i) Behavior Transformers (BeT), which uses an action-based clustering and transformer architecture to perform multi-modal imitation learning, and (ii) BESO, a strong goal-conditioned multimodal IL approach. BESO is a state-of-the-art approach for behavior cloning that has shown better performance on several tasks in the D3IL benchmark compared to Diffusion BC. Table 1 (D3IL comparison) and Table 2 (Overcooked comparison) demonstrate superior performance of MIMIC across baselines on both datasets.

Table 1: D3IL Results

Table 2: Overcooked Results

We will add the above results and those on other tasks in the final version of the paper.

We believe that we have comprehensively addressed all the concerns raised by the reviewer and would welcome any further discussion on any remaining points that could lead the reviewer to reconsider their evaluation.

Dear Reviewer 5Wb5,

Thank you for your time and efforts towards the initial assessment of our work. You raised three main concerns—presentation clarity, cognitive inspiration of our approach, and absence of a strong goal-conditioned baseline—and we’ve worked diligently to address each one of them in our rebuttal. Specifically, our rebuttal provides -- a systematic summary of our exposition aiming to address your presentation concerns, a detailed clarification on the cognitive claims of our work and new experiments on a state-of-the-art goal conditioned baseline.

If there still remains any outstanding questions or concerns, we would greatly appreciate the opportunity to discuss them further and address them promptly. We hope to confirm that all your points have been resolved before the end of the discussion phase.

Thank you again for your review, and we look forward to your updated evaluation.

Sincerely,

The authors of paper #27058

We thank the reviewer for their time and engagement with our work. We are excited to see the positive evaluation of our work and the reviewer’s recognition of our theoretical and empirical contributions. We address the reviewer's specific concerns below, providing clarification and additional evidence.

[1] Comparison with other BC methods

We have conducted new experiments on two additional behavior cloning approaches to compare their performance against MIMIC: (i) Behavior Transformers (BeT), which uses an action-based clustering and transformer architecture to perform multi-modal imitation learning, and (ii) BESO, a goal-conditioned multimodal IL approach. BESO is a stronger model for behavior cloning and has shown better performance on several tasks in the D3IL benchmark compared to Diffusion BC. Table 1 (D3IL comparison) and Table 2 (Overcooked comparison) demonstrate the superior performance of MIMIC across baselines on each of the datasets.

Table 1: D3IL Results

Table 2: Overcooked Results

We will add the above results and those on other tasks in the final version of the paper.

[2] Ablation on inner speech representations and VLM models

We now also ablate our method on two fronts: the VLM and the embedding space. Specifically, we evaluate Qwen2.5-VL-72B-Instruct as an open-source alternative to GPT-4o, and MPNET (all-mpnet-base-v2) as an alternative to CLIP (our choice of mp-net stems from its performance on MTEB benchmark where it is ranked 102 compared to ModernBert’s rank 202 and T5’s rank 177). In the table below, we report the results on the Aligning, and we will include the results on other tasks in the paper.

We observe that GPT-4o with CLIP embeddings outperforms other variations, but also find that both ablations still outperform the base Diffusion BC variant. This shows that MIMIC, even in the absence of a strong LM for inner speech, can enhance performance over the Diffusion BC variant. Furthermore, the lower performance of MPNET highlights the value of vision-language embedding over sentence-only embedding.

Paper formatting concerns

Thank you for catching them. We will update line 49 to say human $\mathcal{H}$ and decision $a_t$.

Figure 2 shows the t-SNE visualization of the latent space of inner speech, where different clusters exemplify the diversity in demonstrations captured through the inner speech. The color bar shows the cluster number that each data point belongs to, and the ellipsoid shows the variance around the cluster mean. We will make these details clearer in the final version.

Thank you for your review and supportive comments about scalability, novelty in cognitive grounding, and concrete experiments. We address your specific concerns below, but all the weaknesses and questions you raise reflect the limitations we already discussed in the Appendix. We do not view these as weaknesses of the work; rather, they are intended to provide suggestions for interesting future research avenues. We think the efficacy and novelty of our framework are already well demonstrated.

[1] Concerns about computational demands and insufficient discussion on scalability in larger, more complex tasks

MIMIC builds on widely used diffusion models and CVAEs—established baselines with well‐characterized computational profiles fully supported by modern research infrastructure—and critically, without adding any extra complexity. Indeed, practitioners routinely deploy similar architectures in production settings where they need behavioral realism. To alleviate the reviewer’s concerns, we’ve included a complete runtime and inference‐time analysis below and will add this expanded discussion to the paper:

Let $T_{CVAE}$ and $T_{diff}$ denote one forward pass through the CVAE and diffusion models, respectively. Over a simulation horizon $H$ with window size $W$, we perform $H$ diffusion passes and $H/W$ CVAE passes, yielding a total complexity of $O(H T_{diff} + H/W T_{CVAE})$). Since both are vision‐conditioned with similar runtimes and $H > H/W$, the diffusion term dominates—so MIMIC adds no inference overhead. Empirically, we confirm below that MIMIC’s simulation runtime matches that of Diffusion BC across representative vision-based environments.

Further, the environments used in our paper are representatively large scale and complex. D3IL is a strong imitation learning benchmark (ICLR 2024) and provides a comprehensive suite of tasks that are currently challenging for AI agents. Similarly, Overcooked is a classical and highly researched environment that presents complex coordination tasks.

[2] Reliance on VLMs, noisy or imperfect visual data and ethical considerations

We are unclear on the source of noise or imperfection that the reviewer is concerned about. We identify two potential sources of noise and address each of them.

(i) Behavior demonstrations: This includes imperfections such as noise in the demonstrated behavior or suboptimal trajectories with respect to the task. We handle this, and it is crucial to focus on such demonstrations as they represent the richness in human behaviors. As we discuss in the paper (line 136), human behavior is noisy, diverse and non-Markovian in nature and hence, we indeed focus on performing well when provided with these kinds of demonstrations. Both the datasets we use comprise only human demonstrations, with noise and suboptimal behavior.

(ii) Visual observations: This includes imperfections such as blurry images or an obscured view. We believe that improvement in VLMs will continue to help address these kinds of scenarios. At the same time, we do not know of suitable datasets, which would prevent an empirical study. And to specifically address the reviewer's comment, the other alternative to relying on VLMs is to use a human-annotated dataset. But this would be strictly more expensive and still prone to noise. We are curious to know if the reviewer has any suggestions on the hybrid approach they mention.

Broadening out the discussion, one approach to address the concerns about reliance on VLMs (and we would also include the CLIP model here) is to compare against other models. Our ablation in Figure 3(d) examines the effect of VLM choice on description quality. We compare o4-mini, and 4o for inner speech, showing GPT-4o’s marginal lead (0.72 success rate), with performance differences reflecting CLIP embedding space alignment. This implies description quality influences but doesn’t dominate performance—demonstrating robustness to VLM variation. We also introduce new ablations on VLM and embedding space: evaluating Qwen2.5-VL-72B-Instruct as an open-source alternative to GPT-4o, and MPNET (all-mpnet-base-v2) as an alternative to CLIP. The table below reports these Aligning-dataset results and we will include the full suite of results in the paper.

We observe that GPT-4o with CLIP embeddings outperforms other variations, but also find that both ablations still outperform the base diffusion BC variant. This shows that MIMIC, even in the absence of a very powerful LM for inner speech, can enhance the performance over the diffusion BC variant. Furthermore, the lower performance of MPNET highlights the value of vision-language embedding over sentence-only embedding.

While we flagged the potential bias issues in existing VLMs and their ethical implications, the research on mitigation strategies for such biases is out of the scope and an entirely different research field. We believe that any response we provide in terms of mitigation strategies would be speculative and counterproductive to the focus of this work.

[3] Formal theoretical analysis, optimality guarantees and generalization

The theoretical foundations of our framework derive from established convergence properties of CVAEs (Kingma & Welling, 2014) and diffusion models (Ho et al., 2020), which provide well-characterized optimization landscapes and asymptotic guarantees. Pursuing additional formal analysis was not the goal of our work and would misalign with our core contribution: operationalizing human cognitive processes that, by their very nature, eschew optimality in favor of generality and behavioral diversity.

The robustness we claim emerges not from convergence proofs but from grounding in Vygotskian cognitive theory (Section 3.1), which explains why human decision-making achieves remarkable generalization. Human cognition's strength lies in generating satisfactory solutions through linguistically mediated deliberation. Our empirical validation across diverse environments (D3IL, Overcooked) demonstrates that incorporating these cognitive principles yields superior generalization compared to optimization-driven approaches, suggesting that foundations from cognitive science provide complementary properties for real-world deployment.

[4] Latent representations, transparency and interpretability

We highlighted the opaqueness limitation to motivate further work on interpretability—a difficult, open problem—while noting that our cognitive grounding, coupled with linguistic control, takes a good step toward building interpretable behavior generators.

We appreciate the reviewer’s clarification about the correspondence of latent representations to the actions. As shown in Section 2, Figure 2, our t-SNE visualization of language embeddings on the Aligning dataset demonstrates the behavior diversity captured by inner speech; we’ll include these plots for all datasets in the Appendix.

We further provide a new analysis of inner speech during steerable behavior simulation via conditional generation, by retrieving the nearest training descriptions in CLIP’s embedding space at each polling step. Below are two examples of different conditions (from the Aligning dataset (with $W=50$ and $t_0=49$), with cosine similarities to the actual inner speech in parentheses. We will include such examples from all datasets in the final version of the paper.

1. Condition: I need to begin with a swift side approach, then slow down to ensure precise alignment.

Inner Speech:

• I am approaching the box from the side, rotating slightly to align smoothly with the fixed box. (0.9471)
• I am approaching directly, then rotate in place to align perfectly. (0.9619)
• I approach with a direct path but make a last-second adjustment to align perfectly. (0.9611)

2. Condition: I need to rotate the box first before aligning it with the target.

Inner Speech:

• I push the box directly without rotation, aiming for a straightforward alignment. (0.9183)
• I am approaching directly, then rotate in place to align perfectly. (0.9614)
• I am starting from a slightly rotated angle, needing a mid-action adjustment to align. (0.9616)

This shows that MIMIC mediates an agent’s actions by generating an appropriate inner speech based on its behavior during simulation. The high similarity with a true description in the CLIP space further shows that the generated inner speech is indeed meaningful.

[5] Multi-agent environments and complex coordination tasks

There may be some misarticulation of the reviewer’s concerns here. The reviewer already acknowledged the value of this work in multi-agent environments and human-AI coordination in the “strengths” section and the concerns refer to the same capabilities. To clarify, Overcooked is a classical benchmark of complex coordination without explicit communication; in our experiments, MIMIC outperforms all baselines across its tasks, demonstrating adaptability and efficacy in behavior generation for multi-agent environments. If the reviewer alludes to our limitations section, we reiterate that those points are meant as a forward-looking agenda—such as scaling to more agents or mixed-motive dynamics—which would require substantial additional research, leading to additional publications and should not be viewed as weaknesses of this paper.

Dear Reviewer 5D71,

Thank you again for your review. In our rebuttal, we’ve carefully addressed each of your points—adding in depth analyses, and detailed clarifications to ensure every concern is fully resolved. We appreciate the considerable effort you’ve already invested and would welcome your continued engagement in the discussion phase. If there are any remaining issues or questions that would aid your evaluation, please let us know and we will be happy to address them further.

Sincerely,

The authors of paper #27058.

We thank the reviewer for their detailed feedback and appreciate their positive comments regarding our novelty, motivation, and strong empirical results. We are grateful for their thoughtful questions, which help us clarify key aspects of our work and strengthen the paper's contribution. Below, we address specific concerns spanning baselines, generalization, and failure modes, providing additional context and evidence where helpful:

[1] Limited Baselines:

We appreciate the reviewer's observation that the version of Diffusion BC we use is a strong baseline for multimodal imitation learning. This choice is based on published benchmarks and our own evaluations. We now present new results for Behavior Transformers (BeT) on two datasets below, confirming that MIMIC achieves superior performance for both D3IL (Table 1) and Overcooked environments (Table 2). We further compare against BESO [1], a goal-conditioned multimodal IL approach. BESO is a stronger model for behavior cloning and has shown better performance than Diffusion BC on several D3IL tasks. Results on both environments demonstrate that MIMIC significantly outperforms BESO. We will include these and the results of other tasks in the revised paper.

Table 1: D3IL Results

Table 2: Overcooked Results

While the reviewer notes comparison with only BC variant, BeT and other approaches like BESO are also BC variants. As we have discussed in the paper, BC offers a simple, efficient, and offline approach with remarkable efficacy across domains. Our choice of BC reflects a requirement for offline learning—precluding environment interactions during learning. This is a critical safety constraint that non-BC approaches violate, potentially introducing deployment risks in high-stakes applications.

[2] Reliance on External Language Models

Analysing how MIMIC scales with various capabilities of language scaffolding is a good suggestion. As we explain below, this is not a limitation of our work. First, generating inner-speech captions with GPT-4o is inexpensive: we make $N/B$ API calls with average context length of $\sim 100$ for text and $\sim B \cdot (85 + 170n)$ for images (with $n$ tiles of $512\times512$ px), totaling $\sim N\cdot 170 n$ tokens which is about 2 dollars for over 400 trajectories, even with high resolution $2048\times 2048$ images. The CLIP model ($\sim 0.5$ B parameters) is likewise lightweight and can be efficiently used during training to generate the inner speech. We will add this expanded discussion in the paper.

Next, we have now also ablated our approach on two fronts: the VLM and the embedding space. Specifically, we have evaluated Qwen2.5-VL-72B-Instruct as an open-source alternative to GPT-4o, and MPNET (all-mpnet-base-v2) as an alternative to CLIP. In the table below, we report the results on the Aligning dataset. We will include the full suite of results in the paper.

We observe that GPT-4o with CLIP embeddings outperforms other variations, but also find that both ablations still outperform the base diffusion BC variant. This shows MIMIC's efficacy, even in the absence of a very powerful LM for inner speech. Furthermore, the lower performance of MPNET highlights the value of vision-language embedding over sentence-only embedding.

[3] Run Time Analysis:

Based on the reviewer’s comment, we performed a complete complexity analysis, which includes inference time analysis, and report below. We will add this expanded discussion to the paper:

Let $T_{CVAE}$ and $T_{diff}$ denote one forward pass through the CVAE and diffusion models, respectively. Over a simulation horizon $H$ with window size $W$, we perform $H$ diffusion passes and $H/W$ CVAE passes, yielding a total complexity of $O(H T_{diff} + H/W T_{CVAE})$). Since both are vision‐conditioned with similar runtimes and $H > H/W$, the diffusion term dominates—so MIMIC adds no inference overhead. Empirically, we confirm below that MIMIC’s simulation runtime matches that of Diffusion BC across representative vision-based environments.

[4] Performance on overcooked and clarification about multiple models

We acknowledge the reviewer's observation regarding our reliance on simulated human proxies rather than empirical human studies. This methodological choice follows current precedents in human-AI collaboration research, where computational validation precedes human subject experimentation. Given the theoretical complexity and technical innovations introduced—specifically the operationalization of Vygotskian cognitive mediation within imitation learning architectures—we adopted a staged research approach. We prioritize establishing computational viability and theoretical coherence before moving to human subject experimentation. We agree that empirical validation with human participants is a useful next phase in this research.

MIMIC and MIMIC-MA are not separate models, but variants of the same architecture using either self-only inner speech (MIMIC) or both agents’ inner speech (MIMIC-MA). As discussed in lines 255-259, this addition is inspired by the mirror neuron theory of social cognition, which is made possible through our framework. We still find that in almost all cases (except Asymmetric advantages), both variants still outperform the baseline while showing that modeling another agent’s speech is often important in multi-agent coordination tasks. This effectively supports the generalization of our approach to a dynamic multi-agent environment.

Performance drop of MIMIC on Asymmetric Advantages: This is a good catch as we suspect an error in this result. First, note that the MIMIC-MA model outperforms diffusion-BC baseline. Hence it was surprising to see its performance drop when using only its own inner speech. Our initial investigation suggests a color‐flip bug—where the ego agent was inadvertently using the other agent’s speech instead of its own (not our intended setup). We’re rerunning the experiments and will update the results as soon as they’re available.

[5] Additional Details on Inner Speech

We thank the reviewer for identifying this important consideration. As shown in Section 2, Figure 2, our t-SNE visualization of language embeddings on the Aligning dataset demonstrates the behavior diversity captured by inner speech; we’ll include these plots for all datasets in the Appendix.

We further provide a new analysis of inner speech during steerable behavior simulation via conditional generation, by retrieving the nearest training descriptions in CLIP’s embedding space at each polling step. Below are two examples of different conditions (from the Aligning dataset (with $W=50$ and $t_0=49$), with cosine similarities to the actual inner speech in parentheses. We will include such examples from all datasets in the final version of the paper.

1. Condition: I need to begin with a swift side approach, then slow down to ensure precise alignment.

Inner Speech:

• I am approaching the box from the side, rotating slightly to align smoothly with the fixed box. (0.9471)
• I am approaching directly, then rotate in place to align perfectly. (0.9619)
• I approach with a direct path but make a last-second adjustment to align perfectly. (0.9611)

2. Condition: I need to rotate the box first before aligning it with the target.

Inner Speech:

• I push the box directly without rotation, aiming for a straightforward alignment. (0.9183)
• I am approaching directly, then rotate in place to align perfectly. (0.9614)
• I am starting from a slightly rotated angle, needing a mid-action adjustment to align. (0.9616)

This shows that MIMIC mediates an agent’s actions by generating an appropriate inner speech based on its behavior during simulation. The high similarity with a true description in the CLIP space further shows that the generated inner speech is indeed meaningful.

Below we also discuss a couple of challenges to provide insights on potential failure modes:

• The fixed polling window W is a critical hyperparameter whose misalignment with task-specific behavioral rhythms can induce failure cascades. Excessive polling generates behavioral instability through premature re-planning (overcorrection), while insufficient polling permits accumulation of distributional drift (undercorrection), particularly problematic in tasks requiring precise temporal coordination.

• The CVAE's stochastic generation may produce inner speech vectors that progressively diverge from behaviorally-meaningful representations. While our polling mechanism provides periodic recalibration (which is a sweet byproduct of our approach as it helps mitigate the brittleness of Behavior Cloning), rapid environmental transitions could outpace the agent's cognitive updating, resulting in misaligned actions.

We will add discussions and examples in regard to these points to the final version of the paper.