Bootstrapping Visual Assistant Modeling with Situated Interaction Simulation

We thank the reviewer for the insightful feedback and thoughtful questions. We especially appreciate your recognition of the novelty and scalability of the BASIS framework and the thoroughness of our experimental evaluations. Below we respond to each of the raised concerns:

R1: Faithfulness of user simulators and evaluation in Exp2

This is an excellent observation. In our work, we consider a good user simulator to possess two key properties:

• (1) Instruction-following behaviors when instructions are clear
• (2) communication behaviors to resolve uncertainty

To assess (1), we note that good instruction-following behavior can be proxied by success rate with oracle assistant (guidance from whom has been verified as effective through our real human study). A high SR@1 in this setting indicates that the simulated user can interpret and execute good instructions like humans. Simulators with very low SR (e.g., o3-mini, Gemini) typically lack the reasoning or grounding capabilities expected of human users and are excluded.

To assess (2), we examine behavioral signals such as average trajectory length and number of user dialogue turns. These serve as proxies for uncertainty expression—e.g., users seeking clarification or hesitating under ambiguous object references.

Ultimately, we balance both criteria to select a user simulator. For example, although the simulator with precise object references achieves slightly higher SR@1, the one using fuzzy object references better mimics humanlike ambiguity and leads to more interactive dialogues. Therefore, we selected the latter as more faithful, even if slightly less effective.

While we have considered this tradeoff, we acknowledge that faithfully simulating human behavior remains a significant challenge. Future work should further investigate how to quantify the similarity between model-expressed uncertainties and those of real users, as well as cover more diverse user behavior distributions.

R2: Interpretation of Exp4 and goal of RQ4 (user transfer vs. environment transfer)

Thank you for pointing this out. The primary objective of RQ4 is to evaluate how well an assistant trained using our framework can assist real human users under realistic conditions. In Exp4, the first row of Table 3 presents the performance of the GPT-4o oracle assistant, which receives privileged environment information; this is not a trained model, but rather serves as a reference to establish an upper bound on achievable performance. The second row in Table 3 reports the performance of our trained assistant with real users and is not intended for system-to-system comparison, but rather to contextualize how close our approach comes to that upper bound.

We appreciate the suggestion to compare the same assistant model across simulated and real users as a more direct test of user transfer. We agree, and will revise the paper to include the following comparison:

Interestingly, both the oracle and trained assistants perform better with real users than with simulated ones. This is somewhat counterintuitive, as sim-to-real transfer often entails performance degradation. We hypothesize that this is because our simulated users—though effective—are limited by their own reasoning and planning capabilities, and sometimes fail to follow instructions that real humans can easily interpret and execute. This suggests that simulated users, while useful for training, may underestimate the true potential of the assistant in real-world interactions. Based on this observation, we believe that absolute performance with real human users is a more reliable evaluation signal than comparisons between simulated and real users—unless the simulator is proven to closely mirror human behavior. We will incorporate this analysis and clarification into the revised version of the paper.

We thank the reviewer for the constructive and thoughtful comments, as well as the recognition of the novelty and clarity of our framework. Below we respond to each of the concerns raised:

R1: Limited technical novelty due to similarity with existing agentic frameworks

We would like to clarify the distinction between our work and the cited literature. While it is true that agent-environment simulation has been widely explored, the core contribution of our BASIS framework lies in the simulation of situated interaction—not only between an agent and its environment, but also in the dialogue and coordination between a user agent and an assistant agent. In contrast, the cited works typically involve a single agent learning by acting in the environment, often with scripted or goal-conditioned behaviors. In our case, the assistant must engage in multi-turn natural language interaction with a simulated user, reason over ambiguous goals, and guide task execution collaboratively. We argue that this user-assistant simulation, grounded in perceptual context and interaction history, is a novel contribution in the domain of training vision-language assistants.

R2: Limited generalizability and absence of environmental sim-to-real transfer

We acknowledge the limitation of evaluating within the Alexa Arena domain and agree that generalizing to broader real-world environments is an important direction. However, we would like to clarify that our goal is not to claim broad real-world deployability at this stage, but rather to demonstrate the feasibility and effectiveness of a simulation-driven framework for training multimodal assistants in a self-bootstrapping manner. As noted in Section 3.3, BASIS addresses a different type of sim-to-real challenge—from interacting with simulated users to interacting with real humans. This is different from the typical use of “sim-to-real transfer” (which often describes simulated environment to real environment). To avoid confusion, we will use “transferability to real users” instead. We believe this is a meaningful and under-explored transfer scenario. Our real-user validation shows that the assistant trained solely via simulated dialogue achieves 72.9% SR@1, or 82% of the oracle’s performance, demonstrating promising generalization from simulation to real human interaction. Exploring other domains and coupling with environmental sim-to-real transfer is a critical next step and will be prioritized in follow-up work.

R3: Limited human evaluation and lack of comparison to external baselines

We appreciate this concern and agree that broader human evaluation would be beneficial. However, the scale of our human evaluation reflects both our resource constraints and a broader practical reality: collecting human interaction data is expensive and time-consuming. In particular, gathering our human evaluation results requires over a total of 1,000 minutes of online interaction between human subjects and our system, where each session requires up to 50 turns of interactions from the user and assistant/environment. The focus of this work is not to present a new state-of-the-art assistant model, but to study the effectiveness of the BASIS pipeline. The design of our experiments centers around evaluating four specific research questions about simulation strategies, prompting methods, and user-agent interaction patterns. We will emphasize and clarify our objective in the revised version. Within this scope, we found the evaluation sufficient to support our claims. While our real-user evaluation is modest in scale (70 tasks), it provides diverse task coverage and reveals systematic strengths and limitations of our approach. We agree that future studies with expanded baselines and evaluation breadth would further enrich this line of work.

We sincerely thank the reviewer for the thoughtful comments and for recognizing the contributions of our work, including the structured simulation framework, thorough experiments, and detailed qualitative analysis. We address each concern raised below:

R1: Generalizability beyond visual assistant with specific settings

We appreciate this concern. Indeed, the current implementation of BASIS is applied within the Alexa Arena environment, which offers a well-scoped domain for testing the feasibility of our approach. Our primary aim is to demonstrate how multimodal LLMs can be developed through simulated user interactions without requiring extensive real-world human data. While the assistant is situated in a specific setup, the framework itself is domain-agnostic and modular: the simulation, modeling, and evaluation stages can be extended to other environments with suitable task definitions and multimodal interfaces. As noted in our future work, we are actively exploring extensions to more complex domains such as household manipulation tasks, which would provide broader validation of the framework’s generality. We view this work as a solid initial step that showcases the framework’s strong potential to be applicable across more diverse scenarios.

R2 & R3: Lack of baseline comparing fine-tuned vs. base model, and synthetic vs. real data

We do not include a comparison with the base Qwen2.5-VL 7B without fine-tuning because we found it had trouble following the output format. Instead, we highlight the following two comparisons that can show the effectiveness of fine-tuning on synthetic data. (see Table 2; also summarized below).

Specifically, we directly compare the Qwen2.5-VL 7B model fine-tuned on synthetic data with (1) the same model trained on a small set of real human dialogues and (2) the stronger GPT-4o model without any fine-tuning. These results highlight two key takeaways:

• For R2: Training on synthetic data boosts Qwen 7B performance beyond GPT-4o’s zero-shot capability, despite GPT-4o being a stronger base model.
• For R3: The synthetic data significantly outperforms real-human data collected for the same tasks, primarily due to its larger scale and broader coverage.

These findings should support the claims in question, and we will emphasize them in our revision.

We thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of our contributions, including the reduction of human supervision, the success of real user experiment and the comprehensive ablation studies. Below, we address the reviewer’s concerns and questions in detail:

R1: Size of real-human training data

We agree that the amount of real human training data is limited, but this reflects both our resource constraints and a broader practical reality: collecting human interaction data is expensive and time-consuming. In our case, gathering the real human data in our experiment already required over 1,000 minutes and significant hosting effort, plus $560 compensation, which is much costlier than generating thousands of synthetic trajectories. Our goal is not to argue against the value of more diverse human data, but to demonstrate that our framework can effectively bootstrap multimodal assistants with minimal human supervision. Improving performance with larger-scale human data is orthogonal to the core contribution of this work.

R2: Only Qwen & GPT-4o evaluated

We appreciate this suggestion. Our primary goal was not to benchmark across models, but to explore how different design choices in simulation affect the performance of a single assistant model. Thus, we focused our computational budget on training and analyzing several variants of the same model architecture for the control study. That said, we did evaluate multiple SoTA models in the user simulation phase (Table 1) to compare the robustness of the simulated dialogue generation.

R3: Missing prior visual-assistant baselines

Our goal in this paper is to evaluate the BASIS simulation framework, rather than to position a new model against existing visual assistant systems. While prior agents could be adapted to our setup, many are either task-specific or trained under different assumptions (e.g., pre-scripted instructions, not end-to-end trainable). To isolate the impact of simulation design choices, we chose to keep the experimental scope focused on variants within our framework. We agree that benchmarking against broader baselines is important and will be explored in future work.

Q1: Integration of external vision backbones or active perception

Thank you for the suggestion. We have not integrated external vision backbones or active perception mechanisms in the current study, as our focus is on evaluating the simulation pipeline rather than vision model enhancements. That said, incorporating stronger vision encoders or active exploration strategies could directly improve object identification performance and is a promising direction for future work.

Q2: Plans for collecting larger-scale human data with diverse communication styles

Indeed, our current experiments are conducted within Alexa Arena, a constrained environment with relatively low variability in user communication. As a next step, we plan to extend BASIS to more complex and naturalistic domains, such as household environments (e.g., TEACh [1]), where user communication exhibits more diverse linguistic and behavioral patterns. This extension will allow us to collect richer human data at a larger scale and further validate the generalizability of our approach.

Q3: Handling real-world camera noise and egocentric motion

We agree that bridging the sim-to-real gap in perception is a critical challenge for real-world deployment. In our current study, we assume relatively stable perceptual input during online evaluation. To address the noise and variability introduced by real camera streams and egocentric motion, techniques commonly used in robotics (e.g., noise injection [2] and domain randomization [3]) can be integrated during training to improve model robustness. While this aspect is not the focus of our current work, we view it as an important and complementary direction, and plan to explore it in future research.

References

• [1] Padmakumar, Aishwarya, et al. "Teach: Task-driven embodied agents that chat." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 2. 2022.
• [2] Ouyang, Hao, et al. "Neural camera simulators." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.
• [3] Chebotar, Yevgen, et al. "Closing the sim-to-real loop: Adapting simulation randomization with real world experience." 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.