Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?

We thank the reviewer for recognizing the novelty of our benchmark and task setup for interactive guidance, and for finding the design of our LiveMamba model interesting and effective for real-time interaction, particularly the “when-to-say” mechanism and the re-planner which effectively enables the model to interact in real-time.

Latency: On a Nvidia A100 GPU, our LiveMamba model (with a vision encoder, Q-former, Mamba model, and a re-planning module) has a real-time factor of 2 on average. This means that it can process input data twice as fast as it becomes available. In detail, we can process 4.1 frames per second on average and the input frame rate is 2 frames per second. A key advantage of the Mamba backbone of our LiveMamba model is its low memory requirement, which scales linearly with sequence length, compared to transformers whose memory requirements scale quadratically with sequence length. The peak GPU memory usage during inference of the LiveMamba model (with the Qwen2.5-32B-Instruct re-planner) is ~32GB to process ~500 frames. This makes it more suitable for deployment on edge devices such as phones or smart glasses with limited memory.

Pre-Training: As described in L184-191 in the main paper, the pre-training phase aligns the Q-Former’s vision embeddings with the language backbone’s text embeddings. We employ a diverse set of image and video datasets for object grounding (LVIS, EPIC-KITCHENS) and fine-grained action understanding (SSv2, EPIC-KITCHENS, Ego4D); these skills are essential for the cooking domain. Without pre-training (LiveMamba (w/o-PT)), we see a significant degradation of performance (on the simple planning set):

We will clarify this in the final version.

Mistake Prediction Accuracy: The reason why the performance of the baseline VideoLLaMA3-7B [54], Video-LLaVA [59, 33], VideoChat2 [29], Video-ChatGPT [37], LLaVA-NeXT [35] models on the mistake precision, recall, F1 and language metrics is poor is because they cannot detect mistakes. The reason for this poor performance on mistake detection is the fact that our LiveCook benchmark requires these models to detect mistakes in a streaming setup. However, these models were trained for offline question-answering tasks. More powerful models such as Qwen-2.5-VL-7B [15] can adapt to our streaming setup due to their strong instruction following capabilities. We explain the reasons for the poor performance of the baseline models in L243-253 of the main paper, and we provide the details of the prompts used for zero-shot evaluation in Appendix G of the supplementary material. We will clarify this further in the final version.

Offline Videos: Thank you for the reference to the highly relevant work of Xiulong Liu, et.al. that highlights the advantages of a real interactive environment. However, there are no extant simulators that can simulate fine grained actions which are common in cooking: e.g., cutting vegetables or adding 1 tablespoon of salt to a dish. Therefore, as described in L88-94 of the main paper, we use real-world offline videos which allows us to create a setup akin to a “non-reactive simulation”, where we task the multi-modal LLM to produce the right instruction or feedback at the appropriate time, but the subject is non-compliant. We will discuss and clarify these further in the final version.

We thank the reviewer for recognizing the importance of the problem we are addressing through the novel LiveCook benchmark and dataset, the clarity and structure of our paper, and the potential of our proposed LiveMamba approach demonstrated through extensive and promising experiments.

Annotation Quality: All annotations are created manually, and to ensure the quality of the annotations, they are manually verified independently by another annotator. The quality of the dataset is further highlighted in the qualitative example in Figure 1 of the main paper and the additional qualitative examples in Figures 4, 5 and 6 of the supplementary material. We also include the LiveCook benchmark data files in the uploaded supplementary materials. The annotation process is detailed in Section 3 of the main paper and Appendix B of the supplementary material. We will clarify this further in the final version.

Generalization: To evaluate the ability of multi-modal LLMs to provide step-by-step guidance, an ideal test domain should be both practical and accessible. We chose cooking because it meets these criteria: its tasks are relevant to daily life, and the specific recipes in our LiveCook benchmark do not require specialized knowledge. These qualities make cooking a robust environment for our study. Moreover, our model can be trained with data from other domains based on data availability, and generalizing to other domains is an interesting direction for future work. We will clarify this in the final version.

The Powerful Re-Planning Module: We would like to begin by clarifying that (as described in L172-181 in the main paper), if a recipe step is performed out of sequence, the re-planning module invokes the Qwen2.5-32B-Instruct model to decide if the current recipe step must be repeated and also update the task plan. Thus, note that the Qwen2.5-32B-Instruct model is invoked only on the advanced planning set in Table 4 of the main paper. In the simple planning set, the re-planning module only returns the next recipe step (thus, the re-planning module ensures that the LiveMamba model does not need to memorize the recipe steps). To better highlight the capabilities of the LiveMamba model without the re-planning module, we add a LiveMamba (w/o-reP) ablation to the results in Table 4 on the advanced planning set:

These results show that even without the re-planning module, our LiveMamba model is comparable to the Videollm-online baseline.

Furthermore, in the turn-based evaluation in Table 5 of the main paper, we do not use the re-panning module as the models are evaluated on each recipe step independently. We again see that our LiveMamba model outperforms state of the art multi-modal LLMs, even without the powerful re-planning module.

IC-Acc 30-second Window: Note that the 30 second window is centered at the end of each recipe step and thus includes the last 15 seconds of the current recipe step and the first 15 seconds of the next recipe step. The average length of a recipe step is 62.05 seconds. Thus the window for calculating the IC-accuracy covers on average the last 24% of the duration of a recipe step and the first 24% of the beginning of the next recipe step. In practice, this is sufficient to account for any ambiguities in the data along with any residual inaccuracies in the annotations while not sacrificing on accuracy (L214 in the main paper). We will clarify this in the final version.

Finally, note that even with this 30 second window, the results in Table 3 of the main paper highlight that the LiveCook benchmark is challenging even for current state of the art models like Qwen-2.5-VL-7B.

Other Public Datasets: The proposed LiveCook dataset is the only dataset that includes step-by-step task guidance along with a diverse range of timed feedbacks covering mistakes as highlighted in Table 1 of the main paper. Therefore, we restrict our evaluation to our proposed LiveCook dataset.

We thank the reviewer for recognizing the novelty of the LiveCook benchmark and dataset in enabling more realistic evaluation, and the innovative design of the LiveMamba model, which enhances real-time, interactive task guidance.

Generalization: To evaluate the ability of multi-modal LLMs to provide step-by-step guidance, an ideal test domain should be both practical and accessible. We chose cooking because it meets these criteria: its tasks are relevant to daily life, and the specific recipes in our LiveCook benchmark do not require specialized knowledge. These qualities make cooking a robust environment for our study. Moreover, our model can be trained with data from other domains based on data availability, and generalizing to other domains is an interesting direction for future work. We will clarify this in the final version.

Reasoning About Non-Linear Task Structures: We agree that the advanced planning set is very challenging for all models, as it requires reasoning about non-linear structures. However, the use of the re-planning module in our LiveMamba model leads to improved performance, as it is able to update the task plan on the fly as described in L168-181 in the main paper. This is highlighted in Table 4 of the main paper where our LiveMamba model performs significantly better compared to the prior work such as Videollm-online [6] on the advanced planning set of our LiveCook benchmark.

Computational Efficiency: Re-planning using the Qwen2.5-32B-Instruct re-planner takes 8.2 seconds on average on a Nvidia A100 GPU. Note that, as mentioned in L109 in the supplementary material, the Qwen2.5-32B-Instruct re-planner is called once every 2.5 minutes on average on the advanced planning set. In terms of throughput, on a Nvidia A100 GPU, our LiveMamba model (system with a vision encoder, Q-former, Mamba model and a re-planning module) has a real-time factor of 2 on average. This means that it can process input data twice as fast as it becomes available. In detail, we can process 4.1 frames per second on average and the input frame rate is 2 frames per second.

Annotation Cost: We rely on manual annotations for our LiveCook dataset and benchmark to maintain accuracy. It is currently not possible to automate this data generation pipeline as demonstrated by the poor performance of state of the art multi-modal LLMs in detecting mistakes in Table 3 of the main paper. Thus, it is crucial to have human expert annotators provide the ground-truth for the mistakes and their timestamps in a video sequence. This also highlights the importance of our LiveCook benchmark and dataset as the first to provide timed instruction and feedback messages covering a diverse range of mistakes in the cooking domain.

Counterfactual Action Descriptions: The counterfactual action descriptions generated using our data augmentation scheme leads to improved performance of our LiveMamba model as highlighted in Table 4 of the main paper (LiveMamba (w/o-DA) vs LiveMamba (ours), where LiveMamba (w/o-DA) is the model trained without the proposed data augmentation scheme). We see a significant increase in performance across all metrics with our data augmentation scheme. We agree that the addition of more nuanced errors could improve performance even further, however, generating such data automatically is challenging with current state of the art multi-modal large language models, and is thus an interesting direction for future research. We will include samples of the augmented data in the final version.

Nuanced Evaluation of Feedback Quality: In addition to the “binary” Instruction Completion Accuracy (IC-Acc) and mistake detection precision (Prec.), recall (Rec.) and F1 metrics, we also include the mistake feedback fluency (BERT and ROUGE-L) metrics. This allows for a more nuanced evaluation of the feedback quality in Tables 3, 4 and 5 in the main paper.

We thank the reviewer for recognizing our contributions: establishing a baseline for the challenging problem of real-time interactive task guidance, the value of our ~100-hour annotated dataset, and the strengths of our evaluations and system design.

Latency and Throughput of the Models: Our LiveMamba model consists of a vision encoder, Q-former, Mamba model, and a re-planning module. As explained in L172-176 in the main paper, only when the user does not perform the intended step (on the advanced planning set), the Qwen2.5-32B-Instruct re-planner will be invoked. Otherwise, the re-planning module just returns the next step in the plan. In terms of throughput, on a consumer Nvidia A100 GPU, our LiveMamba model has a real-time factor of 2 on average. This means that it can process input data twice as fast as it becomes available. In detail, we can process 4.1 frames per second on average and the input frame rate is 2 frames per second. In terms of latency, that is time to generate the first token, is 1.1 seconds on average. Re-planning using the Qwen2.5-32B-Instruct re-planner takes 8.2 seconds on average. Note that, as mentioned in L109 in the supplementary material, the Qwen2.5-32B-Instruct re-planner is called once every 2.5 minutes on average on the advanced planning set.

The target backend infrastructure for our LiveMamba model are edge devices such as mobile phones and smart glasses. A key advantage of the Mamba backbone of our LiveMamba model is its low memory requirement, which scales linearly with sequence length, compared to transformers whose memory requirements scale quadratically with sequence length. The peak GPU memory usage during inference of the LiveMamba model (with the Qwen2.5-32B-Instruct re-planner) is ~32GB to process ~500 frames. This makes our LiveMamba model more suitable for deployment on edge devices with limited memory compared to standard transformer architectures. We will clarify this in the final version.

Windows for Calculating IC-Accuracy: Note that the 30 second window is centered at the end of each recipe step and thus includes the last 15 seconds of the current recipe step and the first 15 seconds of the next recipe step. The average length of a recipe step is 62.05 seconds. Thus the window for calculating the IC-accuracy covers on average the last 24% of the duration of a recipe step and the first 24% of the beginning of the next recipe step. In practice, this is sufficient to account for any ambiguities in the data along with any residual inaccuracies in the annotations while not sacrificing on accuracy (L214 in the main paper). We will clarify this in the final version.

Finally, note that even with this 30 second window, the results in Table 3 of the main paper highlight that the LiveCook benchmark is challenging even for current state of the art models like Qwen-2.5-VL-7B.

The Re-Planning Module: As described in L172-176 in the main paper, if a recipe step is performed out of sequence (on the advanced planning set), the re-planning module in our LiveMamba model is able to decide if the current recipe step must be repeated and also update the plan, this is when the Qwen2.5-32B-Instruct re-planner is invoked. Otherwise, the re-planning module will just return the next step in the plan. We highlight the advantage of the re-planning module in Table 4 of the main paper on the simple planning set through an ablation of our LiveMamba model without the replanning module (w/o-reP). Furthermore, here we add this LiveMamba (w/o-reP) ablation to the results in Table 4 on the advanced planning set:

These results again highlight the advantage of the re-planning module in our LiveMamba setup.

Finally, note that in this work we are primarily focused on the cooking domain, and we find that this re-planning module is sufficient in such cases. Extension of our re-planning module to other domains is an interesting direction for future research.

Generalization Studies: To evaluate the ability of multi-modal LLMs to provide step-by-step guidance, an ideal test domain should be both practical and accessible. We chose cooking because it meets these criteria: its tasks are relevant to daily life, and the specific recipes in our LiveCook benchmark do not require specialized knowledge. These qualities make cooking a robust environment for our study. Thus, we propose LiveCook, the first large scale benchmark and dataset with timed instruction and feedback messages for step by step task guidance. We leave the generalization of our method to other procedural tasks as a direction for future work. We will clarify this in the final version.

Because the LiveMamba model only tuned to recognize mistakes on the cooking data, currently it may not be able to capture errors in other domains. But our model can be trained with data from other domains based on data availability.

Learning from Works in Agentic AI: This is a great point and an interesting direction for future research. We will discuss the MM-ReAct and SayCan papers in the final version.

Demo: We will include demo videos of the LiveMamba system with the final version. We also include qualitative examples in Figure 3 in the main paper and Figure 7 in the supplementary material.

Open-source: We will release both the LiveCook benchmark and dataset and the LiveMamba model code (pending legal approval).

Thank you for acknowledging the rebuttal.

We would be glad if you could let us know if the rebuttal addressed your concerns and if not, we would be glad if you could let us know if you have any additional questions.