Benchmark for Temporal, Ambiguous, and Grounded Embodied Question-Answering

● Although the authors demonstrate the diversity of condition combinations in Figure 3 and claim in lines 228-229 that they measure the diversity of predicates to prevent questions where most conditions are of the same type, they do not provide statistical information on the distribution of predicates. The richness of predicates more accurately reflects whether they meet real-world demands. Could the authors provide more evidence to suggest that the predicate set is diverse enough? For example, a breakdown of predicate types and their frequencies that demonstrates the diversity of the predicate set would be helpful. Additionally, more examples of complex questions with multiple conditions would also help illustrate the diversity.

● The authors use LLM combined with various image-to-text methods to perform multimodal tasks. However, as they mention, these approachs can lead to information loss and redundancy. Why the authors consider only evaluate on LLMs? It is also recommended that the authors could provide experimental results based on multimodal large language models, such as the flagship models GPT-4o, Qwen2-VL, and models developed for handling long temporal sequence data, such as LLaVA-OneVision and mPLUG-Owl3, etc. The evaluation results on these models can better reveal the challenges posed by the proposed benchmark for end-to-end multimodal methods.

● The proposed method heavily depends on the accuracy of object properties annotations; incorrect annotations not only create erroneous samples directly but also interfere with the combination and filtering strategies, leading to more cascading errors. However, as stated in line 251, the properties of objects are extracted by GPT-4o. How do the authors ensure the accuracy of the dataset samples? For instance, have the authors performed any manual checks, or employed additional methods to filter out incorrect samples?

1. Confusing terminologies. They do not follow terminologies that are used in the precedent studies. See questions.
2. Ignorance of the branches of the existing VideoQA benchmarks as they introduced so-to-say “passive EQA” and seem to abandon the efforts to position their studies among branches of text-video studies. They discuss the paper contributions without groundings to video QA references.
3. Lack of the task overviews and dataset examples. Because of the confusing terminologies, the overview of the tasks and settings is not well-depicted.
4. Lack of qualitative analyses with images or figures.
5. Conclusion section is not well-organized. Appendix also seems to lack textual descriptions.
6. Lack of the details of the “multi factor” and why they are considerably novel in their dataset compared to existing datasets. L219 discussed their framework includes diverse predicates, without statistics or qualitative comparisons.

1. The benchmark improves temporal reasoning but does not fully address temporal ambiguities, overly simplifying scenarios by using counters for repeated actions.
2. The structured approach and detailed evaluation might be difficult to scale due to the complexity of processing and generating queries that handle multiple ambiguity layers.
3. Experiments are not solid enough.
4. Paper structure should be optimized.

1. Overall, the article is difficult to read and follow, with significant omissions and formatting errors. For instance, Tables 2-4 lack the top and bottom rules, and the content for Appendices A.1-A.3 is missing. I do not believe the article is ready for publication in its current state.
2. I question the rationale behind posing ambiguous questions that are fundamentally flawed and should not be answered, rather than enhancing clarity by incorporating additional coordinate information.
3. While the authors introduce a novel dataset aimed at addressing ambiguity, it remains unclear whether this dataset is large and diverse enough to effectively evaluate the performance of various EQA models. 4.The selection of baseline models is limited. Although the authors acknowledge the challenges faced by modern EQA methods, they do not provide a comprehensive discussion of the specific limitations these models encounter when addressing ambiguous questions in 3D environments.