MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA

• The authors suggest that in terms of benchmarking, existing egocentric video benchmarks either focus on shorter videos, such as EgoSchema and QaEgo4D or on internet video content, such as video MME. This results in a significant gap in egocentric video understanding benchmarks, but the data for the proposed benchmark EgoMemoria still comes from Ego4D. I do not feel that EgoMemoria is significantly different from the benchmarks mentioned earlier, and the authors need to clarify this point further.

• Is it reasonable to only use GPT-4o in the automated annotation process? When generating QA pairs, I think that videos are equally important as dense captions, which will further ensure the quality of the QA pairs. The authors need to provide ablation experiments to validate that the existing method is more reasonable and ensures higher annotation quality.

• To ensure reproducibility, the authors need to provide the prompt templates for GPT-4o and ChatGPT used in the automated annotation and evaluation processes.

• The benchmark proposed in this work includes 629 videos. The authors classified them by length but did not provide the distribution of videos for each length category. This is crucial for the comprehensiveness and robustness of model evaluation. The authors should further elaborate on the distribution of video lengths.

• In Table 4, MM Ego's performance on Video-MME is not as impressive as LLaVA OV. The authors should provide a more comprehensive analysis to explain the reasons for this discrepancy. Otherwise, I might conclude that the construction method of MM Ego sacrifices its ability to understand general videos in favor of enhancing ego understanding.

• Language bias is inevitable in LLM, but whether it will be influenced by random responses in ego videos also needs to be verified through an ablation study.

• The Global Glimpse Step and Fallback Step described by the author would be more complete and convincing if they could be correlated with mechanisms related to cognitive neuroscience and brain science.

• Figure 5 may be better represented using a word cloud.

• Please note that it is not GPT4-o, but GPT-4o.

If the author can address the questions above, I will improve my rating.

• [W1] The introduced data engine works only with human-annotated narrations. In other words, it is not trivial to scale up since it relies on human labor. The data engine looks like it is designed explicitly for the Ego4D dataset, which has human-annotated video clip narrations. I am not too sure if the data engine is a practical and valuable contribution.
• [W2] The authors have not compared the introduced EgoMemoria with other datasets. Even if it has high-quality data thanks to the use of human annotations, it is hard to say 7k MCQs large-scale. The number of annotations in the test split would be even smaller. I believe the authors should make a table comparing the EgoMemoria with other egocentric datasets regarding the number of clips, annotations (QA pairs), and clip lengths.
• [W3] The authors have not shown the results when using only the EgoMemoria for training since they use an SFT dataset mixture. It would also be better to include the results using only the EgoMemoria for training.

1. The first claim of contribution, the “narration to egocentric QA” data pipeline, I believe, should not be emphasized as a major contribution. This type of generating QA from dense captions have been used in multiple previous works, from the non-LLM era (like TVQA) to the LLM era (like LLama-VID). I believe it is better to tone down this statement.
2. The generated EgoMemoria Benchmark does not stands itself out of many long video understanding benchmarks. Even we narrow down to only egocentric videos, the GroundVQA dataset is also good to be compared and especially be used to test the MMEgo model. I would also recommend the authors to compare a long of long video datasets, providing more proofs that this benchmark is not so incremental.

Overall, I think this paper is proposing a good model, while the benchmark side is relatively weak. I would recommend the authors to either tone down the benchmark in the paper to put more emphasize on this model, or improve the benchmark. I recommend the authors to consider using the datasets of EgoExo4D and EgoExoLearn, both of which also contain dense narrations of egocentric videos and should be very suitable for enriching your benchmark in terms of both size and diversity.

The the model struggles with long videos. Whereas, egovideos are known for always ON camera meaning the ability to process long / unlimited length video is utmost important. It's a general research question for the community.