Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level

The paper would benefit from discussing some failure cases or limitations of the MORA model. Including challenging or negative examples in the experimental section could provide a more comprehensive evaluation and highlight areas for potential improvement.

While the paper proposes a new dataset and method, it could strengthen its contributions by comparing the model's performance not only on GROUNDMORE but also on existing, related datasets. This would help in better situating the improvements MORA brings within the broader research landscape.

I understand the segmentation mask of video frames is relevant to motion, but why the authors emphasise {Causal, Sequential, Counterfactual, and Descriptive} pair. Causal and Counterfactual seem irrelevant to segmentation mask.

1. While the proposed GROUNDMORE dataset is a nice contribution, it may be relatively limited in the variety of videos due to them being sourced only from the categories of family, animal, ball games, and outdoor activities. It could be helpful to expand the dataset to include scenes with multiple moving objects such as multiple cars. This would be helpful for evaluating how robust MLLMs are to different contexts of motion reasoning.

Moreover, it is mentioned in line 273 of the submission that the authors selected short clips which contain high degrees of motion semantics. However, it is not explicitly defined how the degree of motion semantics is actually being computed, such as leveraging optical flow information or other algorithms.

2. Given the video nature of this task, it is not well-motivated in the writing why the authors build their proposed MORA approach off the image-based reasoning segmentation framework LISA instead of the more recent video version, VISA [2]. In theory, the VISA model is trained with the objective of being able to perform spatiotemporal grounding of language queries in videos and should be able to generalize much better to the proposed task.

[2] VISA: Reasoning Video Object Segmentation via Large Language Model, ECCV 2024.

1. I think the definition of "motion understanding" is very vague. The paper seems to equate "action" with "motion" which is not particularly useful - shouldn't motion understanding encode an understanding of how something moves, rather than just what something is doing? As an example, answer "Who fed the dog with a piece of dog food after taking the dog food out from the cabinet?" does not require any understanding of how the dog food was taken out, how the feeding was done, etc. Things like "Feeding a dog" or "taking out dog food" have been studied as part of temporal action localization or action recognition. In this sense, I'm not sure this dataset really tests or requires true motion understanding. I'd love some clarification on this point. Another examples is from Figure 7, the most common verb is "use". I don't really see how segmenting a person using something requires motion understanding.

2. I'm not convinced the questions really require motion understanding. For example, from Figure 2, "Who points at the man in the orange shorts because the man in the orange shorts fouled?" doesn't require understanding that the man in orange shorts fouled. You just need to identify that someone is pointing at the man in orange shorts. Similarly, "Who drops the purple broom after seeing the panda would not leave the basket?" just requires finding a part where the woman drops a purple broom. Much of the paper states that the questions require understanding of motion semantics - this doesn't really seem clear to me, and I think needs more explanation about what that means. To me, it seems like the task is mostly about 1) identify an object with some reasoning module and then 2) performing video object segmentation. If this is the case, then all of the "motion understanding" component is through video object segmentation, and all the baselines proposed (along with MoRA) treat the two tasks totally independently, which runs contrary to the narrative from the introduction of the paper. At the end of the day, this then just becomes a segmentation benchmark and the "motion understanding" is essentially how strong the segmentation method is.

3. Using J&F doesn't seem like the right metric choice. this is essentially grading most of the output on the segmentation quality. Shouldn't there be some component that measures where the correct object was segmented? Is there a way to account for this in the benchmark? This seems to accoutn for the difference between each reasoning method (ViLA, SeVILa, VideoChat2) with the same segmentation method.

1.Although one of the most significant contributions claimed by the authors is to introduce implicit reasoning into the video question-answering and segmentation models, this idea has been previously explored by several works like VISA [1] and ViLLa [2]. It seems that there is no obvious difference in terms of the introduction of implicit textual inputs compared to these existing works, and this makes the contribution and novelty of this work not that impressive.

2.In terms of the Sequential question type in the proposed GroundMoRe dataset, I have some concerns on the effectiveness of it for truly reflecting the model's ability to reason about the temporal relations of motions. For example, as shown in Figure 1, the query input "Who dribbled the ball before he accelerates passing the man in pink shorts?" involves two consecutive motions, but there is only one motion of "dribbling the ball" in the video clip so the model can easily localize the first motion without looking at the second motion to find the correct answer, while this behavior cannot be regarded as temporal relation reasoning. I think similar problem will also occur in other examples mentioned in the manuscript like "the woman opened the refrigerator before taking out the milk". In my view, some samples should be explicitly constructed for this question type, where a same motion occurs twice but one of them is accompanied with another context motion, so the model has to rely on the temporal relation with the context motion to localize the answer.

3.To my understanding, the most interesting contribution of this work is to take the temporal grounding into consideration during the benchmarking process. However, it seems that the evaluation and investigation of temporal grounding capabilities are quite simple. For example, I don't see the authors discuss any metrics regarding the temporal grounding capabilities, such as the typically adopted temporal Intersection over Union (tIoU). Actually I think there should be a more comprehensive metric considering the spatio-temporal localization ability in this work, which can be derived from a simple extension to the existing vIoU metric proposed in Spatio-Temporal Video Grounding [3, 4]. But the current metrics discussed in this work are still focusing on the spatial segmentation.

4.According to the experimental results presented by Table 4, I feel kind of confused that for the Descriptive question type, the performance improvement is still remarkable when replacing the implicit questions with the referring expressions, i.e., the first row and second row for all methods in Table 4, and sometimes this improvement is even more obvious than other question types. Given that the Descriptive questions already convey a lot of description details, why could this phenomenon happen? Intuitively I think the difference between a descriptive question and a referring expression is quite minor, maybe there is just something like whether the concrete subject word is mentioned or not comparing these two kinds of textual inputs?

References

[1] VISA: Reasoning Video Object Segmentation via Large Language Models, Yan et al.

[2] ViLLa: Video Reasoning Segmentation with Large Language Model, Zheng et al.

[3] Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences, Zhang et al.

[4] Human-Centric Spatio-Temporal Video Grounding With Visual Transformers, Tang et al.