What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions

Thanks for taking the time to for reviewing the proposed work. We would have some question that would help us to better understand your points.

Weaknesses/Questions:

1. Novelty is limited: Thanks for the suggestion. We have cited related works such as X-CLIP, Temporal Alignment Networks, ... etc. for related works. Could you please provide the related papers you have in mind for us to cite?

2. Experiments are not convincing: Please provide which part is not convincing. We will provide more details and justification. Also, we will create a table for statistical comparison.

3. Compared methods are out-of-date: We will add additional baseline GLIPv2 as suggested by reviewer tJVe29. Could you let us please know what methods you have in mind for comparison?

Thanks for taking the time to for reviewing the proposed work and for providing valuable feedback! We try to address the raised points in the following:

Weaknesses:

1. Comparison with ActivityNet-Entities : ActivityNet entities actually focuses only on objects in the video, providing bounding boxes exclusively for noun phrases, while our training and evaluation are action-focused, such as 'cutting apples' and 'frying eggs'. We propose to a respective discussion in the related work, but we think that ActivityNet-Entities might not be a good benchmark for the spatial-temporal grounding of actions. (Please see also our answer to Reviewer CNqU)

2. Pointing game metric: Thanks for the comment. The pointing game is a relevant metric for grounding in the detector-free setting see e.g. [1, 2] and many more, where the bounding box proposal wasn't used/provided. We will add the citation to the paper.

[1] Arbelle, Assaf, et al. Detector-free weakly supervised grounding by separation. ICCV. 2021. [2] Eyal Gomel, et al. Box-based Refinement for Weakly Supervised and Unsupervised Localization Tasks, ICCV, 2023

3. Description missing: Thanks for the catch. We will update the reference.

4. How is IoU+Pointing game computed? : We include the details in the supplement section 8.3 Evaluation metrics. IoU+Pointing game combines the evaluation setting from the spatial grounding (Akbari et al., 2019) and temporal grounding (Kuehne et al., 2019) metrics. For each video frame, the prediction is correct when the model predicts the correct action for the frame. Also, given the predicted action as a query, the maximum point of the heatmap aims to be within the desired bounding box. We then computed the Intersection over Union (IoU) over all the predictions with the GT to acquire the final score.

5. Missing references/baselines: Thanks for the suggestion! We found that [5] doesn't provide public code, and [4] achieved similar performance as one of our baseline GLIP. We will include GLIPv2 as our additional baseline.

Thanks for highlighting the strengths of the proposed paper and for providing valuable feedback! As all weaknesses are complemented with respective questions, we will focus on those specifically.

Questions:

1. Visualization to demonstrate the OT for key frame selection?: Thanks for the suggestion! We will work on the visualization and report it soon.

2. Modeling the temporal relation between key frames: Thanks for the question. We would like to answer in multiple ways here: First, we actually evaluate the framework with both, 3D and 2D backbones in Table 2, namely a S3D backbone which captures low-level temporal dynamics as the requested ones. Table 2 also shows that even while CLIP does not capture such temporal dependency it still performs slightly better than the S3D backbone. This can be considered a phenomenon known from various current video approaches using CLIP backbone, such as [1,2]. To address this issue, we offer to include an additional experiment with the TimeSformer backbone, which also utilize a visual transformer that encodes temporal dynamic, for the camera-ready version. To reason why CLIP might perform better than the S3D backbone, we want to point out that our dataset is targeted to be used for grounding instead of classification. Hence, our text queries seldom contain explicit complementary semantics such as pick up and put down as it would be labeled in action classification tasks.

[1] Luo, Huaishao, et al. "Clip4clip: An empirical study of clip for end-to-end video clip retrieval and captioning." Neurocomputing 508 (2022): 293-304. [2] Bain, Max, et al. "A CLIP-Hitchhiker's Guide to Long Video Retrieval." arXiv preprint arXiv:2205.08508 (2022).

3. Justification or empirical evidence for the preference of key points: We have conducted an annotation study in supplement Figure 6,7,9. Our finding is that our annotation reaches high consistency and captures meaningful variety for evaluation. In general we found in the initial phase of the annotation that the center point of action is more intuitive than the bounding box since it is hard to give any guidance of the scale of the bounding box, thus the physical outline for an action, which varies e.g. with respect to annotations and viewpoints of the hands, human, or instrument. As an alternative view, if we would predefined certain label criteria (e.g. always include the hands), the model would not be aware of those criteria without being trained on it, which would defy the idea of a weakly-supervised training without the need for annotation. Instead, we used the current spread of center points to define the outlines of the region that we want the model to localize on.

Please let us know if you have any other questions or concerns regarding our paper or response.

Thank you for reviewing our paper, pointing out the strengths of our work, and providing valuable feedback! We hope to address some concerns in the following:

Questions:

1. Performance on ActivityNet entities - Why can we not use ActivityNet as a benchmark for STVG? : ActivityNet entities actually provides bonding boxes for noun phrases in the captions, namely the objects. It is thus closer related to an object detection task and does not capture any spatial or temporal extent of actions. As our setup and evaluation are action-focused, such as cutting apples and frying eggs, ActivityNet entities might not be a good benchmark here. We offer to include ActivityNet entities and to add the comparison to clarify this. (Please see also our answer to Reviewer tJVe)

2. How long are the videos in the GoundingYoutube benchmark? : Our annotation is based on the MiningYoutube dataset, where each video is around 3-5 minutes with an average of 41.6 labeled action instances per video. Note that each video contains various classes. We will clarify this in the revision.

3. Evaluate the performance of your method using m_tIOU and tIOU@?: Thanks for the comment. The mIOU metric reported in Table 2 is the same as the vIoU in the HC-STVG paper. We will clarify this in the paper. The HC-STVG paper did not report tIoU@ with various thresholds, but we agree that this could be a valuable add-on. We currently work on the tIoU numbers and hope to report them soon.

4. Notation of T in Sec 3.1 : = Thanks for the catch. The T in Section 3.1 should be U. We will correct this typo. The frames were sampled with 5 FPS mentioned in the supplement section 9.1. We will clarify this in the updated version.

5. Sharpest frame sampling: Thanks for the suggestion. It is worth mentioning that we have evaluated the number of frames to sample, including the single-frame strategy, in supplement Table 10. We find using a single frame for training results in low performance due to the lack of temporal dynamics. We further assume that technically, the frames with significant blur should have low visual language similarity, and will therefore not be selected during our training. To validate this intuition, we propose to evaluate the sharpest frame strategy for the camera-ready version as it requires training of the system.