Thanks for your valuable comments. Below are our responses to your concerns:

Q1: However such a claim would be stronger if supported by evidence or examples of frames being correctly clustering based on action semantics.

Here, we give evidence both quantitatively and qualitatively.

• Firstly, during inference, the prediction of the current frame comprises two components: the action clustering block ($P_{AC}$) and the distant neighboring frame Transformer block ($P_{DNTR}$), as indicated in Line 215. As illustrated in the table below, excluding the scores of $P_{AC}$ leads to significant performance drops. | Weight of $P_{AC}$ | THUMOS'14 (mAP%) | |:-----|:----:| | 1.0 |37.5| | 0.0 |34.1|

• Secondly, in the final version, we will visualize the attention matrix $A$ in the object-centric decoder. Following the computation of Eq. 3, we can obtain a representation of the grouping of the input video frames.

Q2: The object-centric decoder unit seems to be confusingly-named, since it is employed "to group frames" (line 149), rather than to focus on specific objects appearing within a frame.

We will clarify that in the object-centric decoder, the term "objects" refers to video frames.

Q3: Another baseline worth exploring would be video-text models (as opposed to image-text) applied via a sliding window.

We opted for ViCLIP [40], a straightforward video-text baseline model, to directly assess its performance for zero-shot online motion detection on the THUMOS'14 dataset. Following the pre-training settings of [40], we employed a sliding window that samples 8 frames for input into the visual encoder, aligning with the CLIP-II methodology. Moreover, we exclusively fed the last frame of the sliding window, aligning with the CLIP-I approach. The results presented in the table below demonstrate that utilizing the video-text pretraining model directly for zero-shot inference in online action detection yields unsatisfactory performance.

ViCLIP can access information from the entire video during pre-training; however, it can only capture very brief video frames during online motion detection inference. This limitation constrains its performance.

Q4: Do you see a difference between "open vocabulary" and "zero shot" action recognition?

The disparity between "open-vocabulary" and "zero-shot" action recognition lies in the fact that open-vocabulary enables the recognition of any category, whereas "zero-shot" ensures that the testing categories were not encountered during training [a].

In Line 106-107, we label our approach as "open-vocabulary" since our model is designed to ideally facilitate the online detection of any actions with the assistance of a vision-language model. In this context, "zero-shot" signifies that our method can be directly applied for inference on downstream datasets without the need for additional fine-tuning.

Q5: How much overlap between the textual descriptions in ActivityNet and InternVid with the category label sets of THUMOS'14 and TVSeries?

By utilizing the method of proximate comparison in natural language, we have pinpointed 8 analogous action phrases within the 20 action categories of THUMOS'14, encompassing 40% of its total categories. Likewise, we have identified 9 similar action categories out of the 30 in TVSeries, representing 30% of its total categories. For more detailed information, please refer to the response To-All-Reviewers-Q3.

Q6: Can you explain "the assumption of low background information for VLM" from line 36?

VLM are primarily designed for classification tasks rather than detection tasks. The optimization process of these models typically operates under the assumption that the background information (such as irrelevant environmental content) in the training data is less significant compared to the foreground content (task-relevant objects or regions). Essentially, these models are trained with the notion that each sample's main focus is on the foreground content, with less emphasis on the background, likely because classification tasks typically involve identifying the main objects in an image without detailed scene analysis. Consequently, VLMs may not be well-suited for detection tasks that necessitate precise differentiation between foreground and background elements. A similar perspective is echoed in STALE [29].

Q7: Can you clarify "fails to reach future frames during training" from line 37?

Given an untrimmed long video, the temporal action detection model can see the features of all frames, while the OAD model can only see the current frame and past frames in each iteration.

Q8: What precisely is the circle operator in equation (1)?

In this paper, the symbol "◦" represent function composition. If $Ψ_{AC}$ and $Ψ_{DNTR}$ are two functions, then $Ψ_{AC}◦Ψ_{DNTR}$ denotes the composition of them, which is defined as $Ψ_{AC}◦Ψ_{DNTR}=Ψ_{AC}(Ψ_{DNTR}(x))$.

Q9: On Line 175, how do you know "when the raw labels are not enough"?

In this context, "raw labels" refer to the captions associated with the selected video clips. Descriptions in large-scale video datasets often lack density. The OAD model utilizes a sliding window to extract multiple frames across the video, easily picking up segments with sparse or minimal captions.

Q10: In Table 2, VO_{ad}^{dagger} refers to OV-OAD, correct?

That's a typo. OVO_{ad}^{dagger} means OV-OAD.

Reference

[a] Hyun, Jeongseok, et al. "Exploring Scalability of Self-Training for Open-Vocabulary Temporal Action Localization." arXiv preprint arXiv:2407.07024 (2024).

Thanks for your valuable comments. Below are our responses to your concerns:

Q1: I wonder if other VLMs, such as ActionCLIP, could mitigate these drawbacks for the proposed task.

Thank you for the suggestion. We plan to enhance our model's performance in future work by incorporating improved feature extractors like ActionCLIP [38].

Q2: The comparison of OV-OAD with OadTR, seems to use different visual encoders/features.

Thank you for pointing out the lack of description here. In line 276, we noted that both the fully supervised methods and our OV-OAD utilize the same feature extractor. Specifically, both the base-to-novel fine-tuning methods and the fully supervised methods employ the same feature extractor, namely CLIP's visual encoder ViT/B. In the final version, we will emphasize this statement at the beginning of the paragraph.

Q3: The author should also compare the method with more recent methods.

Please refer to the response To-All-Reviewers-Q1 for detials.

Thanks for your valuable comments. Below are our responses to your concerns:

Q1: The authors should consider using more recent OAD models.

Please refer to the response To-All-Reviewers-Q1 for detials.

Q2: The improvements on the TVSeries dataset being relatively limited compared to THUMOS'14.

We believe that there are two reasons why OVOAD's performance in THUMOS'14 outperforms the improvements in the TVSeries dataset.

• First, all categories of ANet have different coverage ratios of 40% and 30% for the action categories of THUMOS14 and TVSeries, respectively. Similar text descriptions encountered by the text encoder during pre-training improve the accuracy of visual-text matching. Please refer to the response To-All-Reviewers-Q3 for specific calculations.
• Second, the evaluation metrics are calculated differently for the THUMOS'14 and TVSeries datasets, the former used mAP as a metric while the latter used cAP as a metric.

Q3: It would be beneficial to test on atypical human action datasets, such as creating a benchmark for the HDD dataset.

Thank you for your advice. We have validated the zero-shot performance of OV-OAD on EK100 with non-trivial improvement (1.3% cAP). Please refer to the response To-All-Reviewers-Q2 for detials. In the final version, we will evaluate our method on those challenging online motion detection datasets, e.g., FineAction and the mentioned HDD.

HDD [a] is a multimodal dataset in a driving scenario. In previous studies, e.g., OadTR [39] and Colar [45], non-visual sensors data are generally used as inputs to the model. The non-visual sensors are sourced from the vehicle's controller area network bus, encompassing critical metrics like car speed, accelerator and braking pedal positions, yaw rate, steering wheel angle, and the rotation speed of the steering wheel. In contrast, OV-OAD has only a visual feature extractor and does not directly process sensor data. Directly evaluating our current model on HDD could lead to unsatisfying performance due to the huge data distribution gap between our training (RGB videos for daily actions) and HDD.

Q4: The group embedding is the query, and the frame tokens is the key and value, but why is the dimension of the attention weight matrix $A \in n×k$ instead of $k×n$?

In the action clustering block, the group embeddings serve as the query, and frame tokens serve as keys and values. As shown in Eq.(4), we define $A = softmax(\frac{K@Q^{\top}}{\sqrt{d}}), A \in \mathbb{R}^{n \times k}$, with the softmax operator normalizing over $k$, and the output of the slot-attention block is $A^{\top}V$, which is of shape $k×d$. We will clarify the definition of $A$ in the final version.

Q5: Line 154 mentions that the number of neighboring frames $n$ can be a large value, and Table 4 shows the highest $n$ up to 8. Can the authors present experimental results for higher values of $n$?

Training our model with a larger $n$ than 8 (e.g. 12) needs V100×8 GPUs for ~5 days on ANet. We will add this study in the final version. We state that $n$ can be a large number relative to the number of group embeddings $k$. Generally, $n$ should be more than twice $k$. We will make these statements clear.

Q6: The parameter amount is twice that of LSTR, but the inference speed is six times as fast.

The inference of LSTR is much slower than ours due to two main reasons:

• It requires more input frames than our method.
• It relies on optic flow extraction, which is a slow process.

In particular, LSTR demands a larger number of input frames for optimal performance, using 520 seconds of video for inference on THUMOS'14, while our OV-OAD utilizes only 8 seconds. This means that LSTR requires 65 times more data than OV-OAD, which likely explains why our model's inference speed is six times faster. Furthermore, the primary speed bottleneck for LSTR is the extraction of optical flow (8.1 FPS), whereas for our model, it is the extraction of image features (292.3 FPS).

Q7: The paper lacks explanations for some operations, such as the operator "◦" in Equation 1.

In Equation 1, the symbol "◦" represents the function composition. We will include detailed explanations for clarity.

Q8: It is recommended that the authors add visible variables in the figures.

We will label the input and output tensors' names and dimensions of each module in Figure 2 for clarity.

Reference

[a] Ramanishka, Vasili, et al. "Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

Thanks for your valuable comments. Below are our responses to your concerns:

Q1: The statement "do not use any frame-level annotations" may appear contradictory.

Thank you for highlighting this inaccurate claim. We will rephrase it as follows: "avoid utilizing fine-grained temporal annotations, which involve humans labeling actions frame by frame to ensure no actions are missed."

Regarding our use of ANet, the annotations consist of timestamps and corresponding actions for each temporal segment. However, the timestamps lack the ability to differentiate between consecutive frames, leading to potential mislabeling of frames with incorrect actions. Additionally, instances within ANet may sometimes omit actions or assign incorrect labels.

For example, in sample 5n7NCViB5TU.mp4 from the training set lasting 121.44 seconds and depicting discus throwing by 3 athletes (referred to as Athletes 1-3), there are discrepancies. Athlete 1's action completion was marked within the time window $[22.4, 39.5]$, while ANet indicated $[24.3, 38.1]$ as the segmentation point. Similarly, for Athlete 2, the action completion time window was $[62.5, 73.1]$, with ANet lacking a corresponding segmentation label.

Q2: Zero shot evaluations on more datasets can help indicate the robustness.

We validate OV-OAD's zero-shot performance on EK100, showcasing a non-trivial improvement (1.3% cAP). For further details, please refer to the response To-All-Reviewers-Q2.

Q3: Different improvement for THUMOS’14 and TVSeries.

Please refer to the response To-All-Reviewers-Q3 for detials.

Q4: Comparison with more online action detection models.

Please refer to the response To-All-Reviewers-Q1 for detials.

Q5: The ablations on the action clustering unit.

We will ablate different parts of action clustering block soon. The reason behind using the object-centric decoder lies in its capacity to generate more semantically explicit segments compared to a standard transformer decoder block, owing to its differentiable clustering module. This superiority has been validated in [41-42]. Nevertheless, we will compare it with the aforementioned rival in the following ablation.

Q6: The ablation for not using the final transformer encoder.

Removing the final transformer encoder (also means abandon $L_{contrast}$) would leads to training anomalies and performance drops. Without the textual guidance from $L_{contrast}$, the open-vocabulary capability is significantly compromised. As shown in Table 5 (rows 2 vs. 3). we see that reducing the number of layers in the final transformer encoder from 6 to 3 results in a performance loss.

Q7: Figure 2 needs to be clearly mentioned for clarity.

The object-centric decoder automatically associates $n$ input frame tokens using $k$ learnable grouping embeddings, ultimately outputting grouped tokens (${\cal G}' \in \mathbb{R}^{k \times d}$) which also can globally describe the video clip. We will label the tensor dimensions of the module's outputs in Figure 2.

Here, we assess the performance of OV-OAD on the large-scale video dataset, FineAction, which includes ~4,000 uncut videos categorized into 106 classes encompassing Household Activities, Personal Care, Socializing, Relaxing, Sports, and Exercise. In line with FineAction [b], we conducted online action detection inference on the complete validation set of FinaAction, utilizing the per-frame mean Average Precision (mAP) metric. The subsequent table displays the results, illustrating OV-OAD's notable enhancement compared to the baseline methods.

Finally, we commit to addressing all concerns and incorporating the mentioned experiments in the final manuscript if the paper is accepted.