Boundary Denoising for Video Activity Localization

General Answer

We sincerely thank the reviewer for his time dedicated to assessing our work. While we understand that our manuscript may have been challenging to comprehend, we appreciate the reviewer's feedback and have attempted to address the confusion in the revision. We would like to point out that other reviewers have all expressed positive feedback regarding the presentation of the idea. Nonetheless, we are very receptive to constructive and concrete suggestions that can improve clarity and readability and improve our submission. Follow a general answer tackling the presented weaknesses. In the next comment box, we have a more discussion on Denoising vs Diffusion, and also answer the specific questions asked by the reviewer. Note that all changes to the manuscript are highlighted in blue in the revised version for the convenience of the reviewers.

1- Figure 2. We revised the caption of Figure 2. The reviewer can verify the change in the updated manuscript. We report here the updated text for ease of reference.

“Our proposed model consists of three main components: Encoder, Decoder, and Denoising Training. The input video and text are transformed into video and text embeddings by means of pretrained networks, then concatenated and fed to the Encoder module (left). The output features from the Encoder, termed as memory, are subsequently routed to the Decoder. During training, the Decoder receives in input a set of learnable embeddings, ground truth spans with additional artificial noise, and a set of learnable spans. The objective of the decoder is to enrich instance-level features computed based on the two sets of spans and refine (via denoising) the provided ground truth spans. Note that the GT spans are not used during inference. ”

We would like to clarify that the pipeline refers only to the training phase. The behavior of the model at inference time is presented in the manuscript in Section 3.2, where we explain that the Denoise is disabled during inference. The decoder is the module that refines the learnable temporal spans used as predictions. Further details are present in Section 3.2.3, where we detail the boundary denoising training.

2- DynamicConv. We would like to clarify that DinamicConv is not a contribution of our work. It was in fact, introduced in Sparse-R-CNN [1] and was borrowed as is. We adopt this computation block in pursuit of a good Decoder design and ablate several configurations in Section 4.5. Therefore, DynamicConv is not a necessary block for the correct functioning of our method, yet it is clearly shown to help achieve the best performance in the reported ablation in Table 5, as the module operation is more sensitive to the temporal feature extracted from the temporal span. Nonetheless, we would like to reinforce that our proposed method brings improvements even when such a block is not utilized. See Tab 5, row 1.

We are happy to provide further clarification if the reviewer deems it necessary.

[1] Sun, Peize, et al. "Sparse r-cnn: End-to-end object detection with learnable proposals." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

3- Section 3.2.3. We reorganized this section based on the reviewer's feedback. Please directly check it from our updated version.

4- Missing literature. We would like to kindly invite the reviewer to provide us with a list of relevant literature papers exploring the temporal boundary denoising problem in video localization tasks that were available before the submission of this manuscript. We believe he might find this literature lacking, reason why we decided to focus our effort in this direction. To the best of our knowledge, the relevant literature is included, however, we acknowledge that one might always miss some recent and promising works. We are happy to include missing literature in our related work and compare performance where possible.

5- Evaluation protocols. We would like the reviewer to point out the non-standard evaluation protocols involved in this manuscript. We base our work and evaluation code on the Moment-DETR repository. Please also note that the test-split annotation is not open to the public, so we submit our predictions to their server for a fair comparison. We are, however, open to further entertaining a discussion regarding this topic with the reviewer to clarify any doubts.

6- Additional ablations. We kindly ask the reviewer to provide actionable suggestions on the ablation designs. This would greatly help us in narrowing down the relevant experiments, helping us in designing and analyzing the results against potential expectations the reviewer might have.

General Answer

We sincerely thank the reviewer for his time dedicated to assessing our work, acknowledging its relevance, and providing suggestions on how to strengthen it. We are excited the reviewer has found our work technically robust and well-presented. Moreover, the reviewer's belief that our work makes a valuable contribution to the community is especially gratifying.

In response to the feedback, we have revised our manuscript to ensure consistency in the naming of our model throughout the document. Note that all our changes are highlighted in blue in the revised manuscript for the convenience of the reviewers.

Furthermore, we are very receptive to the idea of including additional visualizations and would welcome any specific suggestions the reviewer may have in this regard. Currently, we provide an Illustration of the boundary denoising process in Figure 1, where we showcase that we can effectively achieve more accurate temporal boundaries through the cascade of denoiser blocks. This figure showcases the efficacy and cascade effect of the denoising tower. Additionally, in the Appendix (Figure 5), we offer visual representations of the solution process for a specific video-query pair, visually illustrating the task.

We are happy to provide additional visualizations following the reviewer's suggestions.

General Comment

We thank the reviewer for his time dedicated to assessing our work. We are pleased the reviewer acknowledged the soundness of our proposed solution and our presentation. Moreover, the reviewer's belief that our work makes a valuable contribution to the community is especially gratifying.

In response to the concerns raised, we have incorporated the relevant citations in Sections 1 and 2, as suggested by the reviewer. Furthermore, we have addressed the mentioned typo and made a few additional corrections. We have also refined the notation in Section 3 of the manuscript to enhance clarity. We assure the reviewer that the adjustments are minor and do not impact any of the equations. We kindly invite the reviewer to review these modifications, which encompass all the suggestions and hope that they will find these changes to be satisfactory. Note that all our changes are highlighted in blue in the revised manuscript for the convenience of the reviewers.

Additionally, we would like to clarify the boundary-related losses adopted in this work, and presented in Section 3.3, are L1 loss and gIoU loss. These losses directly affect and guide the denoising process.

Finally, our approach can be perceived as an advanced form of proposal augmentation. However, it is important to emphasize that the core concept involves controlling the noise level and adjusting the ground truth accordingly while training the model to predict the original temporal span under these specific conditions.

Question 1: Temporal span definition.

We are pleased to inform the reviewer we added a short sentence in the problem formulation to formalize the definition of temporal span. We have also redacted the document to only use the naming “temporal span,” removing any instance of “action span.” We are sure this improved clarity. We thank the reviewer for the suggestion.

Question 2: Clarification from Sec. 3.2.3.

When creating negative temporal spans, for simplicity, we formulate the set as adding noise to a fixed proposal with temporal span (0.5, 0.5) in center-width coordinates. These coordinates translate to (0.25, 0.75) in (t_start, t_end) convention. We find this simple strategy to be effective and robust to different levels of noise.

General Comment

We sincerely appreciate the reviewer's thorough assessment of our manuscript and are happy to hear that they found our proposed method both sound and our presentation of high quality. It is positive to know that our experimental setup effectively demonstrates the superiority of our method over current state-of-the-art approaches and that no further essential ablation studies are required. Moreover, the reviewer's belief that our work makes a valuable contribution to the community is especially gratifying.

Below, we have addressed each of the concerns raised. We hope these responses provide comprehensive clarifications. Should there be any further queries or need for additional clarification, we warmly invite the reviewer to reach out. Note, that all our changes are highlighted in blue in the revised manuscript for the convenience of the reviewers.

1 - Temporal span representation.

The reviewer is concerned about a possible invalid temporal span due to noise injection. First, we would like to clarify that our manuscript uses the start-end convention (t_start, t_end) for temporal span representation, aligning with standard practices in the field. This choice was made for consistency with the literature on this task.

In our implementation, these temporal spans are initially expressed as (t_start, t_end) and then converted into a center-width (center, width) format before adding the noise. To prevent the generation of empty/invalid temporal spans, the width is clamped to a minimum of 1e-4 after adding noise. Furthermore, our implementation of Region of Interest (ROI) alignment can ensure that only valid features are computed to extract proposal features, even in cases where the span width exceeds the video duration.

For a more detailed understanding, we invite the reviewer to consult the code provided in the supplementary material if deemed necessary. Particular attention may be directed to the functions span_cxw_to_xx() and span_cxw_to_xx() utilized during the forward pass operations, as illustrated in files code/model.py:line196, code/model.py:line214.

It is also important to note that before the computation of the loss, the temporal coordinates are converted back to the original (t_start, t_end) format. While the loss computation could, in theory, be performed in the center-width format, we consider this a matter of implementation detail and chose the current approach for consistency and clarity.

2- ROI alignment and DynamicConv.

We would like to clarify that these two modules were not introduced in this work but were adopted from previous work. We are happy to reference further the original manuscripts that introduce them. DynamicConv is directly borrowed from Sparse-R-CNN [1], while ROI alignment was initially introduced in its 2D form in Mask-R-CNN [2] (Figure 3) and successively generalized for 1D signals in [3]. We have updated our Appendix in Section G to include graphical visualizations of the two modules.

**3- Boundary-denoising process. **

We thank the reviewer for the opportunity to elaborate on the role of negative samples in our boundary-denoising process and have taken this chance to enrich the manuscript with additional details in Section 3.2.3.

The distinction between positive and negative samples is only relevant when generating noisy temporal spans. For positive samples, noise is added to the spans annotated in the dataset. Conversely, for negative samples, noise is introduced to predetermined temporal spans. Following this, both sets are combined and processed through the transformer decoder, as detailed in Section 3.2.3.

This unified set then undergoes our denoising process before being presented to the Hungarian Matcher for loss computation. Notably, during this stage, negative samples have little probability of being paired with growth truths. Such pairings, when they occur, are effectively penalized by our chosen loss functions.

The reviewer cleverly noted the possibility of negative samples being denoised in a manner that causes them to overlap with positive ones. While this is a possibility, we would like to emphasize that, in practice, we have not observed this phenomenon to disrupt the training process.

4- Comparison Fairness.

We have structured our response to the reviewer's concern regarding comparison fairness into distinct parts.

I thank the authors for providing the careful response.

The added details and graphical illustrations are indeed helpful, although I am not sure whether such a large modification in the manuscript is allowed during the rebuttal.

After all, most of my concerns are addressed, but I agree with some points of Reviewer 61UD on the other hand.

Hence, I will raise my score to 6, yet still remaining on the borderline.