Unified Static and Dynamic: Temporal Filtering Network for Efficient Video Grounding

We are so encouraged and cheerful to receive your constructive comments, your positive reception of UniSDNet's biological motivation and recognition of our two-stage information aggregation methods are both encouraging and deeply appreciated.

Q1: What is the motivation behind the implementation of Static Semantic Supplement Network? I am wondering how the cross-modal interaction is performed through the MLP layers. To my understanding, the shared weights across different modalities would extract some common features spanning different modalities. Some analytical experiments on this would be beneficial. Also, the architecture design seems similar to that of Transformer blocks except for the self-attention. What happens if we use the conventional Transformer layers?

A1: Thank you for raising the concerns about the motivation of static network. We would like to explain the implementation of our static network from the perspectives of motivation, technology, and experiment.

• Motivation-wise, the implementation of Static Semantic Supplement Network is strongly inspired by the global neural workspace (GNW) theory in biology [R15], that is, for video understanding, the brain will perform 'global broadcast communication' (interaction) of the multi-source information in the early stage of visual perception for the video, while the cognitive consciousness of the brain will strengthen perceptual conditions and delay conditioned reflexes, enabling the human brain to understand new conditioned reflexes brought about by perceptual information updates.

  On the premise of the above theory, this inspires us to utilize residual networks structure to update our understanding of video sequences in the delaying conditioned reflex states by aggregating the perceptual information. Multilayer perceptron (MLP) is proven to be effective and has strong interpretability in modeling multi-level cognitive processing [R19], thus we use MLP to aggregate the perceptual information. The visual consciousness process is modeled as follows：$x^{n+1}$ = $\text{ResMLP}(x^n)$ = $x^n$+$\text{MLP}(x^n)$, where $x^n$ represents the perception condition in the current state, $x^{n+1}$ denotes the reflection result, and $\text{ResMLP}(x^n)$ can enhance the information of the perception condition $x^n$.

• Technology-wise, we concatenate multiple queries and video clips together as a sequence input into the network for fully connected interaction of all tokens, its main function is to supplement and associate semantics between modalities before the latter graph filtering, provide more video descriptions information and significantly fill the gap between vision-language modalities, aiding in understanding video content.

• Experiment-wise, as per your valuable suggestion, we have tested the effect of Transformer as a static network as shown in the table below (Table R6), and as you are concerned, Reviewer 'TT77' has also focused on the ablation study of the static module. From the results, in terms of performance and efficiency, Transformer is close to our method, but our results are better. We speculate that the reason is that our network also includes the second stage of graph filtering. The static network uses a lightweight and stable network, which is more conducive to model training. Using Transformer as a static network increases the weight and instability factors [R20] of the network.

Table R6: Results of different static networks on the ActivityNet Captions.

References:

[R15] Volzhenin et al., Multilevel development of cognitive abilities in an artificial neural network. Proceedings of the National Academy of Sciences, 119(39), pp. e2201304119, 2022.

[R19] Chavlis et al., Drawing inspiration from biological dendrites to empower artificial neural networks. Current opinion in neurobiology. 70, pp. 1-10, 2021.

[R20] Hugo et al., Resmlp: Feedforward networks for image classification with data-efficient training. TPAMI, 45(4):5314–5321, 2022.

Thanks to the authors for the careful response.

I believe most of my concerns are appropriately addressed.

Especially, the ablative experiments regarding the number of queries as well as the results on QVHighlights are substantially valuable.

I will keep my rating unchanged with the hope that the authors will reflect on the discussion made during the rebuttal in the manuscript.

Thank you very much for your recognition of our contributions to NLVG and SLVG tasks. Your constructive comments are exceedingly helpful for us to improve our paper.

Q1: The inspiration from human visual perception biology is not very motivating. Specifically, it is hard to see why a MLP with residual connection is the way to achieve the 'global broadcast communication' of the brain. Either bridge the gap or Simply drop the bio-inspiration and go straight into the technical method.

A1: Thank you very much for your attention to the design motivation of static networks (MLP with residual connection, ResMLP). Here, we clarify the motivation to bridge the gap between the biological theory 'global broadcast communication' and the technical design of static network ResMLP. Sincerely, our work has been strongly inspired by the latest theories of human visual perception [R14, R15]. The global neuronal workspace (GNW) theory mentioned in [R15] is the foundation of conscious work in visual understanding [R15, R16, R17], it can achieve the fusion process of multi-source information in the early stage of visual event recognition, that is, the "global broadcast communication" of the brain. GNW theory indicates that cognitive consciousness has the function of strengthening perceptual conditions and delaying conditioned reflexes [R15, R18], enabling the human brain to understand new conditioned reflexes brought about by perceptual information updates [R15, R16].

On the premise of the above theory, this inspires us to utilize residual network structure to update our understanding of video sequences in the delaying conditioned reflex states by aggregating the perceptual information. Multilayer perceptron (MLP) is proven to be effective and has strong interpretability in modeling multi-level cognitive processing [R19], thus we use MLP to aggregate the perceptual information. The visual consciousness process is modeled as follows: $x^{n+1} = \text{ResMLP}(x^n) = x^n+\text{MLP}(x^n),$ where $x^n$ represents the perception condition in the current state, $x^{n+1}$ denotes the reflection result, and $\text{ResMLP}(x^n)$ can enhance the information of the perception condition $x^n$.

Technically, in this work, we adopt a unified framework of static (ResMLP) and dynamic (graph filtering) structures for video grounding. In the early stage, this global broadcast mode of ResMLP first perceives all multimodal information, while the latter graph filter purifies key visual information. This pre-broadcast learning mode of ResMLP is very necessary as a global biological perception mechanism and has also verified its effectiveness via extensive experiments, as discussed in the paper. We will add the explanation on the rationality of our static network design from a technical perspective and incorporate your suggestion into the paper writing by considering both motivation and technical implements for better clarification.

References:

[R14] Barbosa et al., Interplay between persistent activity and activity-silent dynamics in the prefrontal cortex underlies serial biases in working memory. Nature neuroscience, 23(8), pp. 1016-1024, 2020.

[R15] Volzhenin et al., Multilevel development of cognitive abilities in an artificial neural network. Proceedings of the National Academy of Sciences, 119(39), pp. e2201304119, 2022.

[R16] Cleeremans et al., Consciousness matters: Phenomenal experience has functional value. Neuroscience of consciousness. 2022, pp. niac007, 2022.

[R17] Richards et al., A deep learning framework for neuroscience. Nature neuroscience, 22, pp. 1761-1770, 2019.

[R18] Grover et al., Differential mechanisms underlie trace and delay conditioning in Drosophila. Nature, 603, pp. 302-308, 2022.

[R19] Chavlis et al., Drawing inspiration from biological dendrites to empower artificial neural networks. Current opinion in neurobiology. 70, pp. 1-10, 2021.

Thanks for your insightful and constructive feedback on our manuscript. Your positive reception of our visual perception biological motivation to guide network design and recognition of our method and experiments are both encouraging and deeply appreciated. We will make some clarifications in accordance with your suggestions.

Q1: The introduction reads like a related work. It will be great to make more comparison between this work and previous work. Answering what is wrong with previous works? and where the efficiency and performance gain come from in this paper?

A1: Thanks for your valuable suggestion. Technology-wise, previous methods mostly focus on solving a certain aspect of the Temporal Video Grounding (TVG) task, such as representation learning of language and video self-modality [R1, R2], multimodal fusion [R3, R4], cross-modal interaction [R5, R6], proposal candidate generation [R7, R8], proposal-based cross-modal matching [R9, R10], target moment boundary regression [R11, R12], and so on. Our work actually proposes a new paradigm to establish a two-stage unified static and dynamic semantic complementary new architecture. Its unique characteristics are that

• Processing multimodal signals in a unified ResMLP network, while many previous works are independently encoding the language modality and video modality [R1-R7, R9-R12].
• After the implementation of the above static ResMLP, we introduce a Gaussian nonlinear filtering method to learn the semantic associations within the video and combine it with the back-end proposal generation to promote cross-modal semantic alignment, further developing the proposal-based TVG method.

Our work is inspired by visual perception biology. This unified static and dynamic two-stage architecture performs excellent joint learning of language and video, which achieves state-of-the-art performance on NLVG and SLVG tasks. The ablation experiments in Table 4 of our manuscript demonstrate the effectiveness of each proposed component of this work. We will improve our manuscript based on your constructive suggestions.

Efficiency and performance: Technology-wise, the inference efficiency of our model comes from a streamlined architecture design, which allows for parallel inference of video and multiple queries in a unified ResMLP, saving time and expenses. Our promising model performance mainly comes from our model's comprehensive information interaction with query and video (ResMLP, static network), as well as our detailed inference of video content (DTFNet, dynamic graph filtering network), which was confirmed in the first ablation experiment in Table 4 of the manuscript.

References:

[R1] Xia et al., Video-guided curriculum learning for spoken video grounding. ACM MM, pp. 5191–5200, 2022.

[R2] Rodriguez et al., Memory-efficient temporal moment localization in long videos. ECAL, pp. 1901–1916, 2023.

[R3] Li et al., Proposal-free video grounding with contextual pyramid network. AAAI, pp, 1902–1910, 2021.

[R4] Liu et al., Exploring optical-flow-guided motion and detection-based appearance for temporal sentence grounding. IEEE TMM, 2023.

[R5] Liu et al., Jointly cross-and self-modal graph attention network for query-based moment localization. ACM MM, pp. 4070–4078, 2020.

[R6] Sun et al., Video moment retrieval via comprehensive relation-aware network. IEEE TCSVT, 2023.

[R7] Zhang et al., Learning 2d temporal adjacent networks for moment localization with natural language. AAAI, pp. 12870–12877, 2020.

[R8] Zhang et al., Multi-stage aggregated transformer network for temporal language localization in videos. CVPR, pp. 12669–12678, 2021.

[R9] Gao et al., Fast video moment retrieval. ICCV, pp. 1523–1532, 2021.

[R10] Zheng et al., Phrase-level temporal relationship mining for temporal sentence localization. AAAI, pp. 3669–3677, 2023.

[R11] Zhang et al., Natural language video localization: A revisit in span-based question answering framework. IEEE TPAMI, 44(8), pp. 4252–4266, 2021.

[R12] Liu et al., Skimming, locating, then perusing: A human-like framework for natural language video localization. ACM MM, pp. 4536–4545, 2022.

Thanks all authors for their effort in the rebuttal.

The rebuttal address most of my concerns, so I keep my rating suggesting acceptance of this work.

Thanks

Q2: In the ablation study, individually employing the static network and DTFNet yields significant improvements compared to the baseline. However, the combination of both modules does not exhibit a notably large improvement compared to using either single module. Is there a specific explanation for this phenomenon? The authors should provide more details about the baseline models.

A2: Thank you for raising the concerns about the main network modules. In our manuscript, we propose a method that starts with static global review queries and video content (ResMLP), and then dynamically filters video content to extract important information (DTFNet). On the one hand, observing the experimental results of our static module alone (ResMLP) compared with existing works (in Tables 1 and 4 of our manuscript), previous works have overlooked and performed insufficiently in extracting information from global static reviews. On the other hand, even if the static review information is missed and only basic cross-modal information is obtained, we achieve good video grounding results using the dynamic module (DTFNet) alone. Effective information filtering method (DTFNet) can still extract useful information.

For convenience, we restate Table 4 of our manuscript in the following Table R2. Relatively speaking, the large performance improvement of a single module through the respective exploration of multimodal information has reached saturation (for NLVG task, the R@1, IoU@0.3 of ResMLP and DTFNet are 73.57 and 74.56, respectively.). In this case, the R@1, IoU@0.3 of a single module increases from 73.57/74.56 to the final static-dynamic combination of 75.85, which is not a small improvement. That's why the effect of combining ResMLP and DTFNet is not as much improved as employing them alone. This further demonstrates the effectiveness of our static and dynamic modules.

Table R2: Ablation studies on the static and dynamic modules on the ActivityNet Captions and ActivityNet Speech datasets for both NLVG and SLVG.

About the baseline model, we belong to the scope of the 2D proposal-based method. Specifically, we adopt the same backend decoding method as MMN (including the 2D proposal generation module). For training losses, we use the cross entropy loss between the predicted 2D proposal score map and groundtruth which is consistent with 2D-TAN, and add the contrastive loss proposed in MMN as an auxiliary constraint. That is to say, our method is based on the typical 2D temporal map's backend decoding architecture (2D-TAN and MMN), and by incorporating a static-dynamic module design as the fore-end implementation. Our work further develops the 2D proposal-based method in this field.

We hope our response addresses your concerns. If there are any further questions, please let us know. Thank you for your time and valuable feedback.

Thanks for your recognition of our proposed Dynamic Temporal Filter Network, and your insightful reply is very helpful. We follow your advice and add the new experimental results about comparing multi-query input and single-query input.

Q1: In ResMLP, visual features and multiple query features are concatenated and fed into the network, largely leveraging the information leakage between different queries (because the features incorporate more accurate textual information that describes the video content). If each query is individually input into the network, would this method exhibit a significant performance degradation?

A1: Thank you for your valuable feedback. Here, we attempt to clarify our conclusion from the following two aspects. On one hand, regardless multi-query or single-query modes, the role of Multilayer perceptron with residual design (ResMLP) is designed to capture the associations between query and video, simulating the brain's processing of multi-source information in the early stage of visual perception for the video. On the other hand, yes, we agree with your opinion that multiple queries can indeed provide more semantics to the video and significantly fill the gap between vision-language modalities, aiding in understanding video content.
To eliminate your concern, we have supplemented the relevant experiments and found two facts in Table R1:

• Our model still performs best in single-query input mode, compared to other single query methods. For example, our R@1, IoU@0.7 is 32.25, exceeding the current SOTA methods.

• At present, there are some multi-query methods, but their advantages over the single-query methods are not obvious. How to better utilize multiple queries to assist in video grounding is also a challenge. Our method has advantages in handling both single-query and multi-query input modes.

Table R1: Model size vs. Infer. Speed vs. Performance comparison in different query input modes.