Learning Semantic-Enhanced Dual Temporal Adjacent Maps for Video Moment Retrieval

$**Response to question 1:**$ Thanks for your valuable suggestion. We will realign and reformate tables on pages 6 and 7, We will proceed to realign and reformat the tables located on pages 6 and 7, incorporating your feedback to enhance their overall appearance.

$**Response to question 2:**$ Thanks for your insightful suggestion. We will adhere to your recommendation and consistently use the expression "video moment retrieval" throughout our work.

$**Response to question 3:**$ Thanks for your insightful suggestion. The appearance branch is tasked with modeling visual appearance information, while the semantic branch is designed to distinguish instances with similar appearances by encoding semantic cues. To accomplish this goal, we incorporate predictions from the appearance branch into the semantic branch through distillation, aiding the DTAM in distinguishing features with identical visual content but differing semantics. This is achieved using a well-crafted semantic-aware contrastive loss. Besides, we explore various distillation strategies, including the use of dynamic thresholds, fixed thresholds, and a pre-trained appearance branch. Detailed experimental results showcasing the effectiveness of our approach are presented in Figure 3(d).

As you pointed out, this is indeed our subjective interpretation of the original version. Therefore, in addition to investigating various distillation methods for validation, I have also crafted visualizations of some complex examples to clarify the underlying idea. Notably, we have incorporated a visual heat map as your insightful suggestion. However, please note that due to the constraints of the rebuttal stage, we cannot upload images in the modified version. Thanks very much again.

$**Response to question 1:**$ Thanks for your valuable suggestion. Given that our dual structure processes the input in a parallel manner, it does not impose any significant additional computational burden. Furthermore, we have compared the time overhead of various model variants in Table 4(d). Notably, the complete DATM incurs only a negligible increase in time compared to 2D-TAN, while achieving a substantial improvement in retrieval accuracy. Additionally, during the training phase, the dual structure (APB + SEB) takes approximately 1.6 times longer than the variant that solely utilizes the appearance branch (APB). However, during inference, the dual structure incurs negligible additional overhead.

1. Thanks for your comments. It appears that some misunderstandings have arisen. Equation (7) utilizes the feature embedding derived from the semantic branch, as opposed to the appearance branch, as depicted in Figure 2. Additionally, the algorithm now incorporates knowledge distillation by processing the output confidence of branch A (representing the acquired knowledge) to ascertain high-confidence intervals for both positive and negative instances.
2. In the original paper, the proposed moment-aware mechanism is used in both the appearance and semantic branches, as described in L266 and Figure 2.
3. Thanks for your comments. The primary novelty of our paper does not lie in contrastive learning. Instead, we identified that the integration of semantic and appearance features in the video grounding task leads to suboptimal performance. In response, we introduced a framework that decouples semantics and appearance, and leverages knowledge distillation to transfer the learned knowledge from the appearance branch to the semantic branch.
4. In the original paper, we have provided explanations for the symbols APB, MAM, and SEB in Table 4 for clarity.
5. Thanks for your comments. We have compared the proposed DTAM with SnAG in Table 2 for the ActivityNet-Captions Dataset using C3D features.

$*Explaination*$ [0]: Temporal Adjacent Map (TAM), proposed in AAAI 2019, is an efficient paradigm for processing cross-modal video grounding. In this paper, we follow this paradigm and develop a dual TAM to decouple semantic and appearance features, which is our motivation that interested video clips have similar visual appearance but different semantics. For this reason, the goal of this paper is to explicitly encode semantic and appearance information in a decoupled manner, facilitating differentiation between instances with similar appearance but different semantics. Besides, contrastive learning is only a tool utilized to distinguish various samples that guarantee semantic consistency with a knowledge distillation from the appearance branch into the semantic branch. Besides, we also proposed a novel moment-aware mechanism to augment vanilla temporal adjacent maps so that they can perceive the importance of each moment.

$*Explaination*$ [1]: Thanks for your comments. Although the features extracted by LLM have stronger expressive power, feature optimization is not our focus. The core issue of this paper is that interested video clips have similar visual appearance but different semantics, so we proposed a novel decoupled temporal adjacent map framework to address it. Furthermore, currently popular methods basically use C3D or I3D features. To this end, for a fair comparison, I also used them to verify the effectiveness of the model. Nevertheless, we will also explore algorithm paradigms based on large model features in subsequent work based on the reviewers' valuable suggestions.

$*Explaination*$ [2]: Thanks for your insightful suggestion. The literature [1] elucidates and validates the phenomenon that "previous video moment retrieval methods overlook the flexibility and complexity of moment description, leading to an inability to encode the temporal dependencies between moments that are pivotal for moment retrieval." In the revised version, we will incorporate citations in accordance with your valuable feedback. Moreover, certain complex queries encompass similar content but possess greater semantic differences, as exemplified in Figure 1(c). To validate this, we conducted ablation studies presented in Table 4, exploring the impact on semantic modeling. Furthermore, we provide visual examples in Figure 4, comparing different variants of the diversion model. These results demonstrate that the proposed Dual Temporal Adjacent Map (DTAM) can effectively handle such complex queries.

[1] S. Zhang, H. Peng, J. Fu, and J. Luo. "Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language" in AAAI 2020.

$*Explaination*$ [3]: Thanks for your comments. The motivation behind our work stems from the observation that interesting video clips often exhibit similar visual appearances yet possess distinct semantics. To address this, we explicitly encode appearance and semantics in a decoupled manner, enabling us to differentiate between instances that share similar visual appearances but possess different semantics. Specifically, the appearance branch is tasked with modeling visual appearance information, while the semantic branch is designed to distinguish instances with similar appearances by encoding semantic cues. To accomplish this goal, we incorporate predictions from the appearance branch into the semantic branch through distillation, aiding the Dual Temporal Adjacent Map (DTAM) in distinguishing features with identical visual content but differing semantics. This is achieved using a well-crafted semantic-aware contrastive loss. Additionally, we explore various distillation strategies, including the use of dynamic thresholds, fixed thresholds, and a pre-trained appearance branch. Detailed experimental results showcasing the effectiveness of our approach are presented in Figure 3(d).

$*Explaination*$[4]: Thanks for your valuable suggestions. The underlying motivation of [1] stems from the fact that current models learn dataset-specific knowledge, largely influenced by the distribution of the dataset. Conversely, our motivation is to disentangle visual appearance from semantic information. Both works share a common ground in mining the pivotal factors that impact retrieval performance from a data-centric perspective. However, while [1] endeavors to learn the biases between different data, our method identifies visually similar examples within the data that possess vastly different semantics. To address this, we propose a novel semantic-enhanced dual temporal adjacent map, which models temporal dependencies between moments in a manner that decouples appearance from semantics.