Action Dubber: Timing Audible Actions via Inflectional Flow

1. Formatting Issues

We sincerely appreciate your valuable feedback. We will revise the figure citations, adjust the formatting of Figure 2, correct the subscripts in the equations, and improve the overall layout of the paper. Additionally, we will fix the typographical error in line 376 to ensure clarity and consistency throughout the manuscript.

2. Concerns on Dataset Selection

We selected Kinetics and UCF101 because they serve as standard benchmarks for action counting, which is a core use case of our task. These datasets include a wide range of repetitive, sound-associated actions such as “clapping”, “jumping jacks”, “typing”, and “punching”, which are well aligned with the objectives of audible action localization.

In contrast, datasets like SSv2 and HMDB51 emphasize abstract or context-dependent interactions that are less relevant to our goal. For example, SSv2 includes many object-centric and non-audible actions such as “pretending to open something” or “moving something from left to right”, which lack consistent visual-sound correspondence. Similarly, HMDB51 features a high proportion of semantically complex or non-repetitive actions such as “smiling” or “listening to music”, which are not suitable for sound grounding or counting tasks.

Therefore, our dataset selection is driven by the need to support visually grounded audible action detection, and we believe it is well justified for the specific focus of our work.

1. Ambiguity on Timing Boundaries of Actions

Temporal boundary ambiguity is indeed a well-known and widely acknowledged challenge in general action understanding tasks. Precisely defining the start and end of an action, particularly for semantically complex or continuous activities such as "sit down" or "throw the ball", is inherently difficult due to visual similarity across adjacent frames and the subjective nature of temporal segmentation.

However, unlike general action semantic understanding, our task, Audible Action Temporal Localization, is significantly more constrained and objective. Rather than requiring full temporal segmentation of action intervals, we focus on identifying the specific frames in which a sound is produced by an action. These "sounding frames" are typically easier to annotate consistently, as the accompanying audio provides a clear and reliable temporal anchor. We will clarify this annotation protocol and emphasize the inherent simplicity and objectivity of our task in the revised Dataset Annotation section.

2. Concerns on Dataset Categories

Our task is fundamentally different from the original objectives of Kinetics and UCF101. While those datasets are designed for action classification across a broad range of semantic categories, our focus is category-agnostic, targeting the detection of action patterns rather than semantic understanding. As a result, the Audible623 dataset does not include category annotations but instead provides precise temporal labels for frames where audible events occur. This design is well aligned with the goals of dubbing and audio-visual synchronization, which rely on accurate temporal localization rather than action recognition.

Thanks to the design of our framework, our method demonstrates strong generalization ability. As shown in Table 3, it performs well on datasets with different action categories, such as UCFRep and CountixAV. This confirms that our approach is not constrained by specific categories and can generalize effectively to unseen actions. For instance, our method was successfully applied to animated content in the demo without any additional training.

We will clarify these distinctions and elaborate on the rationale behind our dataset design in the revised manuscript.

3. Audible Action Definition

While audible actions are a subset of general actions, they serve a distinct and crucial role in applications like sound dubbing and repetitive counting. Unlike general action recognition, which often relies on high-level and potentially ambiguous semantic categories (e.g., "playing sports," "interacting"), audible actions are grounded in concrete, low-level visual cues that correlate directly with sound production (e.g., clapping, stomping, tapping). This distinction makes it category-agnostic and more robust to domain shifts. This targeted focus enables reliable grounding of actions with sound, which is essential for many practical tasks.

4. Reuse of TAL Datasets

As mentioned in Section 3.2, existing Temporal Action Localization (TAL) datasets are not directly applicable to our task. The keyframe-level labels required for Audible Action Temporal Localization cannot be reliably derived from the start and end frames of action segments in these datasets. In many cases, the annotations are relatively coarse and do not separate repeated instances of the same action. For example, a kicking sequence with multiple discrete kicks is often labeled as a single continuous segment, whereas our task focuses on detecting each individual audible event.

Additionally, TAL datasets include many inaudible actions that fall outside the scope of our task. Reusing these datasets would require significant re-annotation and filtering to align with our objective. For these reasons, we chose to construct a dedicated benchmark with precise frame-level annotations that better support the goals of audible action localization and audio-visual synchronization.

1. Distinction Between Our Method and TAL Methods, and Applicability to TAL

We respectfully clarify that our method targets a task that is fundamentally different from traditional TAL. While TAL focuses on localizing the full temporal extent of actions, our method is designed to identify precise keyframes where audible events occur. In this sense, TAL addresses event-level localization, whereas our approach focuses on frame-level detection tied to sound-producing moments.

This distinction is important, as audible frames often represent brief, high-impact instants that are not captured by coarse action intervals. As shown in the appendix, our method also performs well on non-audible actions, demonstrating its ability to capture subtle temporal transitions. We believe this fine-grained capability may provide useful insights for improving temporal precision in future TAL work.

2. Generalization Beyond the Training Dataset

Please refer to the reply of Reviewer umcX Q2.

3. Ambiguity of Audible Action && Multiple Audio Sources

While the exact decibel level may vary, our task focuses on visually observable actions that are intentionally associated with sound production, consistent with how humans infer sound-related events from visual cues. To minimize subjectivity, we annotate dominant and intentional audible actions during training, following standard practice in broader action recognition tasks where annotators focus on salient or representative instances of an action (e.g., annotating the main swing in "golf swing" rather than every subtle motion).

For testing, our model responds to motion patterns indicative of sound, regardless of their intensity. For instance, in the case of basketball dribbling, both soft and forceful dribbles share similar motion dynamics, and our model is designed to detect both.

We also clarify that Fig. 1 is purely for illustration purposes; all samples used in training and evaluation come from real-world, in-the-wild videos with natural variations in appearance and motion.

We will revise the manuscript to better explain our annotation criteria and the illustrative role of Fig. 1.

4. Multiple Audio Sources

In scenarios involving multiple audio sources, our method is designed to detect all sound-producing actions present in the video. However, distinguishing between individual sound sources is beyond the current scope of our work. Our primary objective is to accurately identify the timing of audible actions, regardless of their specific source.

5. Quantitative Comparisons with TAL methods

We further evaluated BMN on the Audible623 dataset using a two-stream network that incorporates both video and optical flow features. This allows for a fair comparison with TAL methods that also leverage motion information. Since the source code for Zhang et al. and Liu et al. is not publicly available, we have contacted the authors and will include further discussion in the revised manuscript based on their responses.

6. Concerns on Computational Complexity

Thanks to our lightweight network design, the computational cost of incorporating additional optical flow and inflectional flow features remains manageable. We evaluated the processing time of various methods on the same set of videos. While the inclusion of optical flow does introduce some additional overhead, our method maintains a favorable trade-off and still outperforms most competing approaches in both accuracy and efficiency.

7. Dataset Selection and Annotation

Annotators were instructed to review each video individually and select visually observable audible actions, excluding videos that either lacked audible actions or featured occluded actions. For high-frequency actions such as drumming, which are particularly challenging to annotate precisely, we adopt a frame-level labeling strategy, marking only the frames where the drumstick makes contact with the drum.

As noted in Sec. 3.2, we do not apply category-based filtering during annotation. Instead, the first round of filtering involves manually reviewing each video to determine whether it contains visually identifiable audible actions.

1. User Study

As suggested, we conducted a user study comparing five approaches: TriDet, TransRAC, X3D, Hiera, and our proposed method. We curated a set of eight videos and enlisted 30 participants for the study. Each video was dubbed using sound aligned to the audible action locations detected by each method. Participants were then asked to assess the audio-visual synchronization quality and select the top two methods they perceived as most accurate for each video. The results show that our method was most frequently selected as the best, indicating a clear preference and superior perceived synchronization performance.

2. Concerns on Short Duration & Application Scenarios

We respectfully disagree with this concern, as it appears to stem from a misunderstanding of both the dataset characteristics and our method’s design. The video duration and frame rate in the Audible623 dataset, as well as in other benchmarks used in our study, are consistent with established standards in the action counting and localization community. These datasets are widely adopted to ensure fair comparisons and reproducibility. More importantly, our method operates at the frame level, making it inherently agnostic to the overall video length. This design choice allows the method to scale effectively to longer and higher-framerate videos without degradation in localization performance. As such, our approach remains robust and applicable to real-world scenarios, including platforms like YouTube and professional media content.

3. Methodological Ambiguity in Motion Feature Extraction

We respectfully believe this concern may stem from a misunderstanding of motion feature roles. Depth and disparity capture static spatial structure but do not reflect temporal dynamics. For example, counting the number of times a person jumps depends on detecting repeated vertical motion—something depth alone cannot convey. In contrast, optical flow reflects dynamic changes across frames and is widely recognized as a reliable signal for temporal modeling.

While depth can theoretically be used to extend 2D optical flow to 3D, such usage is rare in practice, as action scenarios that truly benefit from 3D motion modeling are uncommon and often domain-specific. Moreover, incorporating depth adds computational overhead without clear gains for typical action tasks.

We chose Xu et al.’s network for its solid empirical performance and a good trade-off between accuracy and efficiency. While alternatives like FlowNet or PWC-Net are well-known, we did not observe significant advantages for our task. We will clarify this rationale in the revised manuscript.

4. Optical Flow Method Selection

Compared to traditional methods such as DisFlow, modern neural network-based approaches offer significantly improved robustness in challenging real-world scenarios involving occlusion, noise, or complex motion, and they perform consistently well across varying resolutions. Among these, GMFlow demonstrates state-of-the-art accuracy while maintaining inference times comparable to established methods like PWC-Net [1, 2]. Additionally, GMFlow offers a more streamlined and efficient framework for computing bidirectional optical flow, which is beneficial for our application. Based on these advantages and after thorough consideration, we selected GMFlow for optical flow extraction in our pipeline. [1] Xu H, Zhang J, Cai J, et al. Gmflow: Learning optical flow via global matching, CVPR, 2022. [2] Teed Z, Deng J. Raft: Recurrent all-pairs field transforms for optical flow, ECCV, 2020.

Note to Reviewer

We appreciate the reviewer’s time and effort in evaluating our work. While we find that some of the comments may stem from misunderstandings of our method and its design choices, we have addressed each point in detail and clarified the rationale behind our decisions. We believe these clarifications resolve the concerns raised and further highlight the strengths and contributions of our approach. We respectfully ask the reviewer to reconsider the insights and potential impact of our work in light of these responses.