PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling

Thank you for your positive feedback and constructive comments. In regard to the weaknesses and questions raised, we provide our point-to-point responses below:

W1 and Q1: Online assumption and train-test mismatch: How does the model maintain online consistency when it uses future-supervised signals during training?

A1:

Our framework strictly adheres to the online setting during inference. The use of future labels is a deliberate design choice confined to the training phase, and it does not create a train-test mismatch.

1. Strict Adherence to Online Assumptions and Consistency During Reasoning: As detailed in Sec 3.6, at any given time step $T$, the model's prediction relies solely on historical and current features available up to $T$. No future ground truth or features are accessible, ensuring true online consistency at test time.
2. A Deliberate and Principled Training Strategy: The use of future labels during training is an explicitly designed strategy to equip the model with the ability to anticipate the future, which is the core of our Predictive Future Modeling (PreFM) framework. Our goal is to have the model internalize the predictive capability.

In summary, this design choice does not introduce a train-test mismatch and serves as a necessary supervision mechanism. It effectively bridges the gap by teaching the model to generate its own useful context at inference time, addressing the inherent challenge of missing future information in an online setting.

W2 and Q2: The quality and usefulness of pseudo-future features: Can you provide a quantitative analysis of the pseudo-future features’ quality?

A2:

Here we provide the analysis of the quality and usefulness of pseudo-future features:

1. Quantitative Analysis of the Pseudo-Future's Quality

We evaluate them from two perspectives, their semantic similarity to ground-truth event features, and their effectiveness in predicting future events:

• Top-k Similarity Accuracy: We measure if a generated feature vector at a relative future time step (T+1~T+5) is the Top-1 or Top-5 closest match to its corresponding ground-truth class feature embedding, among all 100 classes in UnAV-100.
• Future Prediction F1-Score: We also report the standard segment-level F1-score for predictions made for future time steps.

The results are presented in the table below:

This results provides two key insights:

• First, the pseudo-future features are remarkably realistic. The Top-5 Similarity Accuracy of over 94% demonstrates that the correct event feature is almost always ranked among the top candidates.
• Second, these high-quality features enable strong future prediction performance, as evidenced by the solid F1-scores.

2. Quantitative Analysis of the Pseudo-Future's Usefulness for Inference

We now demonstrate their usefulness for the main task—improving prediction at the current time $T$. Our ablation study on the Predictive Future ($PF$) mechanism in Tab.3(b) in our manuscript directly addresses this. For the reviewer's convenience, we highlight the key comparison below:

The results clearly show that incorporating the $PF$ mechanism, which generates and utilizes these high-quality future features, leads to a significant performance improvement (e.g., +1.6 in Avg F1 Score). This quantitatively proves that the pseudo-future context is actually useful during inference for enhancing the model's understanding of the present moment.

In conclusion, our comprehensive analyses confirm that the pseudo-future features are both realistic in their semantic representation and highly useful for boosting the model's final online parsing performance.

W3, Q3 and Q4: The choice of teacher model: How sensitive is the model to the choice of teacher in MRR? Have you tried other options such as CLIP or AudioCLIP? How much does the downstream performance vary?

A3:

PreFM's performance is robust and not overly sensitive to the choice of the teacher model.

We include this analysis in our original submission in Tab.5 in the appendix. For the reviewer's convenience, we present the key results below:

As the table shows, our PreFM framework demonstrates robust and stable performance across different powerful, pre-trained teacher models. The variation in the final Event-Level Avg F1-score is minor (ranging from 45.9 to 46.3).

This indicates that while the choice of teacher has a slight impact, our MRR approach is not overly sensitive to a specific teacher and generalizes well to other strong multimodal teacher models.

Regarding the reviewer's suggestion to try CLIP: we specifically chose teacher models like AudioClip, ImageBind, and OnePeace because they provide a unified embedding space that includes the audio modality. Standard CLIP models are primarily designed for vision-language tasks and do not process audio, making it unsuitable for providing modality-agnostic supervision for our audio-visual parsing task.

W4 and Q5: Evaluation with clip concatenation: How does PreFM perform in a real streaming setting with long, continuous videos?

A4:

Our primary evaluation is conducted on the UnAV-100 dataset, which consists of long, untrimmed, in-the-wild videos and directly addresses this concern. The LLP dataset acts as a complementary benchmark.

1. SOTA Performance on a Real-World Continuous Video Dataset (UnAV-100)

The UnAV-100 dataset contains 10,790 videos of varying length and it is specifically designed for dense event localization in long, untrimmed, real-world videos. Our SOTA results on UnAV-100, detailed in Tab.1 of our manuscript, provide strong evidence of PreFM's effectiveness. For convenience, we show the key results below and include the latency results:

As the results show, PreFM significantly outperforms all SOTA methods on this challenging dataset (e.g., achieving a +9.3 gain in event-level Avg F1-score over the next best method) while using significantly fewer parameters and lower computational complexity. This demonstrates its strong performance and robustness in realistic, continuous video streams.

2. The Role of the LLP Dataset as a Complementary Benchmark

We designe the modified LLP dataset not as a substitute for real-world data, but as a controlled, complementary benchmark to analyze model capabilities under two distinct and challenging online scenarios:

• Random Concatenation: This setup simulates the rapid and unpredictable scene changes common in online streaming content, testing the model's adaptability.
• Consistent Concatenation: This setup simulates longer, continuous event occurrences, testing the model's ability for consistent understanding and discrimination over an extended context.

This approach allows us to perform a more nuanced assessment of PreFM's robustness, complementing the evaluation on the in-the-wild UnAV-100 dataset.

3. Qualitative Visualization on Continuous Streams

We provide extensive qualitative results in Fig.5 (main paper) and Fig.7 & 8 (appendix). These figures visualize PreFM's superior ability to generate accurate and temporally coherent event boundaries on long video timelines from both datasets.

In summary, PreFM's effectiveness in realistic streaming settings has been thoroughly validated through its SOTA quantitative performance on the large-scale UnAV-100 dataset, supported by targeted analyses on our complementary LLP benchmark and extensive qualitative evidence.

Thank you for recognizing our work and for your constructive comment. In regard to the weaknesses and questions raised, we provide our point-to-point responses below:

W1: Contradictory statements? Lines 256–257 suggest that the PF module significantly boosts performance, whereas Lines 268–269 state that PF yields negligible gains. It would be helpful to clarify these descriptions to avoid confusion.

A1:

Thank you for pointing out the potentially confusing phrasing in Lines 268–269.

In Lines 268–269, we wish to clarify that the performance benefits derived from our pseudo-future mechanism are attributable to the effective learning guided by these targeted future-oriented losses, rather than merely an increase in model capacity. The phrase “negligible gains” refers to the variant in which we merely add the $PF$ architecture’s parameters without applying any future supervision (Row 2 in Tab.3(b) of our manuscript).

We will revise the manuscript to express these points more clearly.

W2: Failure cases. While Figure 5 and the supplementary materials present visualizations, the paper would benefit from a more detailed discussion of the failure cases—when and why PreFM might not perform well.

A2:

Thank you for this excellent suggestion.

We conduct a more detailed analysis on the On-AVEL test set to identify specific failure cases. While the rebuttal format prevents us from including new figures and PDF, we present our quantitative findings and analysis below:

1. Confusion Between Similar Events: We analyze the events with the lowest performance and their most common confusions.

This results reveal that PreFM struggles to distinguish between events that are semantically or acoustically similar. We hypothesize this is because our current framework does not explicitly incorporate a contrastive learning design to better separate the representations of events originating from similar audio-visual sources.

2. Performance in Dense Scenes: We analyze the impact of event density (the number of event classes within a video) on event-level performance.

These results show that PreFM's performance degrade in complex videos containing a large number of distinct event classes. This suggests that while our future modeling is effective, its benefits are less pronounced in scenarios with very rapid scene changes and drastic context shifts.

We will add this analysis and qualitative visualizations of these failure cases to the appendix of our revised manuscript.

W3.1: The On-AVVP subtask requires identifying audio-only, visual-only, and audio-visual events. Does the proposed method predict each type of event separately or simultaneously?

A3.1:

Our method predicts all event types simultaneously.

We follow the standard paradigm for Audio-Visual Video Parsing (AVVP) tasks by treating it as a multi-label classification problem over a unified class set. Specifically, our model's final classifier produces a prediction vector of size $C_a+C_v$ to predict audio-only, visual-only and audio-visual event (an event is considered "audio-visual" when its auditory and visual components are both present) results, where $C_a$ and $C_v$ represent the number of audio and visual event classes, respectively.

W3.2: Could the softmax function be used to normalize the weight w(t) across frames in Eq. 9?

A3.2:

The softmax function can be used to normalize the weight $w(t)$, but we do not recommend it for our specific purpose. Our reasoning is as follows:

• Our current normalization approach, after obtaining the Gaussian distribution weights, can be expressed as: $w(t) = \frac{g(t)}{\sum_{i} g(t_i)}$. This direct normalization preserves the proportional relationship of the Gaussian distribution, creating a smooth focus.
• In contrast, the softmax function is expressed as: $w(t) = \frac{e^{g(t)}}{\sum_{i} e^{g(t_i)}}$. The exponential mapping in softmax would exaggerate the difference between high and low weight values. This would cause the model to overly ignore the edge information (i.e., the earliest historical context and the most distant future cues), which is not what we intend.

Furthermore, we conduct an ablation study on the On-AVEL task where we replace our normalization method with softmax:

As the results show, model performance slightly decreases after switching to softmax. This empirically supports our choice of using direct normalization.

W3.3: What are the training and inference time costs for the proposed method?

A3.3:

The total training time required to converge, inference processing time costs (latency) and model performance of our method are shown below. We use the official code to reproduce the second-best baseline methods CCNet on the On-AVEL and MM-CSE on the On-AVVP tasks to provide further context:

As the results demonstrate, our PreFM framework offers substantial improvements in training and inference time costs.

W4: Four minor issues.

A4:

Thank you very much for your careful reading and these valuable suggestions to improve our paper's clarity and completeness. We will address all of them in the revised manuscript:

• Module Name Inconsistency: We will revise the module name in Line 157 from "current enhancement" to "current refinement" to ensure it is consistent with Fig.2.
• Clarity of Fig.2: We will revise Fig.2 to visually connect the "Focal Temporal Prioritization" module to the final loss computation. This will better illustrate its role in weighting the training objective.
• Training/Inference Visualizations: As suggested, we will add a new figure to the appendix that visualizes the training and inference processes as described in Sec 3.6.
• Additional Citations: Thank you for pointing out these highly relevant papers. We will add and discuss the suggested works (CPSP and UWAV), along with other recent related papers, in our Related Work section to provide a more comprehensive background.

We appreciate this detailed feedback, which will certainly help us improve the quality of our paper.

Thank you for your positive feedback and valuable suggestions. In regard to the weaknesses and questions raised, we provide our point-to-point responses below:

W1 and Q1: Prediction Mechanism: How exactly does the model predict future cues? What is the temporal horizon? Can you provide more technical details about how future audio-visual cues are predicted? What is the prediction horizon and how do you handle prediction uncertainty?

A1:

Here we provide a clearer explanation of the predictive future modeling mechanism, temporal horizon and uncertainty handling:

1. Prediction Mechanism:

Initial Future Modeling: We first use a set of learnable query tokens ($Q_a , Q_v$) for future time steps. These queries attend to the current audio-visual features via an attention mechanism to generate an initial, context-aware pseudo-future ($\tilde{F}_f^a, \tilde{F}_f^v$ in Eq.3).

Refinement via TMCF: This initial, and potentially noisy, pseudo-future is then refined within our Temporal-Modality Cross Fusion (TMCF) stage. Using unified hybrid attention (UHA) block, this step produces an augmented and more reliable future representation ($\hat{F}^a_f,\hat{F}^v_f$ in Eq.4 and Fig.3).

Supervision: The entire model is learned end-to-end, supervised by losses on these future predictions, specifically the classification loss $\mathcal{L}\_f$ and the modality-agnostic representation loss $\mathcal{L}_{mrr}$. This process effectively teaches the model to map the current context to reliable future events.

2. Temporal Horizon

The prediction horizon (the length of the pseudo-future window) is a hyperparameter $L_f$ and our default setting is $L_f=5$ seconds. This choice is empirically validated by our ablation study in Tab.4 (in the appendix), which shows that $L_f=5$ achieves the best balance between providing sufficient future context and avoiding excessive predictive noise.

3. Handling Prediction Uncertainty

Our framework handles this inherent uncertainty through the Temporal-Modality Cross Fusion (TMCF) stage. TMCF refines the initial, potentially noisy pseudo-future by enforcing further cross-modal and cross-temporal interactions (as described in Lines 151-156 and Fig.3 in our manuscript), which can be broken down as follows:

• Self-Attention: Models the internal temporal structure and consistency within the predicted future sequence.
• Cross-Modal Attention: Ensures the predicted audio and visual future streams are coherent with each other.
• Cross-Temporal Attention: Grounds the uncertain future prediction in the more reliable current context, effectively calibrating and denoising it.

The effectiveness of this design is quantitatively demonstrated in Tab.3(c). The full TMCF achieves higher F1-scores on future predictions (e.g., 57.5 at T+1) compared to ablated versions (e.g., 55.4 with "Self" attention only), proving that our fusion mechanism successfully handles the uncertainty and improves the reliability of the pseudo-future context.

W2 and Q3: Claims about "real-time applicability" need quantitative support.: Can you provide the concrete real-time performance? What are the actual processing times and computational costs compared to baseline methods in practical streaming scenarios?

A2:

We conduct a detailed timing analysis of processing latency to further substantiate our real-time claims. For the reviewer's convenience, we present a comprehensive comparison below, including the performance, latency and other efficiency metrics in Tab.1 and Tab.2. All tests are conducted under a single RTX 3090 GPU.

Taking the On-AVEL task as a representative example, we observe that:

• Low Latency: PreFM has a latency of only 19.3 ms per inference. This is approximately 7 times faster than the next best performing method, CCNet, which has a latency of 133.3 ms.
• Superior Efficiency: This real-time performance is achieved while using fewer resources: +9.3 Avg F1-score with only 2.7% of the parameters and 0.6% of the FLOPs compared to CCNet.

In the revised manuscript, we will add these latency results and a concise discussion to fully substantiate our “real-time applicability” claims.

W3: Missing Ablation Studies: There's insufficient analysis of which components contribute most to the performance gains. The interplay between predictive multimodal future modeling, modality-agnostic representation, and focal temporal prioritization needs better investigation.

A3:

1. Individual Component Contributions

To better isolate which component contributes most, we conduct additional ablation studies by adding each component individually to our baseline model (which uses a context window of $L_c$ but no other improvements). The results are summarized below:

This analysis clearly shows that Pseudo Future Mechanism ($PF$) provides the most significant performance gain when applied individually, underscoring its central role in our framework. Other components like $w(t)$ and $\mathcal{L}_{mrr}$ also provide substantial improvements.

2. The Interplay Among Components

Our original ablation study in Tab.3(a) is designed to reveal the interplay between components. Based on that table and the new results above, we provide a more focused analysis of their synergy:

• Synergy between $PF$ and $\mathcal{L}\_{mrr}$: $PF$ alone improves the Avg F1-score by +1.6 and $\mathcal{L}\_{mrr}$ alone improves by +0.9. When combined $PF$ with $\mathcal{L}\_{mrr}$, it yields an improvement of +3.4 (row 5 in Tab.3(a) ). This strong synergistic effect suggests that while the $PF$ mechanism is the primary driver for performance by providing future context, the $\mathcal{L}_{mrr}$ acts as a powerful regularizer and guide. It enriches the pseudo-future predictions by forcing them to align with the robust, semantically-rich feature space of the teacher model, leading to a more effective and generalizable representation than either component could achieve alone.
• $w(t)$ as a Consistent Enhancer: Adding $w(t)$ to the $PF$ model (row 4 vs. row 3 in Tab.3(a) ) boosts the gain from +1.6 to +2.2. Similarly, adding it to a model with $PF$ and $\mathcal{L}_{mrr}$ (row 7 vs. row 5 in Tab.3(a) ) elevates the gain from +3.4 to +4.6. This consistent improvement highlights the unique value of $w(t)$: by applying a Gaussian-weighted focus to the loss, it compels the model to prioritize making a precise and robust prediction at the most critical moment—the present $T$—thereby sharpening the final online decision-making process.

Q2: Baseline Fairness: How were the baseline methods adapted for online processing? Are you comparing against methods specifically designed for online settings or offline methods retrofitted for streaming?

A2:

1. Justification: Establishing Baselines in a Clear Research Gap

We compare PreFM against SOTA offline methods that we adapt for the online setting. This is a deliberate and necessary choice due to a clear research gap in the field because:

• Existing Audio-Visual Event Parsing methods (like CCNet for On-AVEL, or MM-CSE for On-AVVP) are exclusively offline and designed to process entire videos at once, possessing no native online capabilities.
• Existing Online Video Understanding methods, conversely, are almost entirely unimodal (vision-only). They focus on tasks like action detection and are not equipped to handle the multimodal audio-visual parsing required by On-AVEP.

Given this landscape, the most rigorous and scientifically valid approach is to adapt the strongest SOTA methods from the relevant offline domain to create the first powerful set of baselines for this new online task.

2. Fair Adaptation Protocol for Online Processing

We detail the fair adaptation protocol in the appendix (Lines 637-647) in our manuscript. For convenience, we summarize the process here:

• For On-AVEL baselines, which require full-video inputs, we feed the model the available stream up to the current time $T$ and mask all future frames beyond $T$ with zero-padding.
• For On-AVVP baselines, which operate on fixed-length clips, we feed the model the most recent segment ending at $T$ (e.g., from $T-9$ to $T$).

This protocol guarantees that all methods, including our PreFM, operate under the exact same online constraints, creating a fair and valid comparison.

Dear Reviewer VEsL,

We greatly appreciate your time and consideration of our submission. As the author-reviewer discussion period is nearing its end, we kindly ask if you could review our responses to your comments at your earliest convenience.

This will allow us to address any further questions or concerns you may have before the discussion period ends. If our responses satisfactorily address your concerns, please let us know.

Thank you very much for your time and effort!

Sincerely,

The Authors of Submission #2457

Thank you for your positive feedback and constructive comments. In regard to the weaknesses and questions raised, we provide our point-to-point responses below:

W1: Despite the results, the methodical contribution is quite weak. At its core, the work simply teases out (k) representations using some "prototype" query vectors via attention, as a proxy for the supposed future.

A1:

Our core methodological contribution lies in proposing a novel framework, PreFM, which addresses the new and challenging task of Online Audio-Visual Event Parsing (On-AVEP) through the Predictive Future Modeling mechanism, not in using query vectors for future prediction.

The Predictive Future Modeling mechanism is not simply about "teasing out representations": Learnable queries are trained to actively anticipate plausible future cues based on the current context. This is followed by cross-modal and cross-temporal interactions to augment the initial, potentially noisy pseudo-future, which is then integrated back to enhance the current representations.

This simple yet effective framework achieves SOTA performance with high efficiency: It delivers a +9.3 Avg F1-score improvement while using only 2.7% of the parameters of the SOTA model.

W2: What motivates the specific sequence of UHA operations? Why were those particular inputs chosen? What happens if you simply apply another round of attention (as in the "Pseudo future" section)?

A2:

1. Justification for the TMCF Design

TMCF design is not arbitrary but is motivated by the core challenge of refining the noisy initial pseudo-future sequences. The key design principles are:

Parallel Operation in UHA: The UHA block processes multiple context sets in parallel (by summing their attention outputs as Eq.1 in our manuscript), not in a rigid sequence.

Logical Flow in TMCF: The whole TMCF process involves (a) Future Augmentation first, followed by (b) Current Refinement. This order is crucial: we must first refine the initial, potentially noisy pseudo-future to make them more reliable. Only then can this higher-quality future context be effectively integrated back to enhance the current representations.

Principled Input Selection: The inputs for the UHA block in TMCF are chosen to enable three critical interactions for robust future modeling (as described in Lines 151-156 and Fig.3 in our manuscript):

• Self-Attention: To model the internal temporal structure within the predicted future sequence.
• Cross-Modal Attention: To ensure the predicted audio and visual futures are coherent with each other.
• Cross-Temporal Attention: To ground the uncertain future prediction in the reliable, immediate past, thereby calibrating and denoising it.

This multi-faceted fusion is essential for generating a high-quality, reliable future context.

2. Contribution of TMCF vs. Another Round of Attention

To empirically validate that the gains come from our specific design rather than merely an increased parameter count, we conduct a comprehensive ablation study, expanding upon Tab.3(c). We compare our full TMCF against several variants: convert all modality fusion into temporal fusion (T only), convert all temporal fusion into modality fusion (M only) and convert all fusion layer into self-attention (Self), including the reviewer's suggestion of "simply applying another round of attention" (Simple Self).

The results show that simply adding a round of self-attention performs poorly, and our full TMCF outperforms the baseline by a significant margin (+1.7 Avg F1) with the same parameter count. This strongly demonstrates that the performance gain is attributable to the principled fusion mechanism of TMCF, not just the additional parameters.

W3: The paper does not clearly articulate its contributions. Can the authors explicitly state the contributions and novelties in the introduction?

A3:

Our main contributions are three-fold:

1. We introduce Online Audio-Visual Event Parsing (On-AVEP), a new paradigm for real-time multimodal understanding. To our knowledge, this is the first work to systematically address the challenge of parsing audio, visual, and audio-visual events from streaming video. We further establish that success in this paradigm requires two critical capabilities: (a) accurate online inference from limited context, and (b) real-time efficiency to balance performance with computational cost.
2. We propose the PreFM framework, a novel and efficient architecture for On-AVEP. PreFM's core innovations include: (a) Predictive Multimodal Future Modeling mechanism to overcome the critical problem of missing future context ; and (b) a combination of Modality-agnostic Robust Representation and Focal Temporal Prioritization to enhance model robustness and efficiency during training, providing an insightful approach to multimodal real-time video understanding.
3. We establish new SOTA performance with unprecedented efficiency. Extensive experiments on two public datasets show that PreFM drastically outperforms previous methods (e.g., +9.3 Avg F1-score on UnAV-100), while using a fraction of the computational resources (e.g., only 2.7% of the parameters of the next best model), validating it as a powerful and practical solution.

We thank the reviewer for this constructive suggestion, and we will add the summary to the end of our introduction in the revised manuscript.

W4: What were the bottlenecks in prior methods, and how does this work address them differently? Can the authors clarify how their parameter count compares to prior art, and justify the huge differences?

A4:

1. Bottlenecks in Prior Methods and Our Solutions

The bottlenecks of prior methods in an online setting are two-fold:

• Fundamental Offline Design: These models are designed to process the entire video sequence to leverage global context for video understanding, which is fundamentally infeasible in a real-time streaming scenario.
• Prohibitive Computational Cost: Their reliance on processing long sequences with heavy model architectures results in high latency and memory usage, failing to meet the demands of real-time applications.

Our PreFM framework is designed to address these bottlenecks:

• Natively Online Architecture: PreFM operates on a short, sliding window of past information, making it naturally suited for streaming data.
• Context Enrichment via Future Prediction: Instead of requiring global context, PreFM's core innovation, Predictive Future Modeling, explicitly infers the necessary future cues from the local window. This enriches the limited context for robust real-time parsing.
• Training for Efficiency: This is complemented by Modality-agnostic Robust Representation and Focal Temporal Prioritization, which boost inference accuracy while maintaining a lightweight training process.

2. Parameter Count and Justification

We calculate trainable parameters for all models using a consistent protocol (sum(p.numel() for p in model.parameters() if p.requires_grad)) for a fair and transparent comparison. We show the parameter results in Tab.1 and Tab.2 in our manuscript. Below is a summary for the reviewer's convenience:

The significant difference in parameter counts stems from: Prior Methods (e.g., UnAV, CCNet) rely on a large transformer backbone to process a very long sequence of video features and capture long-range dependencies across different time spans. PreFM, in contrast, is built upon a custom-designed, lightweight UHA block. The purpose of the UHA block is to efficiently perform two targeted tasks on a short history window: fusing current temporal-modal features and intelligently modeling the future.

W5: What are the implications of the method on training time ?

A5:

PreFM offers a compelling advantage in training time.

We use the official code to reproduce the second-best baseline methods CCNet on the On-AVEL and MM-CSE on the On-AVVP tasks. These results, including the number of epochs, total training time required to converge, and performance, are summarized in the table below. All tests are conducted on a single RTX 3090 GPU:

Due to its lightweight and efficient design philosophy, Our PreFM framework demonstrates a significant advantage in training efficiency.