Aligning Audio-Visual Joint Representations with an Agentic Workflow

We thank for the valuable comments and answer the raised questions below.

W1: Details in the Planning Section

The 'Planning' stage of our workflow utilizes a predefined module that automatically selects appropriate actions based on audio-visual misalignments identified by the LLM. Each action, such as noise reduction or speed adjustment, is applied using default settings from established libraries (e.g., SciPy), which are optimized for general usage. This automation does not need manual intervention, allowing our workflow to scale efficiently across numerous examples without requiring manual tuning of parameters. We will illustrate that our action setting follows prevalent designs and does not require manual adjustment in our revised manuscript.

W2: Textual Descriptions for Planning

We understand the raised concern. In practice, we empirically find the textual descriptions generated by the mLLMs (i.e., Gemini, Video-LLaVA) provide a robust basis for LLM (i.e., Vicuna) to plan effective actions. These descriptions from mLLMs indeed encapsulate key audio and visual characteristics, enabling our automatic workflow to make informed decisions about necessary adjustments. Our transformation into language form leverages the interpretability and rich semantic understanding capabilities of mLLMs, ensuring that the planned actions are both relevant and effectively targeted to address specific misalignments.

W3: Technical tricks

We would like to show that our agentic workflow based on multiple mLLMs are capable of aligning audio-visual data pairs to benefit representation learning. This cycle alignment form fully leverages the generalizations of mLLMs to process data. Our technical contributions (or tricks) reside as follows:

1. We pioneeringly develop an agentic workflow containing reasoning/planning, tool use, and reflection steps for AV (audio-visual) data alignment. In contrast, prior agentic workflows are not developed by considering all these steps. For example, [a,b] mainly propose planning, [c,d] mainly develop tool use, and [e,f] mainly introduce reflection. Moreover, there is no agentic workflow developed for AV data alignment in the literature.
2. We convert AV data in the language form to fully exploit the reasoning/planning ability of LLM. This is because we empirically find out the mLLM/LLM reasons and plans more effectively in pure language form rather than AV form, which might due to the imperfect alignment between language and other modalities.
3. For each data sample, we save the final action corresponding to the video and audio file name and the preprocessed audio wav file. Our AVAgent is based on Vicuna-v1.5-7b and we adopt LoRA tuning so as not to affect the original reasoning ability of LLM. LoRA tuning enhances our LLM’s memory and context retention capabilities, enabling it to make more informed decisions regarding AV synchronization. We implement LoRA tuning on the LLM to fine-tune it specifically for tasks that require a deep understanding of the audio-visual context. This involves adjusting the LLM’s parameters to remember better and integrate lengthy audio-visual sequences.

We will add these illustrations in our revised manuscript.

[a] AutoGPT https://github.com/Significant-Gravitas/AutoGPT

[b] Visual Programming: Compositional visual reasoning without training. CVPR'23

[c] Llava-plus: Learning to use tools for creating multimodal agents. Arxiv'23

[d] Gpt4tools: Teaching llm to use tools via self-instruction. NeurIPS'23

[e] Self-refine: Iterative refinement with self-feedback. NeurIPS'23

[f] Reflexion: Language agents with verbal reinforcement learning. NeurIPS'23

Dear Reviewer GoMh,

We hope this message finds you well.

Thanks again for your insightful suggestions and comments. As the author-reviewer discussion time is ending, we want to ensure that we address any remaining uncertainties or questions you may have.

After carefully considering your comments, we have taken the necessary steps to provide additional clarifications, particularly regarding the technical details and contributions. We hope that our explanations contribute to a clearer understanding of our workflow pipeline.

We are open to providing further clarifications or conducting additional experiments if needed. Please feel free to reach out to us with any further inquiries.

Once again, we are thankful for your time and attention to our work.

Best regards,

The Authors

We thank for the valuable comments and answer the raised questions below.

Capability of the Proposed Workflow

We agree that there is much room to improve our actions. At present, we focus on the audio-visual downstream scenarios such as audio-visual classification, sound localization, segmentation, and separation. These scenarios are related to audio-visual recognition where we are able to handle normal audio-visual sync and noisy problems. The results in Table 2-4 have shown our superior performance. In the future, we will tackle these abnormal/challenging problems by incorporating learning-based enhancements that are capable of dynamic and context-sensitive adjustments. This will allow for more robust handling of time delays and overlapping audio sources, significantly enhancing the system's utility in real-world, noisy environments.

Capability of Multi-modal LLMs in Handling Multiple Audio Sources

Our current implementation primarily utilizes Gemini for single audio source scenarios to ensure clear and focused audio captioning. While Gemini is capable of handling multiple audio sources, in this study, we deliberately focused on single-source audio to maintain clarity and control in our experimental setup. No special processing was applied to adapt the multi-modal LLM (mLLM) specifically for audio captioning beyond its original training, ensuring that our results are broadly applicable and reproducible using standard LLM configurations.

Effect of Comprehensive Audio Preprocessing

To explore the potential of integrating comprehensive audio preprocessing steps, such as noise filtering, volume normalization, and blanks filling, we conducted additional experiments. These steps were applied randomly across all audio samples before training the audio-visual representation models. As shown in Table 5 in the original draft, our approach with agentic alignment significantly enhances downstream task performance compared to the vanilla baseline using random actions. We further conducted experiments on including all actions as preprocessing for all audio samples in Table 5 in the attached pdf, confirming that our agentic alignment can substantially benefit audio-visual learning, especially in complex acoustic environments.

Dear Reviewer siDn,

Thank you for your continued engagement and support. We appreciate your comments regarding the use of challenging tasks.

Most videos in the existing audio-visual datasets are single-source, and our agentic workflow has been very effective in aligning audio-visual joint representations for diverse downstream tasks. Furthermore, the performance of existing audio-visual representation learning methods has been significantly improved.

Based on your suggestions, we further applied our agentic workflow to another challenging multi-source video dataset, AVSBench-MS3, for audio-visual segmentation. In this dataset, all the videos contain multiple sources from 23 classes, e.g., a video of a man singing while playing violin and piano. The challenging task is to predict the pixel-level segmentation task for each sounding source. Our agentic workflow can adapt multi-modal LLM (mLLM) specifically for audio captioning to discriminate each sound source as in-context input for action planning to achieve alignment. The comparison results with the strong baseline [UFE] are reported in the Table below. These improvements demonstrate our method's potential as a beneficial preprocessing step for challenging multi-source tasks.

[UFE] Audio-Visual Segmentation via Unlabeled Frame Exploitation, CVPR’24.

We hope this multi-source visual-audio task can respond to your concerns. Thank you once again for your insightful comments.

We thank for the valuable comments and answer the raised questions below.

W1: Clarification of Visual Alignment and Temporal Synchronization Scores.

In our framework, visual alignment scores are computed using ImageBind, which assesses the similarity between visual data and modified audio descriptions. Temporal synchronization scores are derived from X-CLIP embeddings, providing a robust measure of how well the audio content temporally aligns with the visual frames. Audio embeddings are generated using a standard log-spectrogram approach, with a sampling rate of 10s audio at a sample rate of 16000Hz to ensure consistency with video frame rates. The alignment scores are provided as the context input into LLM to determine which action should be used for the next iteration. Therefore, these alignment metrics are computed after any audio editing to provide feedback context for iterative refinement in the next step.

W2: Comparison with Previous Methods Using Only Preprocessing Steps

To isolate the impact of our agentic alignment preprocessing, we retrained several state-of-the-art models using both the standard preprocessing methods and our enhanced approach. The comparisons are reported in the Tables 1-3 in the attached pdf. Our agentic alignment preprocessing boosts all models in terms of diverse downstream tasks.

W3: Suggestions on Datasets Introduction

Thanks for your suggestion. We updated the detailed settings in Table 4 in the attached pdf to introduce all the datasets.

Q1: Audio Model

The audio captions in our system are generated using Google Gemini, which has shown robust performance in general audio understanding tasks across diverse datasets.

Q2: Release of Action Planning Paired Data

We will make the action planning paired data publicly available. This dataset includes video-audio data pairs, and the corresponding actions that are from the action pool illustrated in Fig. 3. For each data sample, we save the final action corresponding to the video and audio file name in a json file, which is formatted as [{video_audio_file1: action1}, {video_audio_file2: action2}, ...]. We also share the preprocessed audio wav file with more researchers to use for extended usage. Each video-audio data pair is meticulously curated to demonstrate specific alignment and synchronization challenges and their solutions.

Q3: Frequency of Computing VLM Alignment and Temporal Synchronization Scores

1. VLM alignment and temporal synchronization scores are computed following each proposed action to provide immediate feedback.
2. Initial scores are also computed before any audio editing to establish a baseline, ensuring that each action's impact can be accurately assessed.
3. These scores are computed each time after the proposed actions for the LLM to reflect on.

Q4: STFT and Log Spectrograms

We would like to clarify that [48] also used STFT, which they did not explicitly mention in the paper but STFT was used in their provided source code. To compute the log spectrograms, we follow their implementation and apply an STFT using approximately 50ms windows with a hop size of 15ms, resulting in an input tensor of size 128 × 196 (128 mel frequency bands over 196 time steps) on the original audio waveform.

L1: Generalization of Preprocessing Method

To demonstrate the general applicability of our preprocessing method, we have conducted more experiments where we apply our techniques to several state-of-the-art models across different tasks. Results, as shown in Tables 1-3 in the attached pdf, illustrate marked improvements in performance, highlighting our method's potential as a beneficial preprocessing step for a wide range of audio-visual applications.

We thank for the valuable comments and answer the raised questions below.

W1, Q1: Dataset Specification Clarity

For our evaluations, we specifically utilized the AVSBench-semantic subset, which focuses on semantic segmentation tasks. This choice is pivotal as it directly relates to our method's ability to enhance audio-visual alignment, which is critical for accurately segmenting complex scenes based on both audio and visual cues.

W2, Q2: Applying to Other Frameworks

To demonstrate the general applicability of our preprocessing method, we conducted additional experiments where we apply our techniques to several state-of-the-art models across different tasks. Results, as shown in Tables 1-3 in the attached pdf, illustrate marked improvements in performance, highlighting our method's potential as a beneficial preprocessing step for a wide range of audio-visual applications.

Q3: Performance on AVVP

We further conducted experiments on AVVP to quantitatively compare the segment-level results in Table 6 in the attached pdf. Adding our agentic alignment to the strong baseline [VALOR] boosts the performance in terms of all metrics.

[VALOR] Yung-Hsuan Lai, et al. Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser. NeurIPS 23.

Q4: Computational Demands Discussion

The computational demands of our method are primarily driven by the use of large-scale LLMs and multimodal processing. While our current implementation is designed for offline preprocessing of datasets before model training, understanding the computational requirements is crucial for practical application. Future iterations of our work will focus on optimizing these processes to allow for more efficient real-time applications, potentially broadening the method’s applicability in resource-constrained environments.

Dear Reviewer RJkZ,

Thank you for your continued engagement and for acknowledging the improvements and clarification. We appreciate your comments regarding the use of the AVSBench-semantic subset and our method's performance enhancements in AVVP tasks.

Our experiments were conducted using NVIDIA Tesla A100 GPUs. This choice was driven by the need for high computational power to handle the intensive processing demands of our multi-modal LLM-based workflow. The preprocessing time for each dataset varies based on the size and complexity of the audio-visual content. On average, preprocessing each example in the datasets took approximately 0.05 minutes. This includes time for initial alignment checks and the application of the initial corrective actions proposed by our AVAgent. The total time to process the complete dataset for the LLP dataset containing around 11,849 examples was approximately 9.8 hours.

We hope this additional information addresses your queries satisfactorily and illustrates the thoroughness with which we have approached each aspect of our study. Thank you once again for your insightful comments and for the opportunity to enhance the clarity of our work.