OmniAudio: Generating Spatial Audio from 360-Degree Video

We sincerely appreciate your recognition of our demo page and our dataset and all your valuable feedback. We plan to open-source the codebase in April 2025 to facilitate community-driven improvements in this direction and welcome the reviewer's specific recommendations on cutting-edge techniques worthy of prioritized exploration. We hope our response fully addresses your concerns and questions.

Open-Source Sphere360 Dataset

Please check our Response to Reviewer nf2N under Open-Source Sphere360 Dataset.

Updated Table 2 with Model Size, Inference Time, and TFLOPS

We sincerely apologize for the typos in Table 2 in the submission. Please check our Response to Reviewer Xvna under Claims that AudioSpace achieves SOTA performance As shown in the updated Table 2, AudioSpace indeed achieves state-of-the-art performance on both in-domain Sphere360 and out-domain YT360-Test. We also added the number of parameters, inference time, and TFLOPS into comparison. Although AudioSpace has slightly more parameters than MMAudio+AS, it achieves notably faster inference than all baselines.

Extended V2A Validation

We further compared AudioSpace with baselines on the perspective video-to-stereo audio (V2A) generation task on VGGSound. Under the same multimodal setting as MMAudio (i.e., + Text modality), AudioSpace+Text notably outperforms SOTA MMAudio and sets new SOTA performance on the traditional V2A task, in addition to the new 360V2SA task.

More Ablation Study on Dual-Branch Design

Thank you for the insightful suggestion for more FOV settings. We attach more dual-branch experimental results in Response to Reviewer nf2N under [Combining Equirectangular Projection with Multiple FOV Cuts]. They further demonstrate that our "Frontal + ERP" dual-branch design outperforms other dual-branch variants. We will add these experimental results and analyses to the revised paper.

Limitation and Future Work

Thank you for the kind reminder. Please refer to Appendix F for discussions of limitations and future work.

Related Works on Panoramic Video Processing

We greatly appreciate your guidance in strengthening the contextualization of our work. In the revised paper, we will add discussions on advancements in panoramic video processing into Section 2 Related Work.

User Study

In this work, we employ human evaluation based on the Mean Opinion Score (MOS) to quantitatively assess both spatial audio quality (MOS-SQ) and video-audio alignment faithfulness (MOS-AF), as detailed in Section 5.1. Additionally, we present a comprehensive case study in Section 5.3 to comparatively analyze the generated outputs of AudioSpace against baseline methods. Supplementary qualitative results are included in Appendix H. We will further expand it with additional case studies in the revised paper. To enhance reproducibility and community accessibility, we are planning to deploy an online interactive demo platform via Gradio and Hugging Face Spaces for real-time generation and evaluation.

Usage of Terminology

Thank you for this suggestion. We agree that "perspective video" more accurately conveys the conventional narrow-FOV nature of the inputs compared to panoramic formats. In the revised paper, we will replace all instances of "FOV video" with "perspective video", and add a footnote in Section 3.1 clarifying that "Perspective video" denotes standard rectilinear projections with ≤120° horizontal FOV, distinct from 360° equirectangular formats.

The Integration of more recent Flow-matching strategies

We sincerely appreciate the reviewer's constructive suggestion regarding flow-matching strategies. The proposed coarse-to-fine self-supervised pre-training and dual-branch design are agnostic to flow matching strategies. Our current framework achieves state-of-the-art performance while maintaining the fastest inference speed in both 360V2A and V2A tasks. While our experiments demonstrate that the existing configuration optimally balances accuracy and efficiency for target scenarios, we fully agree that deeper integration of advanced flow matching strategies could unlock further theoretical insights.
We plan to open-source the codebase in April 2025 to facilitate community-driven improvements in this direction, and welcome the reviewer's specific recommendations on cutting-edge techniques worthy of prioritized exploration.

Thank you for recognizing our motivation and demo, as well as acknowledging our experimental validations. We hope our response addresses all your concerns and questions.

Updated Table 2 and Inference Latency

We sincerely apologize for the typos in Table 2 in the submission. Please check our Response to Reviewer Xvna under Claims that AudioSpace achieves SOTA performance  As shown in the updated Table 2, AudioSpace indeed achieves state-of-the-art performance on both in-domain Sphere360 and out-domain YT360-Test. We also added the number of parameters and inference time into comparison. Although AudioSpace has slightly more parameters than SOTA MMAudio+AS, it achieves notably faster inference than all baselines.

Combining Equirectangular Projection with Multiple FOV Cuts

We are grateful for your suggestion regarding the dual-branch approach and its comparison with dual-branch variants. Our dual-branch design employs a 120° frontal view to concentrate on prominent sound sources while integrating panoramic features as contextual conditioning to overcome the limitations of visual information beyond the central view. To elucidate the impact of combining equirectangular projection with multiple FOV cuts, we have assessed three innovative FOV-cut strategies to replace the local FOV video: hexadirectional 360° cuts (ERP+6cuts) which capture views from front, back, left, right, up, and down directions, quadrant cuts (ERP+4cuts) which capture views from front, left, right, and back directions, and bipolar cuts (ERP+2cuts) which capture views from front and back directions.

The results show that our dual-branch strategy still achieves the best audio quality, reflected in FD and KL, and achieves similar spatial metrics compared to multiple FOV views.

Ratio between non-spatial data and FOA data for pre-training

The pre-training corpus consists of approximately 2M samples from general non-spatial audio datasets (detailed in Section 5.1) and 100K samples from the specialized Sphere360 FOA dataset, yielding a ratio of 20:1 between non-spatial and spatial audio data. Table 3 shows that coarse-to-fine pre-training substantially improves FD and KL than pre-training with non-spatial data only or FOA data only, confirming that general audio tasks effectively complement the specialized FOA domain.

Open-Source Sphere360 Dataset

We fully agree that transparency and reproducibility are critical for the research community.

1. Open-Source Release: Sphere360 dataset with metadata (including all youtube_id identifiers and timestamps) and semi-automated construction pipeline are released at https://anonymous.4open.science/r/Sphere360-CF51/.

2. Legal-Compliant Data Access Due to YouTube's Terms of Service restrictions, we cannot directly redistribute original videos. Instead, we provide a version-controlled crawling script that reconstructs the raw dataset using the provided youtube_id list.

This framework ensures that researchers can fully reconstruct our dataset while complying with content distribution policies. We will refine the documentation based on community feedback to ease the procedure.

More Different Audio Events in Appendix

Thank you for the valuable suggestion. We will expand Appendix by incorporating additional examples of diverse audio events to better demonstrate the coverage and diversity of our dataset.

More Detailed Figure for Dual-Branch Architecture

Thank you for highlighting this critical architectural detail. We will revise Figure 10 to explicitly illustrate the dual-branch fusion process with the following additions:

1. Local Feature Pathway (FOV-Audio Alignment)

   • Visual FOV patches are processed through a linear layer to align their dimensionality with audio latents.
   • The adapted FOV features are directly combined with audio latents via element-wise addition before entering the Diffusion Transformer, ensuring pixel-level spatial correspondence.

2. Global Context Pathway (360° Scene Guidance)

   • The full 360° video features are condensed into a global descriptor using max-pooling, capturing holistic scene semantics.
   • This global descriptor is fused with the diffusion timestep embedding through element-wise addition, providing consistent scene-level conditioning across all transformer layers.

We sincerely appreciate your recognition of the strong motivation and exceptional quality of our paper. We hope our response thoroughly addresses your concerns and questions.

Updated Table 2 and Inference Latency

We sincerely apologize for the typos in Table 2 in the submission. Please check our Response to Reviewer Xvna under Claims that AudioSpace achieves SOTA performance. As shown in the updated Table 2, AudioSpace indeed achieves state-of-the-art performance on both in-domain Sphere360 and out-domain YT360-Test. We also added the number of parameters and inference time into comparison. Although AudioSpace has slightly more parameters than MMAudio+AS, it achieves notably faster inference than all baselines.

Roles of Training Stages

AudioSpace includes a coarse-to-fine self-supervised pre-training stage (Stage 1) and a spatial-aware supervised fine-tuning stage (Stage 2).

1. Roles of Training Stages

Stage 1: Utilizes large-scale unlabeled audio datasets (~2M samples) to build foundational audio understanding capabilities, crucial for 360-degree video-to-spatial audio generation. It focuses on General Audio Distribution Learning through masked audio context modeling, enhancing audio feature representation and temporal coherence.

Stage 2: Tailors the model for spatial audio generation from 360-degree video. Key aspects of Stage 2 are:

• Modality Alignment: Integrates video data for precise spatial audio generation.
• Task-Specific Optimization: Refines the model for the 360V2SA goal, balancing audio quality and spatial accuracy.

2. Validation on 360V2SA Task

The table shows that after Stage 1, using 40% of Stage 2 data matches Stage 2 full-data baseline, 60% outperforms it, and 100% substantially exceeds it, in FD and KL: Stage 1 leads to reduced reliance on labeled data and increased robustness.

3. Further Validation on V2A Task

We validate the effects of training stages on traditional V2A tasks on Sphere360. Stage 1 reduces FD by 16.3% relative and KL by 19.2% relative, confirming its effectiveness in improving audio quality.

The two stages complement each other. Stage 1 builds a strong audio base, while Stage 2 adds modality-specific constraints, preventing overfitting to limited video-audio pairs. The tables highlight the necessity of both stages. Without pre-training, the model struggles with coherent audio, even with video guidance.

Video Length Constraints

Our training uses fixed 10-second clips and rotary positional encoding rather than absolute positional embedding for flexible inference lengths. However, the 10-second limit of the video datasets restricts generated audio to under 10 seconds.

The self-supervised pre-training helps with audio coherence, but reliability beyond 10 seconds is uncertain. This is due to potential degradation in modality alignment and feature consistency. Future research is needed to develop extended temporal modeling for long-form audio synthesis.

Scene Transition Cases

Thank you for raising this interesting and insightful question. Our training and demo data mainly use fixed-camera 360° videos for immersive experiences. To test AudioSpace’s robustness to scene transitions, we used 50 single-scene videos, paired them into 25 groups, and spliced 5-second segments into 10-second clips with abrupt transitions. We added interactive spatial audio for these clips on the demo page - [Scene Transition Cases]. Results show that in some examples, our model can generate transitions that are natural and seamless. It even integrates the content from both segments into a coherent whole—for instance, when the two segments showcase performances by different musical instruments, the synthesis can merge them into a balanced duet at times. However, because our training data predominantly consists of single-scene videos, the model occasionally emphasizes one segment over the other, sometimes overlooking the acoustic nuances of the less dominant scene.

More Visual Comparisons for Ablation Studies

Due to the time limit of the first rebuttal phase, we will add more visual comparisons for the ablation study in the final response.

We sincerely appreciate your constructive feedback and detailed suggestions. We hope our response below fully resolves your concerns and questions.

Claims that AudioSpace achieves SOTA performance

We sincerely apologize for the two typos in Table 2 main results and typos and boldfacing errors in Table 3 and Table 4. These were completely unintentional oversights on our part. As shown in the updated Table 2 below, AudioSpace indeed achieves state-of-the-art performance on both in-domain Sphere360 and out-domain YT360-Test.

1. Open-Source Commitment

Codebase and model weights will be open-sourced by April 2025. The Sphere360 dataset is already released at https://anonymous.4open.science/r/Sphere360-CF51/.

2.Updated Table 2 (Bold=Best, Highlight fixed numbers):

Fixed typos in YT360-Test results. AudioSpace achieves best objective and subjective results and fastest inference speed:

3. Extended V2A Validation:

AudioSpace also excels in video-to-audio generation. Please see Response to Reviewer cQFE for results.

4. Fixed Table 3 and Table 4:

AudioSpace results (row1 in both tables) are correct and best in FD, KL, and most spatial metrics. Fixed Tables 3 and 4 are on the demo page. Coarse-to-fine pre-training and dual-branch design are superior to alternatives.

Fairness of Experimental Comparisons

1.Fair Comparison Protocol

All baselines in Table 2 follow their original pre-training and are fine-tuned on Sphere360 with the same setup.

Uniform pretraining is unsuitable as baselines need multimodal pairs (text/video+audio), while AudioSpace focuses on audio diversity through domain-specific pretraining.

2. Architectural Superiority: Table 3 w/o PT (trained from scratch on Sphere360) and Table 2 show structural advantages: Without pre-training, AudioSpace surpasses all pretrained-finetuned baselines on Sphere360 in all objective metrics.

Threshold for audio-visual similarity score for data cleaning

The threshold of 1 corresponds to the original cosine similarity score scaled by 10 for easier processing, which does not impact comparative results or conclusions. The threshold was chosen based on video-audio alignment control and data analysis.

360˚ Video Representation

We appreciate the question about motion representation in ERP-formatted videos. Our dual-branch design addresses ERP limitations as follows:

1. Geometric Distortion in ERP

ERP causes planar distortions and boundary discontinuities, conflicting with CLIP encoders’ planar continuity assumptions, leading to feature misalignment. Table 4 shows single-branch ERP-only model suffers from severe feature distortion (FD: 97.83 vs. our 88.30) and semantic inconsistency (KL: 1.87 vs. our 1.58), indicating naive planar encoding fails in spherical contexts.

2. Dual-Branch Design with Frontal Perspective

ERP distorts global spatial relationships, but motion trajectories remain coherent within FOV. Our temporal modeling FoV branch isolates motion modeling to distortion-free frontal regions, ensuring stable motion encoding, as shown by Y-channel intensity variations in Fig.13.

3. Limitations and Future Directions

Objects entirely outside the frontal viewport may challenge our current approach, though they represent <5% of training data in typical 360° videos. Future work will explore geometric-aware feature rectification to address ERP distortions at polar regions.

Clarification of Inference

During inference, the frontal-view segment of the panoramic video is used for local conditioning, and the full panoramic video for global conditioning. Both features guide the diffusion transformer's ODE solver (Euler method, CFG scale=5). The audio patents are decoded by the pre-trained spatial VAE decoder to produce the final spatial audio outputs.

FOV selection criteria

During training and testing, we use the frontal-view large FOV (120°) as the default FOV video. This choice aligns with real-world conditions where front-facing viewpoints capture main sound sources.

Typos

Thank you for your reminder. Line 158 should refer to Figure 8, not Table 8, for the audio event distribution. We will make all the fixes in the revised paper.