AV-Cloud: Spatial Audio Rendering Through Audio-Visual Cloud Splatting

We thank the reviewer for a thoughtful review, valuable feedback, and recognizing the innovative use of Audio-Visual Anchors and Cloud Splatting, comprehensive experimentation, and clear presentation. We address the questions and specify the intended revisions below.

W1: AVCS Explanation

In the general response section we provided a more detailed explanation of the AVCS module. We place the details relevant to the reviewer's raised point below for convenience as well. We will add these details to the methods upon revision. The output of the Visual-to-Audio Splatting Transformer can be defined as follows:

1. Attention Mask a_{ki}: Indicates the contribution weight of each anchor, showing how much each anchor influences the spatial audio effect. Weighted sum the latent Audio Embedding e_i to get mixture Audio Embedding e’. The mixture mask m_m is derived from e’ through an MLP.

2. Output: The final integrated Relative Vector embedding of anchors w.r.t the target viewpoint pose. The difference mask m_d is derived from the Output embedding through an MLP.

In Equation (2), the softmax function is applied to compute attention weights a_{ki}. This softmax function normalizes the weight of each anchor contribution, enhancing the spatial audio effect to match the listener's perspective pose. Higher weights indicate greater influence, allowing the AVCS module to dynamically adjust the anchor contribution for audio rendering based on the listener's viewpoints. Please see the visualization examples in Figure 6.

W2: Spatial Audio Render Head (SARH)

SARH utilizes a one-layer residual structure with two convolution modules to enhance stereo output from Equation (4). The skip connection directly takes the output stereo channels from Equation (4) S_L and S_R. And the residual path includes:

• Time Filters: Adjust the energy distribution of the mixture output spectrogram S_m across the time domain to match reverberation effects.

• Conv2D Layers: Smooth and enhance the time-frequency domain energy distribution post Time Filters. The input to the Conv2D layers consists of four parts:

  • Output of the Time Filters convolved with the mixture spectrogram S_m.
  • Mixture mask m_m​.
  • m_L=−m_d for the left channel or m_R=+m_d for the right channel post-process.
  • Original stereo output from Equation (4): S_L for the left channel or S_R for the right channel post-process.

The Conv2D layers process these inputs to generate the left and right residual channels. The final result is obtained by adding these residual outputs to the original S_L and S_R spectrograms from Equation (4).

Contribution of key components

We studied and verified the contribution of key components in SARH in ablation studies described in Sec 4.3 and Table 2. In particular:

• Baseline Model (w/o AVCS and residual convolution modules): Utilize MLP to predict two acoustic masks. Compared to the full AVCloud, the baseline does not perform well across all five metrics, indicating limited generalization to novel views when relying solely on viewpoint poses. (Line 302-305)

• w/o AVCS:

  • Replace AVCS module with an MLP. w/o AVCS is unable to capture the relationship between left-right channel energy and viewpoint poses well, reducing the LRE accuracy by 20% compared to our full AVCloud (Line 306-310).
  • Compared with the Baseline, the residual convolution module can improve perceptual quality (0.357 -> 0.318), reverberation effect (0.13 -> 0.08), magnitude (0.488->0.468) and spatial effect metrics (1.329 -> 1.124) largely.

• w/o time: Removing Time Filters to study its effect. Reverberation time error increases from 0.074 to 0.084, and LRE error increases by 5%, demonstrating the Time Filters’ crucial role in adjusting time-domain energy distribution, impacting reverberation time and overall acoustic quality. (Line 327-329)

As suggested, we conducted two additional ablation experiments to study the effect of 1) two binaural masks and 2) Convolution 2D layers in the residual block.

• w/o 2 masks: Direct prediction of two-channel spatial effect transfer mask from the AVCS output (instead of predicting difference mask and mixture mask separately). As a result, LRE significantly increases from 0.936 to 0.983.

• w/o conv2d: Removal of 2D convolutional layers in the residual block and retaining Time Filters only. Without Conv2D layers, the Time Filter module handles reverberation but fails to distinguish between left and right channels well, indicated by LRE increasing from 0.936 to 1.019. Other metrics remain relatively stable. The need for Conv2D layers stems from reliance of residual blocks to post-process in the frequency-time domain.

We will include these results with detailed explanations and analysis in our final revision.

W3: Challenging elements such as interfering sounds

In our real demo, we successfully handled sounds like wind and bird chirping since the reference input captured these sounds. However, we indeed acknowledge that the scenario of dealing with highly noisy environments would pose a limitation to the approach, particularly scenarios where the ground truth contains noise, but the reference sound does not. Moreover, our current system is designed for real-time processing and its components are thus of lightweight nature and are not designed to cope with extreme noisy cases.

Future solutions could be to consider additional pre-processing/processing such as noise reduction, sound separation, or the design of a higher capacity model. These are beyond our current scope but could be integrated with future developments. We appreciate the reviewer’s insight and will consider this in future works.

We thank the reviewer for a thoughtful review, valuable feedback and recognizing the novelty of AV Anchors for 3D audio-visual scene reconstruction.

W1 & Q1 Difference between AVCS and Q-former

While AVCS and Q-former are transformer-based structures, they serve different purposes and utilize transformer outputs in distinct ways.

Q-former serves as an intermediary between a frozen image encoder and a frozen Language Model. Its query is a set of learnable vectors designed to extract visual features from the frozen image encoder. The query acts as an information bottleneck, providing the most useful visual features for the Language Model to generate the desired text output.

In contrast, AVCS is designed to learn features that adapt to view poses from a 3D scene representation to derive the audio spatial effect transfer function. This function converts monaural reference sound into stereo audio at the listener’s viewpoint. Each query in AVCS corresponds to a specific frequency band, and the output is the integrated Relative Vector, embedding the viewpoint pose relative to the 3D anchors' scene representation. In contrast to Q-former, AVCS involves explicit projection logic that adapts to the viewpoint pose in the 3D world, rather than implicitly extracting visual features relevant to audio.

Specifically, in the AVCS module, each Audio-Visual Anchor is projected to the head coordinate system of the target listener, and the anchor features are then integrated for each audio frequency band using the Visual2Audio Splatting Transformer. The attention mask indicates the contribution weight of each anchor, showing how much each anchor influences the spatial audio effect. This mechanism dynamically adjusts the contribution weights of anchors for audio rendering based on the listener's viewpoint. The outputs and attention weights are used to derive the mixture and difference audio masks separately for the audio transfer function.

The Visual-to-Audio Splatting Transformer works as follows:

Input

1. Query: Frequency embedding for each audio frequency band. Shape (F, C)
2. Key: Combined RGB visual feature of each anchor and its Relative Vector in the listener's head coordinate system. Shape (N, C)
3. Value: Relative Vector. Shape (N, C)

Output

1. Attention Mask a_{ki}: Indicates the contribution weight of each anchor, showing how much each anchor influences the spatial audio effect. Shape (F, N). Weighted sum the latent Audio Embedding e_i (N, C) to get mixture Audio Embedding e’ (F, C)
2. Output: The final integrated Relative Vector embedding of anchors with respect to the target viewpoint pose. This embedding is highly relevant to the 3D pose of the listener. Shape (F, C)

In Equation (2), the softmax function is applied to compute attention weights a_{ki}. This softmax function normalizes the weight of each anchor contribution, enhancing the spatial audio effect to match the listener's perspective pose. Higher weights indicate greater influence, allowing the AVCS module to dynamically adjust the anchor contribution for audio rendering based on the listener's viewpoints. Please see the visualization examples in Figure 6.

We thank the reviewer for a thoughtful review, valuable feedback and recognizing the importance of 3D audio visual scene synthesis and our contribution of proposing the novel parallel pipeline for audio and visual rendering. We address the questions and specify the intended revisions below.

W1 - Primer on SfM

SfM (Structure from Motion) reconstructs a 3D environment from a sequence of 2D images. SfM points are distinct and recognizable points in a scene detected in images. SfM points are used to determine the camera viewpoint and 3D coordinates leading to a detailed 3D point cloud. In Line 64-65, we included the reference for SfM [1] and SfM details are described in [1, Sec. 4.2]. As suggested by the reviewer, we will include a primer for SfM in the appendix that expands upon the short abstract of SfM and SfM points above and includes some key details from [1].

W2 - SfM and their clustering for AV generation

We found SfM points to be very informative to AV reconstruction due to:

1. They capture detailed 3D scene geometry, representing the physical boundaries and surfaces from which sounds can reflect, diffract, or be absorbed. This geometric information is key for rendering realistic audio-visual scenes and can be reused by any emitter and listener within the scene. Recent audio renderers [42, 32] also use point-based representations, showing more effective generalization with the same training data (Lines 109-128).
2. SfM points are also used to initialize visual renderers, e.g. 3D-GS[3], allowing for parallel synchronization of audio and visual rendering.

Clustering: Clustering reduces the density of raw SfM point clouds, enhances computational efficiency while maintaining key geometric boundaries and surfaces (Lines 144-146, visualized in Figure 6).

We will further clarify our motivation in the introduction and method sections.

W3 - AV anchors their use for AV generation

After clustering, we initialize the anchor features using: 1) 3D coordinates; 2) RGB values; and 3) Latent Audio Embedding e_i​ (Lines 140-151) for a point-based audio-visual representation. Specifically, the Audio Embedding captures how the anchor region contributes to sound propagation for different listener viewpoints, similar to boundary bounce points in INRAS [42].

AV Anchors derive the spatial audio transfer function via the AVCS module (Sec 3.2). We have included an additional, more practical, and detailed explanation of the AVCS module in our response to all reviewers and we will incorporate it in the revision as well. Each anchor is projected into the listener's head coordinate system, adapting to head orientation with the following components:

1. 3D Coordinates: Calculate the Relative Vector, serving as Key and Value for the Visual2Audio Splatting Transformer.
2. RGB Values: Positional encoding for the Key to enhance scene understanding and distinguish between Anchors at different locations (lines 170-174).
3. Latent Audio Embedding: Weighted by the Attention Mask to get the mixture Audio Embedding. The mixture mask is derived from this embedding through an MLP, manipulating mixture sound magnitude changes. The final integrated Relative Vector from the transformer is used to obtain the difference mask through MLP.

W4 - Difference between AVCS and AV-Mapper

AV-Mapper (AVNeRF) relies on RGB and depth images, making it dependent on image data to adapt to viewpoint changes. In contrast, AVCS module decouples the transfer function from images, using point-based (AV anchors) scene representation. By anchor projection and the Visual-to-Audio Splatting Transformer, our approach adapts to view poses from a view-independent 3D scene representation.

Advantages:

1. Reusable anchor representation for all viewpoints, enabling better generalization to novel viewpoints with better accuracy, fewer parameters, and higher inference speed (Table 1).
2. Explicit adaptation to 3D world coordinate system, eliminating the need for visual rendering to obtain strong visual cues. The approach can achieve more efficient and accurate audio-visual synchronization without reliance on visual render results to obtain strong visual cues.

W5 - Subjective Tests Justification

We particularly set the subjective test instructions to target evaluation of the alignment of spatial audio effects with visual content. Participants were instructed to select the video where spatial audio matches the visual content, focusing on left-right ear effects and overall synchronization. Instructions included:

“… Select the video that features the spatial audio effect best matching the visual content.… Pay attention to the varying left-right ear spatial effects… Evaluate each video comprehensively for spatial effects, audio continuity, and quality…”

This test complements quantitative metrics like LRE, providing a comprehensive assessment. Our method received 48% of the votes, outperforming other methods, demonstrating its effectiveness in creating an immersive audio-visual experience.

W6 - Comparison with NACF

We thank the reviewer for recommending to include NACF in the evaluations. Similar to INRAS [32] and NAF [42], NACF does not include viewpoint-based reweighting, which leads to less effective representation. AVCloud, with dynamically reweighted Audio-Visual Anchors, significantly improves audio rendering metrics. The Spatial Render Head also enhances reverberation effects, achieving a more accurate acoustic metric (RTE). We will include these references and results in our revision.

RWAVS Dataset:

Replay-NVAS Dataset: