AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis

Weakness

W1: Method part isn't clearly stated We will re-draw Figure 2 (Overview of our proposed AV-GS) to make it more clear and straightforward. And update all figures to PDF.

W2: PCA feature of learnable acoustic points (The PCA plot and the correlation matrix discussed below have been provided in the PDF attached above, in the common "Author Rebuttal") For RWAVS scene 6, we analyze alpha by plotting its first two principal components and applying k-means (k=3) clustering, to project colors onto G_a. It can be observed that most of the objects are grouped by the same color. Next, we manually match scene objects (e.g., fireplace, walls) with material labels (e.g., 'brickwork', 'solid wall' respectively) from an absorption database [6]. Using these absorption coefficients, we assign absorption vectors (Dim: [1x6]; coefficients for 6 frequencies from [6]) to every point that corresponds to the object (matched with the label). A correlation matrix of these absorption coefficients, across the points (grouped by clusters), shows high self-correlation within clusters and lower cross-correlation across clusters, i.e., the points within the same clusters have similar absorption values, and less correlated absorption across the clusters. While these clusters do not explicitly highlight fine-grained material properties such as Young's modulus, they provide implicit material cues for binauralization.

Additionally, we demonstrate the effect if alpha is frozen during our training (initialized using SH from vanilla 3D-GS). As shown in the table below, constraining alpha to SH results in a significant drop in binarualization.

Combined with the PCA/correlation matrix above, and this experiment, we hypothesize that the learned alpha is able to pick up implicit material properties on top of SH to aid binauralization. We agree that alpha captures an implicit, rather than explicit properties like Young's modulus, material representation derived from raw RGB images and will clarify this in the revised version.

W3: confirm if same train-test split and evaluation code as baseline Thank you for pointing this out. We adopt the same training and test split and the evaluation code provided by AV-NeRF (available on Github) and hence making the comparison fair. Please note AV-NeRF provides train and test json files, and we adopt the same dataloader from their code, which makes the comparison fair.

W4: Position-guidance. Isn't distance an important cue? Great question! We found using only the view information for position guidance suffices for the model performance. We found that this normalization in Eq. (3) provides numerical stability during training. With regards to the distance being an important que, please note that the position embedding of the listener location is already a part of the binauralizer, please check the Fig. 7 in A.2 We skip the arrow (showing X_L/X_S into the binauralizer) in the Fig 2, to maintain brevity and not deviate focus of the novel contributions. Instead we tried to highlight this in our Binauralizer Fig. 7 in A.2. We will update this in the revised version.

W5: Does energy decay loss help? Thank you for pointing this out. We adopted the decay loss in addition to our proposed losses, however we did not find any improvement. (please note, for the RWAVS dataset, we compute the l1 loss between the decay curves for the GT and the predicted spectrograms of the binaural audio).

Following RAF[7] we also ablated the weight of the decay loss in the overall loss computation. However, in contrast to RAF's findings where decay loss improves temporal domain metrics, but worsens temporal-frequency domain metrics (STFT error), we find that it worsens both temporal (Envelope distance (ENV)) and temporal-frequency domain metrics (spectrogram magnitude (MAG).

References: [1] Gao, Ruohan, and Kristen Grauman. "2.5 d visual sound." CVPR'19

[2] Luo, Andrew, et al. "Learning neural acoustic fields" NeurIPS'22

[3] Su, Kun, Mingfei Chen, and Eli Shlizerman. "INRAS" NeurIPS'22

[4] Liang, Susan, et al. "AV-NeRF." NeurIPS'23

[6] C. Kling. Absorption coefficient database, Jul 2018.

[7] Chen, Ziyang, et al. "Real acoustic fields" CVPR'24

[8] Tang, Zhenyu, et al. "GWA" SIGGRAPH'22

Weakness

W1: Explicit modeling of alpha We appreciate the reviewer's insight regarding the explicit modeling of alpha.

"there is nothing constraining alpha to be physics-based or correlated with material parameters"

While we acknowledge the importance of deriving alpha from material properties input, we would like to clarify that AV-GS is motivated by the latest trend towards scalability and adaptability for real-world scenes like RWAVS[4] and Real acoustic field[7], where only RGB frames are available – a more practical and least constrained setting. In contrast, other works [5] [8] using absorption coefficients rely on synthetic scenes and pre-existing absorption databases, matching object labels with material descriptions which are available for synthetic scenes but typically unavailable in real world scenarios as considered in our work.

Please consider that currently, anyone with a phone and a binaural microphone can capture all input data that is required to train an AV-GS model (~4 min of recording for a single room scene) - something not possible if explicit material properties are required for model training.

We agree that alpha captures an implicit, rather than explicit, material representation derived from raw RGB images and will clarify this in the revised version. (Also, please see Q2 below, where we demonstrate that alpha, although implicitly learned, is related to material properties.)

W2: Explicit modeling of acoustic field network Explicit sound propagation modeling is indeed valuable, but our acoustic field network design is intentionally non-trivial. We learn acoustic properties on 3D-GS points, accumulating points within an empirically validated vicinity of the listener and source. We fuse implicit material (alpha) with learned position guidance, using point view directions relative to listener/source. This is further facilitated by audio-aware adaptive density control to priortise point density across texture-less regions -something vanilla 3D-GS lacks, but is important for NVAS. We agree explicit beam-tracing within audio-guided 3D-GS is an interesting future research direction.

Regarding Binauralizer, as mentioned in L166, it is not a part of our contribution. It was motivated in [1] and adopted by AV-NeRF[4]. We further improve it by deriving holistic conditions from 3D scenes, specifically- learning implicit material properties, fusing with position guidance, and adaptively densifying the point-based representation w.r.t audio reconstruction loss.

W3: Qualitative audio samples Because of rebuttal instructions regarding not sharing links, we have provided the AC with an anonymized link to the audio samples. Apologies for the inconvenience. We will also make our code repo and all rendered samples publicly available post the review period.

W4: Typos Apologies! we will address all the typos in our revised version.

Question

Q1: only 6 scenes from Soundspaces? We chose six indoor scenes from the SoundSpaces dataset to ensure consistency and valid fair comparisons with previous works, specifically NAF (NeurIPS'22), INRAS(NeurIPS'22), and AV-NeRF(NeurIPS'23), which established benchmarks using these six scenes. They represent a diverse range of environments: (1) Two single rooms with rectangular walls, (2) Two with non-rectangular walls, and (3) Two multi-room layouts. Currently both RWAVS and soundspaces-synthetic are publicly available, and hence used for our fair and consistent validation. We can incorporate this into our revised version with similar data (binaural audio and RGB pairs) for other scenes.

Q2: Interpretability of audio-guidance parameter (The PCA plot and the correlation matrix discussed below have been provided in the PDF attached above, in the common "Author Rebuttal") For RWAVS scene 6, we analyze alpha by plotting its first two principal components and applying k-means (k=3) clustering, to project colors onto G_a. It can be observed that most of the objects are grouped by the same color (3 clusters - orange, green, blue). Further, we also manually match scene objects (e.g., fireplace, walls) with material labels (e.g., 'brickwork', 'solid wall' respectively) from an absorption database [6]. Using these absorption coefficients, we assign absorption vectors (Dim: [1x6]; coefficients for 6 frequencies from [6]) to every point that corresponds to the scene object matched with the material label from [6]. A correlation matrix of these absorption coefficients, across the points (grouped by clusters), shows high self-correlation within clusters and lower cross-correlation across clusters, i.e., the points within the same clusters have similar absorption values, and less correlated absorption across the clusters. While these clusters do not explicitly highlight fine-grained material properties such as Young's modulus, they provide implicit material cues for binauralization.

Additionally, we demonstrate the effect if alpha is frozen during our training (initialized using SH from vanilla 3D-GS). As shown in the table below, constraining alpha to SH results in a significant drop in binarualization.

Combined with the PCA/correlation matrix above, and this experiment, we hypothesize that the learned alpha is able to pick up implicit material properties on top of SH to aid binauralization.

References: [1] Gao, Ruohan. "2.5 d visual sound." CVPR’19

[4] Liang, Susan, et al. "AV-NeRF." NeurIPS’23

[5] Ratnarajah, Anton "Listen2Scene" IEEE VR’24

[6] C. Kling. Absorption coefficient database, Jul 2018.

[7] Chen, Ziyang, et al. "Real acoustic fields" CVPR’24

[8] Tang, Zhenyu, et al. "GWA" SIGGRAPH’22.

Weakness

W1: Value of T_g T_g is inspired by vanilla 3D-GS paper (which empirically found a threshold of 0.0002) and consistently our threshold of 0.0004 was determined through empirical testing.

In 3D-GS optimization (trained for NVS), the point's gradient is compared with the T_g threshold, and is decided whether the point should be cloned/split or not. Particularly in this case the gradients correlate to ability of the points to contribute to the visual geometry of the scene when they are splatted in a particular camera-view direction. In our NVAS case, the gradient of the points correlate to the contribution of the point in providing an acoustic guidance when generating binaural audio for a given {X_L, X_S} pair. Please note NVAS is not limited to only the camera-view direction, but rather vicinity of the listener and sound source.

Our empirical choice is in-line with 3D-GS, and also intuitively higher -because we adopt a dual stage training, wherein the rough geometry is already provided to 2nd stage (Structure-from-Motion points are provided to the first stage), and comparatively considered for a much different sample of points (volumetric camera view Vs listener and source vicinity). A higher value helps to save compute cost.

Given a reasonable memory budget, ideally one would want to empirically decide on the threshold for every different vicinity size.

Minor comments We sincerely appreciate the discussion regarding general acoustics, we will incorporate this in our next version with the 15% vicinity plot. We will relate our Intro section to the wave-based (good for lower frequency bands --> resulting in diffusion, scattering .. i.e. wave properties) and Geometric acoustic-based methods (good for higher frequencies --> behaves like rays --> ideally specular reflections .. i.e. ray properties).

Question

Q1: S omni-directional? Right, the sound source is omni-directional, we adopt the same dataset as AV-NeRF [4]. We will mention this explicitly in our revised version. To extend AV-GS for directional sources, directional components can be added (perhaps weighing the vicinity sphere in the direction).

Q2: Notations Sincere apologies for the confusion. V_C and V_A: We tried to use V to represent generic views; V_C - visual and V_A - audio modality.

V_C = {I | p}; V_A = {a_{bi} | p, a_{mono}}

For the current scope of input used by AV-GS, you are right in saying, V_C = I and V_A = a_bi. However, in a more generalized NVAS setting where one may supplement the RGB with a depth view, in which case V_C = I+D (RGB + Depth).

I and a: As mentioned these are both observations, however, 'I' is a 3D matrix (RGB) hence, uppercase. 'a' is the 1D audio (time series sequence) and hence lower case. We will update this.

Tuple or set: Apologies, all should be notated as tuples, and k_L and k_S.

Q3: Dropping points outside the vicinity …, optimal vicinity is not the largest one tested. Why? Again, Great question! Although it might intuitively feel enlarging the vicinity size would help, however please note in the context of the GS learned point based representation, we are typically looking at a scale like ~400k points for a single room scene. Increasing the vicinity would account for adding this huge number of parameters which would make it harder for the field network to be burdened - It is a trade-off. We did try voxelizing the representation, so that we can deal with lesser points (and in turn lesser alpha's) despite increasing the vicinity, but it worsened the performance -we hypothesize this is because the alpha parameter for the points post-voxelization cannot be computed with a simple average (or even a MLP, like anchor points - Scaffold GS [9]). A specialized attention based selection of points for the vicinity might be an interesting future direction to explore, but please note not all points closer to the source or listener are actually (more) helpful.

Q4: Wouldn't many of the problems in this paper still exist in the monaural listener context as well? It is correct in noting that many of the challenges in this paper are relevant to monaural contexts as well. The primary contribution of AV-GS lies in capturing a holistic context of the entire scene and generating context within the 3D scene. Binauralizer arch. is adopted from AV-NeRF (who partly adopted from [1]), not claimed as a novel bit of this work. In smaller shoebox rooms (like the one's experimented in Richard et. al ICLR'21, and Office scene in RWAVS), AV-GS's improvement is less pronounced due to limited room geometry impact. But, AV-GS shows significant improvements in multi-room more complex scenes (e.g., "house" and "apartment" in RWAVS) with wall occlusions, where AV-NeRF's AV-Mapper is less efficient. (scores in Table 1 of paper) Multi-room scenes give rise to multiple wall occlusions - where AV-NeRF's AV-mapper approach is inefficient.

Q5: Value of T_g Please check W1 above.

Limitations: Broader impact unclear Thanks, we will clarify. Once an AV-GS has been trained to accurately model room geometry and material properties, synthesizing highly realistic binaural audios in real time should be possible. This could be used to create misleading audio experiences, such as simulating the presence of people or activities that are not actually occurring, which could be used to deceive or manipulate individuals. With regards to rendering large scenes, we agree, that generalization to multiple room (4 or 5, or a multi-storey house) is indeed challenging. Additionally, the problem with adopting an explicit points based representation is that the number of points increases drastically for larger scenes, where although capturing fine-grained details is not necessary (as in the case of NVS), a rough geometry is still a crucial requirement for points to be able to guide NVAS.

References:

[9] Lu, Tao, et al. "Scaffold-gs" CVPR'24

Questions (and weaknesses)

W1, Q1: how the chosen strategy for aggregating the acoustic context in G_a affects model performance. L156-7: "We obtain the condition for binauralization by averaging the context across all points in G_a, post dropping points outside the vicinity of the listener .."

In every iteration a random listener location is sampled. Please note from Eq. (3) that the formed context is already conditioned on the direction (position-guidance), which is already an explicit attention. The context averaging step happens only for the points within the vicinity (which is a smaller subset of all points that are representing the whole scene, ~400k).

We did try to learn an attention-based weighing among the vicinity points; however we often found that the weight predictor collapses terminating the optimization under a poor local minima (i.e. all points are assigned the same weight, 0.5 for a sigmoidal activated weight predictor) due to the sufficiently varied points provided in every iteration (the vicinity keeps changing drastically across every iteration depending on the sample listener location). One might think to assign weights to all points in the scene instead of only the ones within vicinity, however this is very compute intensive for a single iteration.

W2, Q2: the rationale behind using L_v, and how the model performs without the loss. Apologies for not being clear. Our intuition behind introducing alpha is to help us determine each point's contribution to the context used by the binauralizer for generating binaural audio (for a particular listener location). The regularization term encourages lower alpha to prevent over-reliance on a few points, and make the contribution of points (for forming the context) well-distributed, promoting a diverse and representative context which is key for generalization to unseen listener locations. [10] proposed a similar regularization term however their purpose is significantly different from ours, where they use it for volume minimization to curb 'visual overlap' and enable faster raymarching. We empirically show below that for AV-GS generally lower values of \lambda_a (contribution of regularization loss, Eq. (6)) are preferred.

W3, Q3: initialization of alphas during point densification in G_a. That's a great question. We tried warm initialization using both repeat (similar to vanilla 3D-GS and Scaffold-GS) as well as the nearest neighbor (NN). In comparison to random initialization, they perform almost the same. We hypothesize this firstly because the number of points (within the vicinity) that satisfy the gradient condition are much smaller, secondly, the densifying step is carried after intervals of 100 iterations, which gives the optimization (which happens every iteration) sufficient space to curb discontinuity, if any. (Not to forget the outlier removal step, carried after an interval of 3k iterations).

W4, Q4: generalization of the model to more challenging datasets like Matterport3D. We currently validate AV-GS on a real-world RWAVS and a synthetic dataset Soundspaces, which is in line with prior work in the field of novel view acoustic synthesis - NAF (NeurIPS'22), INRAS (NeurIPS'22), AV-NeRF (NeurIPS'23). As for challenges, RWAVS (real world dataset) is more challenging than Matterport3D (synthetic), especially the house and apartment scenes involving multiple real rooms in a single scene. Currently both RWAVS and soundspaces-synthetic are publicly available. We can incorporate this into our revised version with similar data (binaural audio and RGB pairs) for Matterport3D.

W5: Audio examples Because of rebuttal instructions regarding not sharing links, we have provided the AC with an anonymized link to the audio samples. Apologies for the inconvenience. We will also make our code repo and all rendered samples publicly available post the review period.

W6, Q5: Minor: i) L274-5, do these 4 views cover the full 360 degree? ii) Para in L285-90 - repeated text (i) That is right, all 4 perspective views cover the 360 degrees of listener view. In Table 3, despite the 360 view AV-NeRF falls short of binauralization performance due to lack of a combined holistic scene condition, unlike our method. (ii) Got it, we will remove L285-90 it in the update.

Limitations: Sorry for the confusion, the impact was provided in the "Appendix / supplemental material". We will update it.

Impact: Once an AV-GS has been trained to accurately model room geometry and material properties, synthesizing highly realistic binaural audios in real time should be possible. This could be used to create misleading audio experiences, such as simulating the presence of people or activities that are not actually occurring, which could be used to deceive or manipulate individuals.

References:

[10] Lombardi, Stephen, et al. "Mixture of volumetric primitives for efficient neural rendering." ACM ToG

Thanks for the responses. Could you answer/comment on the following?

1. Lambda_a: what was the value for it in the paper?

2. Audio examples: thanks for the audio sample, but I think the paper needs a lot more qualitative examples, given that it is an audio spatialization paper. A few suggestions from my end: 1) same mono audio played at different locations in the same scene (2-3 scenes of different sizes and layouts should be evaluated) + qualitative comparison with baselines, 2) repeat 1 for different mono audio samples.

I would urge the authors to add all the results tables and analyses provided in the rebuttal, in the next draft.