Novel View Acoustic Parameter Estimation

We thank the reviewer for their critical assessment of our work. In the following we address their concerns point by point. Corresponding changes to the manuscript are highlighted in blue.

Weaknesses

• W1. Our ultimate goal is indeed rendering of plausible acoustics for virtual and mixed reality applications. For this, as noted by the reviewer, we need RIRs that are perceptually plausible, but not necessarily physically accurate. The task of predicting physically accurate time-domain RIRs from limited information about a scene is challenging. Most other work in deep learning acoustics focuses on predicting RIRs with an emphasis on physical accuracy rather than perception. But this comes at the cost of requiring more complex input information (e.g. full meshes with material properties, or a large number of reference RIRs), or limited generalization to unseen scenes. An alternative approach is to use algorithmic reverberators that are capable of generating synthetic, perceptually plausible RIRs, when conditioned with accurate acoustic parameter values. Therefore, we predict acoustic parameters needed to drive these reverberators, instead of predicting the RIRs directly.

  To demonstrate this full pipeline, we have modified the manuscript to include an example of this process. We used META XR Audio SDK [1], an algorithmic reverberator, to generate RIRs. This engine uses an artificial reverberator that renders RIRs by matching the target T60s. We informally compare the sound quality between a reference RIR, an RIR generated by conditioning on the output of the model, and an RIR generated by conditioning on the noisy outputs of the baseline. The RIRs conditioned on the model predictions are perceptually close to the reference. Here we do not consider the early reflections; this is similar to other approaches in virtual acoustics that focus only on the late reverberation part, as in [2]. We added this explanation to a new section, Appendix F, in the manuscript.

• W2. We acknowledge the reviewer's suggestion, but believe our original comparison to NAF and INRAS does make sense. We extract acoustic parameters from the IRs predicted by these models, since predicting RIRs directly is the approach they take towards accurate sound rendering. We then evaluate the performance of the acoustic parameters against our approach, as RIRs must encode acoustic parameters accurately in order to be perceptually plausible for sound rendering.

Questions

• Q1. We agree with the reviewer's suggestion regarding the title. While we do use listener pose as an additional input in the spatially dependent parameter case, we have updated the title and corresponding abbreviations for greater clarity.

[1] Acoustic Ray Tracing for Unity Overview. https://developers.meta.com/horizon/documentation/unity/meta-xr-acoustic-ray-tracing-unity-overview, 2024. Accessed: 22 November 2024

[2] Alessandro Ilic Mezza, Riccardo Giampiccolo, Enzo De Sena, and Alberto Bernardini. Data-driven room acoustic modeling via differentiable feedback delay networks with learnable delay lines. EURASIP J. Audio Speech Music. Process., 2024(1):51, 2024

Questions

• Q1. We have addressed the weaknesses directly in our response above this one.

• Q2. We use standard parameters (for example, also studied in acoustics [3]) and described in ISO standards [5]. These parameters can be computed as broadband quantities, or at specific frequency bands, by first filtering the RIR and then computing the parameters on the filtered RIRs. Specifically, for the experiments in Table 1, the parameters we study are C50, T60, EDT, and DRR. They are computed at six frequency bands, centered at 125, 250, 500, 1000, 2000, 4000 Hz. This gives us a total of 24 output channels/heatmaps predicted by our models. In Table 2, we study the case of spatially-dependent parameter prediction, where the acoustic parameters are computed from spatially filtered signals. We use C50 at three frequency bands (centered at 250, 1000, 4000Hz), computed for five fixed orientations, for a total of 15 channels.

• Q3. Following the clarification in Q2 on the acoustic parameters used. We obtain the ground truth heatmaps by processing the acoustic parameters computed on the ground truth RIRs. This process uses masked average pooling to assign the value of the parameters to nearby pixels. The mask ensures that only pixels with a receiver contribute. This is needed because the number of pixels does not necessarily match the number of receiver positions. We also apply a Gaussian blur to the ground truth heatmaps as a low pass filters that attenuates large variations in neighboring pixels. This is because acoustic parameters typically do not have very large variations in close proximities. We have edited section 3.2 of the manuscript, to improve clarity where this process is described.

• Q4. The scene geometries in the MRAS dataset are not from any existing dataset such as SoundSpaces or Matterport, but custom-generated to provide a larger dataset with more complex geometries and material properties. Some examples of the floormaps of the MRAS scenes are shown in Figure 1, and a more detailed look at the statistics of the dataset are shown in Appendix A; they suggest that MRAS has greater geometric and acoustic diversity than existing datasets like SoundSpaces 1.0 or Matterport 3D. We intend to release the MRAS dataset for public use, contingent on paper acceptance.

• Q5. By generalization, we mean that we can infer acoustic parameter for scenes that are not used during training and are only seen at test time. This is described in Section 5.4 and Table 1. We concur with the reviewers that training and/or testing on real-world data would further demonstrate generalizability; however, large-scale public datasets are not available for this purpose. Datasets like [6] or [7] provide a very limited number of scenes and RIRs.

• Q6. We thank the review for pointing this references. We have incorporated these suggestions into Section 2 of the manuscript.

[3] Philipp G¨otz, Cagdas Tuna, Andreas Brendel, Andreas Walther, and Emanu¨el AP Habets. Blind acoustic parameter estimation through task-agnostic embeddings using latent approximations. In 2024 18th International Workshop on Acoustic Signal Enhancement (IWAENC), pages 289–293. IEEE, 2024

[5] ISO 3382-1:2009 Acoustics — Measurement of room acoustic parameters. https://www.iso.org/standard/40979.html, 2009. Accessed: August 2024.

[6] Ziyang Chen, Israel D Gebru, Christian Richardt, Anurag Kumar, William Laney, Andrew Owens, and Alexander Richard. Real acoustic fields: An audio-visual room acoustics dataset and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21886–21896, 2024

[7] Florian Klein and Sebasti`a V Amengual Gar´ı. The r3vival dataset: Repository of room responses and 360 videos of a variable acoustics lab. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

We thank the reviewer for their critical assessment of our work. In the following we address their concerns point by point. Corresponding changes to the manuscript are highlighted in blue.

Weaknesses

• W1. While we do use 3D meshes to obtain the floormaps in this work, this is not a requirement more generally. Floormaps can be obtained easily for real-world rooms from building construction plans, or by roughly measuring the area manually. There is a significant amount of work on floormap extraction from other low-cost modalities that do not require complete geometric knowledge of the scene, such as from sparse images [1] or from IMU data associated with walking trajectories [2]; however, we assume that demonstrating these methods are outside the scope of this work. We utilize a 3D mesh-to-floormap extraction process only as an illustrative demonstration, and because we are using synthetic room data for training and testing. We add additional text in Section 3.2 to clarify this.

  We choose to use a floormap as it is a simplified, low-complexity representation of a scene, and demonstrate that we can infer acoustic parameters without the need for more dense information such as natural images, meshes, or material properties. Regarding the small resolution of the floormap, based on our ablation study of slicing heights (Appendix E.4), it appears that fine furniture detail is not required to infer acoustic parameters accurately - the most important information includes room boundaries and doorways.

  Finally, the advantage of predicting a heatmap of acoustic parameters in the shape of the floormap is that, with a single inference step, we are able to obtain the acoustic parameters for the entire scene, without requiring an explicit source-receiver query at the model output stage. This allows us to model how the sound characteristics -- and in turn, RIRs -- change over the space in the room.

• W2. There may be a misunderstanding here. Firstly, our models do not predict RIRs; we predict a 2D map corresponding to the room's floormap, where the pixel values are determined by acoustic parameter values. In room acoustics, ``acoustic parameters" is defined as a set of parameters that describe key characteristics of an RIR [3], and can be used to reconstruct plausible RIRs in conjunction with other information [4]. In this work, we want to predict acoustic parameters since we can use this information to condition an algorithmic reverberator to generate a synthetic, plausible RIR, while using lower complexity visual and acoustic context as input to the model. We have clarified this further in the Introduction.

• W3. Acoustic parameters, just like an RIR, depend on both the source and receiver position. The heatmap that we predict can be thought of as a collection of the acoustic parameter values at all possible receiver locations paired with a given source position. Therefore, in Eq. 2 of the manuscript, $\mathcal{A}_{E_t}$ depends only on the target source $E_t$. Each channel of the model output provides such a heatmap for a single acoustic parameter (T60, C50, etc) at a particular frequency band. For example, in the experiment described in Table 1, four acoustic parameters (C50, T60, DRR, EDT) for each of six frequency bands ( centered at 125, 250, 500, 1000, 2000, 4000 Hz) leads to 24 output heatmaps produced by the model for a given source $E_t$. This explanation has been added to the text, in Section 3.3.

[1] Arnaud Gueze, Matthieu Ospici, Damien Rohmer, and Marie-Paule Cani. Floor plan recon-struction from sparse views: Combining graph neural network with constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1583–1592, 2023.

[2] Claudio Mura, Renato Pajarola, Konrad Schindler, and Niloy Mitra. Walk2map: Extracting floor plans from indoor walk trajectories. In Computer Graphics Forum, volume 40, pages 375–388. Wiley Online Library, 2021

[3] Philipp G¨otz, Cagdas Tuna, Andreas Brendel, Andreas Walther, and Emanu¨el AP Habets. Blind acoustic parameter estimation through task-agnostic embeddings using latent approximations. In 2024 18th International Workshop on Acoustic Signal Enhancement (IWAENC), pages 289–293. IEEE, 2024

[4] Alessandro Ilic Mezza, Riccardo Giampiccolo, Enzo De Sena, and Alberto Bernardini. Data-driven room acoustic modeling via differentiable feedback delay networks with learnable delay lines. EURASIP J. Audio Speech Music. Process., 2024(1):51, 2024

We thank the reviewers for their critical assessment of our work. In the following we address their concerns point by point. Corresponding changes to the manuscript are highlighted in blue.

Questions

• Q1. While we do use 3D meshes to obtain the floormaps in this work, this is not a requirement more generally. Floormaps can be obtained easily for real-world rooms from building construction plans, or by roughly measuring the area manually. There is a significant amount of work on floormap extraction from other low-cost modalities that do not require complete geometric knowledge of the scene, such as from sparse images [1] or from IMU data associated with walking trajectories [2]; however, we assume that demonstrating these methods are outside the scope of this work. We utilize a 3D mesh-to-floormap extraction process only as an illustrative demonstration, and because we are using synthetic room data for training and testing. We add additional text in Section 3.2 to clarify this.

  Finally, we claim that the work generalizes to unseen scenes, because we test on room scenes that have not been used in training. Most state-of-the-art work based on implicit representations such as NAF or INRAS are limited to interpolation within a single scene, and cannot generalize to completely different geometries.

• Q2. We have conducted an ablation study on floormap construction, detailed in appendix E.4. In general, we found little variation in performance based on how maps are constructed. This is mainly because the acoustic parameters we tested do not depend much on fine details in the scene like furniture. The most important features are walls that define the scene boundaries and doorways that connect distinct rooms. It is possible that the slicing height may impact the model's performance in a heavily cluttered scene, which in turn may impact the distribution of acoustic parameters; however, these types of scenes are unlikely in synthetic datasets and real-world scenes.

  We have highlighted the text clarifying this in Appendix E.4.

• Q3 The low pass filter is a 2D gaussian with a kernel size of 9x9 and standard deviation of 1. We have added this detail to Section 3.2.

[1] Arnaud Gueze, Matthieu Ospici, Damien Rohmer, and Marie-Paule Cani. Floor plan recon-struction from sparse views: Combining graph neural network with constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1583–1592, 2023.

[2] Claudio Mura, Renato Pajarola, Konrad Schindler, and Niloy Mitra. Walk2map: Extracting floor plans from indoor walk trajectories. In Computer Graphics Forum, volume 40, pages 375–388. Wiley Online Library, 2021