SonicSim: A customizable simulation platform for speech processing in moving sound source scenarios

Q1: Expanding testing to include other real-world data, perhaps by offering qualitative results on some in-the-wild examples (where speech is from actual conversations rather than recorded playback), would give a more comprehensive picture of how models trained on SonicSet generalize

A1: We sincerely thank the reviewer for their suggestion. We chose ReaLMAN as the real-world speech enhancement dataset because it is the only dataset that features recordings of moving sound sources, making it particularly suitable for testing speech enhancement baseline models trained on the SonicSet dataset.

Following your recommendation, we re-recorded speech data from 16 speakers across six different scenarios, tailored for both speech enhancement and separation tasks. For the speech enhancement task, each speaker read 10 sentences from the LibriSpeech dataset in a given scenario, with varying environmental noises added randomly during the recording process. For the speech separation task, pairs of speakers in each scenario read 15 sentences from the LibriSpeech dataset at random intervals, also with randomly added environmental noise. These recordings included varying overlap rates between speakers. These data will be made available for subsequent academic research.

Using the methods described above, we constructed a new real-world recorded dataset （HumanSEP and HumanENH）and re-evaluated 19 baseline models (including both speech separation and enhancement models). In this experiment, we use NISQA and Whisper-medium-en to compute WER as metrics since there are no real labels. We used this dataset to compare with speech separation and enhancement models pre-trained on different datasets, as shown in Tables A1 and A2. The experimental results show that the models pre-trained on the SonicSet dataset can be better generalized to real conversation data. We have added this part to the revised manuscript.

Table A1: Comparative performance evaluation of separation models trained on different datasets using the HumanSEP dataset. The results are reported separately for "trained on LRS2-2Mix", "trained on Libri2Mix" and "trained on SonicSet", distinguished by a slash.

Table A2: Comparative performance evaluation of enhancement models trained on different datasets using the HumanENH dataset. The results are reported separately for "trained on VoiceBank-DEMAND", "trained on DNS Challenge" and "trained on SonicSet", distinguished by a slash.

Q4: Adding tests with varied noise levels (like signal-to-noise ratios (SNRs)) could help demonstrate SonicSet’s ability to handle diverse noise conditions.

A4: To showcase SonicSet's capability to handle diverse noise conditions, we adjusted the signal-to-noise ratio (SNR) of the test set noise to include levels of -5 dB, 0 dB, 5 dB, and 10 dB. We then evaluated both speech separation and enhancement models on these test sets. The results are presented in Tables A4 and A5.

Table A4. The performance comparison of different speech separation models under various signal-to-noise ratio (SNR) levels with environmental noise, where the SNR range is [-5 / 0 / 5 / 10].

Table A5. The performance comparison of different speech enhancement models under various signal-to-noise ratio (SNR) levels with environmental noise, where the SNR range is [-5 / 0 / 5 / 10].

Generally, as the SNR increased (i.e., the noise level decreases), the performance metrics improve for all models. The experimental results across different SNR levels confirm that models trained on SonicSet perform robustly under varying noise conditions. This demonstrates SonicSet's effectiveness in preparing models for real-world applications where noise levels can vary significantly. By providing diverse noise scenarios during training and testing, we ensure that the models can generalize well and maintain high performance in both low and high SNR environments.

Q5: It’s not clear to me how flexible SonicSim is regarding scene layouts and material properties. It would help if the authors explained what can be adjusted versus what is fixed. This would allow readers to understand any limits in SonicSim’s adaptability.

A5: We have incorporated this content into the revised manuscript. Regarding scene layout, SonicSim, leveraging the capabilities of the Habitat-sim platform, can import a variety of 3D scenes and add additional objects within these scenes. However, users cannot modify the 3D scenes themselves, as they are fixed during the import process. In terms of material properties, SonicSim supports a wide range of materials and objects, and users can modify the following physical and categorical attributes of materials:

1. Absorption: Represents the fraction of radiation or energy absorbed by the material, with the remainder potentially being scattered or transmitted.
2. Scattering: Indicates the material’s ability to scatter waves, which impacts the propagation paths of sound waves or light.
3. Transmission: Defines the material's ability to allow radiation to pass through.
4. Labels: Categorize or label materials to help identify their physical attributes.
5. Damping: Represents the damping effect of the material, influencing energy dissipation.
6. Density: Refers to the material's mass per unit volume, impacting its interaction with sound and energy. These features provide users with the flexibility to simulate a diverse range of acoustic environments and scenarios, contributing to SonicSim's versatility and realism.

Q1: Have the authors considered implementing advanced full-reference, as well as no-reference MOS metrics such as NISQA or SQAPP, or even non-matching reference metrics?

A1: Thanks to the reviewer for suggestion. We have included the NISQA evaluation metric in our speech separation and enhancement benchmarks to assess the quality of the models, as shown in the Tables 2-6 and 10-11 of the paper.

Q2: How well does SonicSim perform in dynamically noisy environments, where background sounds change over time? Would it be feasible to test in such conditions to increase practical applicability?

A2: It is important to clarify that each background noise sample in our dataset includes diverse content that changes over time. For instance, a single noise sample may contain sounds such as doors opening, footsteps, and walking, which occur at different moments within the same noise sequence. This design ensures that the simulated noise scenarios closely resemble real-world environments, thereby enabling better generalization to real-world test data.

Q6: The paper mentions several challenges in audio simulations like obstacles, room geometry, room surface but none of them end up being a factor studied by the paper.

A6: We randomly selected 300 audio files from the Librispeech test-clean set. 200 of these were used to generate noisy mixtures containing two speakers, and the other 100 were used to generate noisy audio from a single speaker. The same audio files were used in both cases. To fix the scene, we chose a cuboid room with no obstructions but included room materials and fixed the positions of the microphone, sound source, and noise. In the experiment on occlusion, we used Habitat-sim's API add_object to place objects between the microphone and the sound source to simulate the occlusion in sound wave propagation. When studying the effect of room shape, we chose a cylindrical room from the Matterport3D dataset and fixed the relative positions of the microphone, sound source, and noise. In addition, we completely removed the root material information from the experiment to explore the effect of the material.

We selected the two methods with the best speech separation and enhancement performance and verified the impact of different environmental factors on the model performance. The experimental results are shown in Tables A2 and A3. The results show that material has the most significant effect on model performance, while the impact of room shape is relatively small. However, all these factors have a certain degree of interference in model performance. Therefore, a simulated dataset using these environmental factors is generated to train the model and improve its generalization ability. This analysis has been added to Appendix A.

Table A2: Comparison of the performance of speech separation models for different environmental factors.

Table A3: Comparison of the performance of speech enhancement models for different environmental factors.

Q7: Some aspects of the audio generation settings like the microphone type, monoaural vs binaural might be good to showcase.

A7: Thank you for your suggestion. We have added the hyperparameter settings for generating audio to the Appendix C. The hyperparameter settings include aspects such as microphone types, sound source starting and ending points, and microphone positions. Additionally, we have updated the code to provide a higher-level API that allows users to directly manipulate these hyperparameters, ensuring greater ease of use.

Q1: What exactly is SonicSim trajectory function "We then employed SonicSim’s trajectory function to calculate and generate a movement path for the sound source" ?

A1: The SonicSim trajectory function is built upon the Habitat-sim platform and computes potential trajectories between a specified starting point and endpoint. These trajectories are generated using Habitat-sim's path-finding API, specifically the habitat_sim.ShortestPath() method. Subsequently, SonicSim utilizes the trajectory to synthesize moving sound sources based on the method described in A2.

Q2: The details of the moving source simulation could be improved a lot and is a major concern in the paper. Is the paper following some known method ? Some mathematical description of what is going on in the moving source case is desirable, especially for description in Section 3.2.2 ?

A2: Thank you for raising this question. We employed the widely used method of smooth interpolation [1, 2] to transition between RIRs at different positions, ensuring that the audio effect does not exhibit abrupt changes due to rapid shifts in source position. To enhance the clarity of the moving sound source simulation process, we have added the following details in the revised manuscript. SonicSim selects the starting and endpoint positions of the sound source in a 3D space and then uses Habitat-sim's path-finding API to generate a navigable trajectory. The trajectory points are used to construct a smooth interpolation path and calculate the corresponding interpolation weights, enabling realistic spatial movement effects during audio synthesis. Assuming the source audio signal is $\mathbf{s}(t)$ with duration $T$, and the room impulse responses (RIRs) $\mathbf{h}_j^c(t)$ describe the transmission characteristics from each position $j$ to the receiver, where $j = 1, 2, \dots, N$ represents $N$ discrete positions and $c$ is the audio channel index, the process is as follows:

1. Convolution at Discrete Positions:
   The source signal is convolved with the RIR $\mathbf{h}_j^c(t)$ at each position $j$, resulting in the convolution response:

$$\mathbf{y}_j^c(t) = \mathbf{s}(t) * \mathbf{h}_j^c(t),$$

where $*$ denotes the convolution operation.

2. Linear Interpolation Between Positions:
   Since moving sound sources transition smoothly between discrete positions, interpolation between the convolution results of adjacent positions is necessary. We introduce a linear interpolation weight $\alpha(t)$, which indicates the degree of transition between positions $j$ and $j+1$. The weight $\alpha(t)$ ranges from $0$ (fully at position $j$) to $1$ (fully at position $j+1$). The interpolation weights are calculated based on the Euclidean distance between the moving source's current position and the neighboring positions:

$$\alpha(t) = \frac{d _ j - \text{dist}(\mathbf{r} _ j, \mathbf{r} _ t)}{d_j - d _ \{j+1\}}$$

where $d_j$ and $d_{j+1}$ are the spatial distances between positions $j$ and $j+1$, and $\text{dist}(\mathbf{r}_j, \mathbf{r}_t)$ is the distance between the receiver's current position $\mathbf{r}_t$ and position $j$.

3. Weighted Average of Convolution Results:
   At each time step $t$, the interpolated audio signal is computed as the weighted average of the convolution results at adjacent positions:

$$\mathbf{y}(t) = (1 - \alpha(t)) \cdot \mathbf{y} _ {\mathbf{i}(t)}^c(t) + \alpha(t) \cdot \mathbf{y} _ {\mathbf{i}(t)+1}^c(t),$$

where $\mathbf{i}(t)$ denotes the position index at time $t$, $\mathbf{y} _ {\mathbf{i}(t)}^c(t)$ and $\mathbf{y} _ {\mathbf{i}(t)+1}^c(t)$ are the convolution results at positions $j$ and $j+1$, respectively, and $\alpha(t)$ is the interpolation weight.

This approach ensures the synthesis of audio signals that reflect the smooth spatial movement of the sound source, providing a realistic simulation of moving sound sources.

Q3: Is the speed of the moving source a factor in the whole simulation process ?

A3: Thank you very much for your insightful question. We have included more details about speed in the revised manuscript. In SonicSet, we need to keep the audio length at a fixed length of 60s to facilitate our testing. In the simulation process, SonicSim adjusts the speed of movement based on the trajectory length to ensure compatibility with the fixed audio duration. For instance, the movement speed for a 10m-long trajectory is faster than that for a 5m-long trajectory. Since we randomly sample different 3D scenes during the simulation, the movement speed varies randomly, reflecting the diversity of real-world scenarios.

Q1: SonicSim is constrained by the level of detail in 3D scene modeling.

A1: We agree with your observation, and we have discussed this issue in the limitations section. The challenge arises because low-fidelity 3D scene modeling often introduces significant voids or gaps, which prevent rays in the audio simulation process from undergoing reflections, refractions, or diffraction. This, in turn, leads to inaccuracies in the simulated RIR. One potential solution is to smooth and refine the coarse 3D scene, thereby filling the voids in the mesh. This approach can mitigate the impact of this issue and improve the accuracy of the simulation.

Q2: For evaluation a subjective rating could have been used as done in DNS Challenge 2023 which would be more convincing than the objective metrics used

A2: In this study, we employed evaluation metrics commonly used in speech separation and enhancement research. These objective metrics provide a reliable reflection of model performance. While human evaluations can offer insights from a broader perspective, conducting them for our benchmarks would be prohibitively costly and time-intensive. Specifically, our evaluation involves 19 baseline models (spanning both speech separation and enhancement tasks) under two noise types. Assuming a test set comprising 40 samples of 60 seconds each and evaluations from 100 individuals, the process would require approximately 760 hours, which is beyond our feasible resources in terms of both time and cost.

To address this limitation, we have supplemented our evaluations with DNSMOS and NISQA to assess differences among speech separation baseline models (see Tables 2-4 and 10 in the paper). For the enhancement task, we have included NISQA to evaluate performance differences among speech enhancement baselines (see Tables 5,6 and 11 in the paper). These additions ensure comprehensive and practical evaluations within our constraints.

Q3: What is the 5-hour dataset described in C.5 called (please give it a name - it is not SonicSet).

A3: We have named this dataset RealSEP in the revised manuscript to distinguish it from SonicSet.