BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models

We thank the reviewer for your constructive comments. We will address your concerns below and will revise our paper following your suggestions.

Performance on the public dataset (Claims And Evidence)

We agree that the proposed method performs comparably to the baseline (BinauralGrad) on some metrics, but we respectfully clarify that our approach outperforms the baseline on key perceptual metrics (PESQ and MRSTFT). These percetual metrics are often prioritized in audio synthesis tasks as they correlate better with human perception of quality and intelligibility, whereas minor waveform-level differences (L2) may not impact perceived audio quality.

Implementation details of baselines (Claims And Evidence)

For all baselines, we trained the models from scratch on our new dataset and tested them on the same dataset. We did not perform pretraining or cross-dataset evaluation. We will include additional implementation details in our revision.

Model size and speed (Claims And Evidence)

We thank the reviewer for pointing out this issue. We present the model size and inference speed for all baselines below. We test the inference speed on a single 4090 GPU. The audio sampling rate is 48 kHz, and the audio length is 683 ms. As shown in the table, our model achieves the fastest inference speed among generative models. Our model acheives a more favorable trade-off between performance (Table 1) and inference speed compared to the baseline approaches.

Method motivation (Methods And Evaluation Criteria, Other Strengths And Weaknesses)

Real-time, high-quality spatial audio is essential for immersive applications such as gaming, VR/AR, and cinematic experiences. These applications motivate the design of a high-quality, efficient, and streaming binaural audio generation framework.

For quality, we frame the binaural audio rendering task as a generative problem and introduce a high-fidelity generative model. To improve inference efficiency, we choose flow matching models over diffusion models because they render high-fidelity spatial audio with fewer inference steps. We further reduce inference steps by using the midpoint solver and an early skip strategy. To satisfy the streaming requirement of real-world applications, we design our causal U-Net architecture and continuous inference pipeline that processes audio input in chunks. Our method is designed to meet real-world needs, and our model successfully addresses these challenges.

Use of SGMSE baseline (Experimental Designs Or Analyses)

Although SGMSE is not intended for binaural audio rendering, we included it as a baseline due to its U-Net-based diffusion model. While BinauralGrad also uses a diffusion model, it employs a WaveNet architecture. We used SGMSE to examine the impact of different architectures on the results.

Omission of SGMSE results in Table 4 (Experimental Designs Or Analyses)

Since SGMSE was not evaluated on the public dataset, it was not included in Table 4. To expand our experiment, we test SGMSE on the public dataset and report its performance below. Our model outperforms SGMSE with noticeable margins. We will include these results in the revision.

L2 error scale (Experimental Designs Or Analyses)

The difference in scale is due to the audio volume in the two datasets not being the same. Since the audio volume in our collected dataset is lower than in the public dataset, we used a different scaling factor during testing.

Difference between BinauralFlow and simplified flow (Questions For Authors)

The key difference between our approach and the simplified flow lies in the flow function and vector field. Our flow function is defined as $\phi_t(\mathbf{z}) = t\mathbf{y} + (1-t)\mathbf{x} + (1-t)\sigma \mathbf{\epsilon}$, where $\mathbf{x}$ and $\mathbf{y}$ are mono and binaural audio, respectively, $\mathbf{\epsilon}$ is Gaussian noise, $t$ is the time step, and $\sigma = 0.5$ in our experiments. The simplified flow uses $\phi_t(\mathbf{z}) = t\mathbf{y} + (1-t)\mathbf{x} + \sigma \mathbf{\epsilon}$ with $\sigma$ near zero (e.g., $1e{-}4$), reducing to a deterministic interpolation $\phi_t(\mathbf{z}) = t\mathbf{y} + (1-t)\mathbf{x}$. Consequently, our vector field $\mathbf{y}-\mathbf{x}-\sigma\mathbf{\epsilon}$ introduces stochasticity, enabling variability in generative tasks, unlike the deterministic $\mathbf{y}-\mathbf{x}$ in the simplified flow.

We appreciate your positive feedback on our work. We will address your questions in the responses below.

Related work (Claims And Evidence)

We thank the reviewer for pointing out these related works. PeriodWave designs a multi-period flow matching model for high-fidelity waveform generation. FlowDec introduces a conditional flow matching-based audio codec to noticeably reduce the postfiler DNN evaluations from 60 to 6. RFWave proposes a multi-band rectified flow approach to reconstruct high-fidelity audio waveforms. These works are all related to flow matching models and show the effectiveness in generating high-quality waveform signals. We will include our discussion in the revised paper.

Real-time factor (Claims And Evidence, Methods And Evaluation Criteria)

We calculate the real-time factor of our model for different numbers of function evaluations on a single 4090 GPU. The audio sampling rate is 48 kHz, and the audio length is 0.683 seconds. As shown in the table, when NFE is set to 6, the real-time factor is 0.239. If we sacrifice some performance for faster inference, setting NFE to 1 results in an RTF of 0.04. Our model demonstrates potential for real-time streaming generation.

Discussion on Matcha-TTS and CosyVoice 2 (Claims And Evidence)

We carefully reviewed the papers and code of these two works. Matcha-TTS employs a 1D U-Net model with 1D ResNet layers and Transformer Encoder layers. Neither the ResNet layers nor the Transformer Encoder layers are causal, which means that Matcha-TTS does not achieve time causality or support streaming inference. In contrast, our model is fully causal and supports streaming inference.

CosyVoice 2 introduces a chunk-aware causal flow matching model that uses causal convolution layers and attention masks to enable causality. However, the CosyVoice 2 model does not include feature buffers for each causal convolution layer, which may result in audio interruptions and discontinuities during streaming inference in real-world scenarios.

We will include our discussion of Matcha-TTS and CosyVoice 2 in our revision.

Experiment of parallel model (Claims And Evidence, Methods And Evaluation Criteria)

We compare the model's performance with and without buffers to examine the effectiveness of our causal model design. The results show that BinauralFlow with buffers achieves higher quality than the model without buffers. Additionally, in Figure 7 in the main paper, we show that excluding buffers causes noticeable artifacts in the generated spectrograms.

Sway Sampling (Claims And Evidence)

As suggested by the reviewer, we use Sway Sampling with different coefficients ranging from -1 to 1 to systematically evaluate our model. The results are shown in the table below. Changing the coefficients does not lead to significant changes in the quantitative results. However, we observe that setting coefficients greater than 0, which shifts the time steps to the second half, results in better qualitative outcomes. Specifically, background noise becomes more realistic when the coefficient is increased. These results support the rationale behind our early skip strategy.

Sound dataset (Experimental Designs Or Analyses)

Thank you for the suggestion. We plan to conduct this experiment in our future work.

Demo (Supplementary Material)

As recommended by the reviewer, we have created a demo page and will release it following the acceptance of our paper.

GAN-based models (Questions For Authors)

Utilizing GAN-based models, such as PeriodWave-Turbo, to enhance inference efficiency is a promising direction. We plan to discuss this possibility in our paper and explore it further in future work.

Reshape and linear projection (Questions For Authors)

We tried using waveform directly without STFT/iSTFT but it did not lead to superior performance. We are still interested in waveform-based approaches. We will explore the reshape method and linear projection proposed in WaveNeXt and discuss them in our paper.

We appreciate your positive feedback on our work. We will address your concerns and include your suggestions in the revision.

Dataset (Questions For Authors)

Regarding open-sourcing the dataset, we fully understand the importance of reproducibility and transparency in research. However, due to privacy constraints and participant confidentiality, we are unable to publicly release the full dataset. To ensure the reproducibility of our work, we will release all implementation code, training scripts, pretrained model weights, and a test subset that has been carefully curated to exclude any personal information.

Different solvers (Questions For Authors)

Besides the Midpoint solver, we test the Euler and Heun solvers. The Euler solver is a first-order solver and Midpoint and Heun solvers are second-order. We set the number of function evaluations (NFE) to 6 and present the results below. Although the Euler solver yields lower error values than the Midpoint solver, it fails to generate realistic background noise. Setting NFE to 6 is insufficient for the Heun solver, which requires 30 steps to achieve comparable error values. In conclusion, the Midpoint solver provides the best trade-off between error values, qualitative results, and inference efficiency.

STFT-based complex spectrograms (Questions For Authors)

A mel spectrogram is derived by mapping the STFT-based complex spectrogram to the mel scale. In this process, only the magnitude is retained, while the phase is discarded. Spatial audio rendering relies on precise phase information to capture interaural time differences (ITD) and interaural phase differences (IPD). These phase cues are essential for accurately conveying spatial positioning in binaural audio, ensuring a realistic 3D auditory experience. Therefore, we use STFT-based complex spectrograms, which retain both magnitude and phase information.

Latency (Questions For Authors)

We test the inference latency of our streaming model using a single 4090 GPU. The audio sampling rate is 48 kHz and the audio length is 0.683 seconds. We vary the NFE from 1 to 10 and report the corresponding inference times below. We also report the real-time factor (RTF), which is calculated by dividing the processing time by the actual time. If RTF is less than 1, the system runs faster than real-time. With NFE set to 6, the real-time factor is 0.239. Reducing NFE to 1 improves inference speed at the cost of some performance, yielding an RTF of 0.040. These results demonstrate our model's potential for real-time streaming generation.

Streaming vs non-streaming models (Questions For Authors)

For a sequence of audio chunks, the streaming model buffers intermediate features to enable seamless streaming inference, while non-streaming model processes each audio chunk independently without buffering. We visualize the spectrograms generated by both the streaming and non-streaming models in Figure 7 of the main paper, where the non-streaming models generate noticeable artifacts between audio chunks. Below, we present a quantitative comparison between the two. The streaming model achieves better audio quality than the non-streaming version, producing smoother and more continuous audio generation.