InfiniteAudio: Infinite-Length Audio Generation with Consistent Acoustic Attributes

1. Insufficient innovation. This paper is more like solving an engineering problem rather than a scientific research problem. The methods used are mostly tricks rather than general methods. For example, if we switch from AudioLDM to Tango/Stable Audio, is the proposed method applicable? Do the parameters need to be further adjusted? This method regards mel-spectrogram as a picture, and borrows methods more from vision, without analyzing the characteristics of the audio itself.
2. The dataset and experimental results of baselines are deviated from the popular SOTA methods. Why does TTA only use 500 audio samples from audiocaps as the testset instead of all of them? At present, the FD of the classic method of audio generation can reach about 20. The FD of the AudioLDM in Table 2 is much worse than the result reported in the original paper. What is the reason?
3. Lack of experimental details: The paper aims to generate long audio. How long is the audio generated in the experiments? An important innovation of the paper is the adjustment of the sampling step length. What is the specific length? For example, what is the parameter P in the appendix A? 4、The main problem is that the generated audio is poor. I listen to some demos and the generated speech has the problem of fluctuating sound energy. This problem is mainly due to the lack of consistency and temporal coherence between the audio segments. The paper does not discuss and test this. The generated long-duration audio has noise and distortion. In the later stage of the generation, the generation collapses. I suspect that this is due to the sampling step size problem. The paper lacks a discussion on this issue.

• Theoretical analysis of proposed method is not shown enough. Especially, theoretical validity of using multiple diffusion steps at the same time.
• Limited applicability. It proposes a way for addressing the issues of VoiceLDM, but not clear if it is also beneficial for other diffusion models and task setting.

Major concerns

1. improper evaluation metrics

Table.3 uses the word error rate (WER) as a metric for the evaluation of audio-speech simultaneous generation (TTAS), and interprets the lower WER scores as "better" result.

The definition of TTAS follows that of VoiceLDM: synthesizing speeches according to not only the content of speech, but also a "descriptive prompt" that specifies gloabl acoustic features.

According to the task definition, when the descriptive promt indicates a noisy environment, "speeches accompanied with very noisy environmental sound" could be the correct generation result, and such a noisy speech may lead to higher WER score. Consequently, a higher WER score in the TTAS part of table.3 does not necessarily indicates a worse generation.

A support to the above observation is that, the CLAP score of the proposed method is worse than all other 3 methods, while its WER is better than others. If the model less reflects "noisy descriptive prompts", WER can be improved, but we cannot conclude such a behavior as a good one.

2. Less contribution to TTS community

As mentioned in line 424 & 425 in the manuscript, the high WER of VoiceLDM (baseline for TTLS and TTAS) is partly because VoiceLDM cannot include all speech contents within 10 seconds. InfiniteAudio further improves the "speaker inconsistency" problem in VoiceLDM. Although there is no explicit explanation on the cause of this inconsistency, I assume it is because of the lack of an explicit speaker embedding in VoiceLDM.

After listening to the demo page, I totally agree that InfiniteAudio greatly improved the above issues of VoiceLDM, but I don't think this improvement is reusable to the broad community, as these improvements are quite limited to the design of VoiceLDM, a model that is very different from modern TTS systems.

1. Bad WER caused by fixed duration generation. Modern TTS systems does not assume a fixed duration of the output, and can predict a proper duration or speech style according to the content text. Go to the demo page of styletts2, the first sample is 1 min 29s long. Modern TTS systems will not miss content text in their generated results.
2. Bad speaker consistency caused by the lack of explicit speaker embedding. InfiniteAudio address this problem by sharing the identica QKV features across the time axis. I think the QKV features implicitly contain the speaker id. So the proposed "sharing QKV" technique is conceptually similar to "introducing a speaker embedding to VoiceLDM", which is the common practice in modern TTS systems.

Although I understand VoiceLDM is the most available TTAS model by now, earlier works such as VALL-E can already generate noisy speeches when a noisy audio prompt is given, so it is unclear to me which one is more valuable, keep improving VoiceLDM or introduce the support of descriptive prompt to successful TTS models. Solving problems that do not exist in other (more common) TTS methods hardly contribute to the broader community.

Minor concerns

1. unclear definition in paper writing.

After reading Sec.4, I thought the term "InfiniteAudio" means the combination of techniques listed in all sub-sections of Sec.4. However, in table.2, table.3 and table.4, we can frequently observe terms like "InfiniteAudio w/ xxx technique". This indicates an inconsistent definition of the term "InfiniteAudio" across different tables, which is confusing.

2. unclear testing condition for baselines and proposed models

"InfiniteAudio" can generate audio of arbitrary duration, hence it is important to clarify the targeted duration within an evaluation.

Unfortunately, the targeted duration is missing for table.2. In the demo page, durations of 10s, 20s, 30s, 1min and 2min are included. If scores in table.2 are computed with mixed durations, I doubt if these scores are meaningful. Mixed duration can significantly affect subjective listening tests.

3. Missing baselines that can be simply implemented

The FIFO (first-in, first-out) strategy is interesting, but the concatenation-based strategy mentioned in line 46 is never evaluated in the paper. I think it is important to compare the FIFO strategy with an overlap-concatenate stategy, as both of them can control memory usage by fixing the input size of the diffusion model.

%%%It is know that AudioLDM can do time inpainting, i.e., given the first 4-sec and some descriptive text, AudioLDM can naturally inpaint the remaining 6-sec. If we combine this inpainting capability with concatenation, we can generately very long audio with a fixed input size.

Overall, the major weakness of this paper is about the clarity. Details of techniques are not well disclosed and the some of the terms in equations are not explained. Another concern is about the generalizability of the timestep optimization. If this is dataset dependent and only a very specific scheduling can yield good result, it'll become an obstacle for people to apply this method in future works.

Please find detailed comments on clarity below:

• The explanation about QKV sharing is insufficient. I would suggest to show the formulas explicitly, or add reference to previous works. I suppose here the proposed work adapted the feature injection in [Tumanyan et al., 2023]. Considering the proposed model shifts 1 frame per timestep, sharing QKV at the first N frame is equivalent to sharing QKV in the first N steps of the backward diffusion, which is a reasonable setup. However, in Appendix D1, there's a case of "QKV sharing on 1 frame". If there's only one frame, it should be the same as "no sharing"?

[Tumanyan et al., 2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proc. CVPR, 2023.

• Regarding the guidance alternation, what's the definition of $\epsilon^{odd}$ and $\epsilon^{even}$? From the text context I could infer that the idea is to remove the effect of $c_{cont}$ from the previous segment, but this is not shown in Eq 4 and 5, in which all the $c_{cont}$ are without indices.

• Regarding the timestep allocation about the curved denoising, the definition of "initial", "middle" and "final" region should be stated clearly. Another concern is, it could be that each dataset requires different schedule. This could make it difficult to apply this strategy in other applications. Moreover, based on those figures in Appendix D3, although the attention map share some common pattern for different prompts, the entire map still varies across prompts. This could mean that the proper partitioning of initial, middle and final region is sample dependent.

• Although with guidance alternation and QKV sharing, the CLAP score of proposed method is still lower than the VoiceLDM baseline on TTAS. Furthermore, adding QKV sharing seems impacted the CLAP score. This could be a hard trade-off on whether to use QKV sharing in practical applications.

• Regarding the claim about the quality of proposed method, in figure 8, it's claimed that proposed work generates "high-quality" long audio. But when I listened to the sample with same prompt on the demo site, I can clearly perceive the artifact. The intelligibility of the sample is acceptable, just that the voice timbre and some background sound are clearly distorted. Moreover, when compared to VoiceLDM in TTAS, the voice quality of VoiceLDM is still clearly better than proposed work. On the other hand, the background sound of TTAS is distorted (e.g. motorcycle engine) for both models (AudioLDM and InfiniteAudio).