High Fidelity Text-Guided Music Editing via Single-Stage Flow Matching

1. Although the paper claims to introduce the flow matching model for text-to-music generation, it is apparent that its performance on subjective metrics does not match that of Stable Audio. Furthermore, the authors have not provided an evaluation of Stable Audio's objective metrics.

2. Need a more comprehensive literature review. Important editing methods like DITTO or MEDIC are not in the literature review. Also need to compare with MusicMagus in zero-shot editing.

3. This work proposes a new regularized inversion method; however, the primary distinction from ReNoise lies in the alteration of noise prediction to velocity prediction. While the results achieved are commendable, the contribution lacks sufficient novelty.

4. The authors assert that their method retains similarity to the source audio; however, it obtains a lower score on the CON metric in subjective evaluations compared to AudioLDM2-Music. Moreover, the demonstrated demo reveals inadequate preservation of essential content, raising concerns about the validity of their claim regarding the maintenance of the original audio's critical elements.

5. The paper claims capability of handling 30 seconds of music, yet it predominantly conducts comparative experiments using MusicCaps, which features a duration of only 10 seconds. It is advisable for the authors to include comparisons with longer music datasets, such as the Song Describer Dataset.

6. The authors have not yet made their code and model publicly available, which should not be presented as a contribution point in the paper.

• Components of this work is a combination from existing works. Its novelty lies within the modification to the regularization method of ReNoise, but it lacks theoretical support/analysis on the effectiveness of such modification. It should be worth to have more discussion and analysis on, for example, why removing L_pair improves the result, whether there's a sweetspot for lambda_pair, and what's the side effect of removing L_pair. Empirically, it might be helpful to have a grid search based on both lambda_pair and lambda_KL, and measure the KL-divergence between the latents from different approaches. Visualizing the distribution of the latents with low dimensional projection such as t-SNE or UMAP could also be interesting.

• The quality evaluation could not show whether the proposed model is qualified to be called as "high-fidelity". Although the the FAD score is the lowest for the proposed model, but FAD_vgg cannot evaluate the content beyond 16k due to the limitation of the VGG model. The experiment included MusicCaps as the reference set in the subjective evaluation, but the audio of MusicCaps is a subset of AudioSet, which is obtained from Youtube video, thus unlikely to be in high-fidelity format(uncompressed 44.1 or 48 khz). Moreover, the subjective test result on overall quality does not favor proposed work (around the same as MusicGen-medium, but not at the same level of StableAudio). One way to justify this is to compare the generated samples of the proposed model with another reference set which is natively 48khz. A held out set of the internal 48khz dataset could serve this purpose.

• Line (046-047) “editing methods from the computer vision domain, which are exclusive to diffusion models (Novack et al., 2024; Zhang et al., 2024; Manor & Michaeli, 2024)” is not wholly true. Optimization methods like Novack et al., 2024 are agnostic to the sampling process (and in fact, the flow-matching equivalent has already been explored [1]), and guidance methods like Zhang et al., 2024 are also agnostic to sampler (as nothing specific about their method requires DDIM inversion as the inversion method).
• In general, Section 2.4 is not clear enough. It is hard to tell which parts of the the algorithm from ReNoise are being kept and which are novel, and in general as there is very little textual explanation of the method (and only consistent referrals back to the ReNoise paper), it is hard to tell where the novelty lies. As this is a claimed core contribution of the paper, more time should be spent in section 2.4 to make the differences between the baseline and their method explicitly clear, and to explain the algorithm in depth textually. Along with this, Figure 1 is very hard to parse, and a more semantic diagram (rather than ground truth control flow) may be more helpful.
• 2.5.2 is also very much understated, and is hard to parse without an intimate knowledge of the references. As the paper uses this method to show its increased performance, it should be made explicit in the paper how this works and what the goal of the method is (e.g. is this a training or inference time change? Why do we want to run the Hungarian algorithm?)
• I wonder at a high level that the overall framing (and title) of the paper is somewhat misplaced. On line 047, it is stated that “Despite recent efforts, no approach has yet shown the ability to perform high-fidelity generic style transfer across various music editing tasks,” which I think is the right framing for the inversion task used here, as the example samples show much more strengths in “stylistic” changes rather than grounded content changes (i.e. they very much sound “inspired” by the original samples rather than editing them). This is a fine approach for the paper’s framing, but this is the only mention of style in the entire paper, and thus much of the paper sounds like its talking about some other, more grounded form of editing.
• While previous works have used it, it is publicly documented that the VGGish backbone for FAD is not correlated with human perception [2], and thus some other backbone (like CLAP) should be used. This shouldn’t be too much of an issue, as all the models tested here are open-source (with the exception of Stable Audio, but Stable Audio Open exists) to run directly.
• Overall, the technical results are somewhat weak. While it is mentioned that in the limitations the goal is music editing and not to outperform other TTM models (line 511), this is at odds with their first main stated contribution of making an improved TTM generation model in general (line 097). The results against existing baselines show no clear strengths other than moderate latency speed ups. It also seems like between the baseline ReNoise and the proposed method, the differences are small.

As a whole, the paper seems to be caught somewhere between being a modeling paper (in its proposal of ways to improve training of FM-based TTM models) and a music editing paper (through its FM inversion method), weakening it in both respects. The strongest modeling results are only with respect to an internal non-TTM baseline and do not seem to get comparable results with current SOTA methods, and a full ablation of FM TTM modeling is given to the space dedicated to the editing results. And conversely, without the modeling contributions, it seems like the core contribution then rests solely on the FM inversion technique, in which its novelty is hard to assess. I do not think the present version of the paper is ready for acceptance, though I think changes to spending more time on the editing results (as those seem comparatively stronger than the modeling changes) as the main contribution and overall framing would improve the paper substantially.

[1] Ben-Hamu, Heli et al. “D-Flow: Differentiating through Flows for Controlled Generation.” ArXiv abs/2402.14017 (2024): n. pag.

[2] Gui, Azalea et al. “Adapting Frechet Audio Distance for Generative Music Evaluation.” ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2023): 1331-1335.

Overall:

1. Music generation part:

• The applicability of FM-based generative models for audio data is already explored in some prior work (for example, Text-to-Music Generation [1] and Text-to-Audio [2]). Therefore, simply applying FM-based model to text-to-music generation task (even 'single-stage') does not contribute significantly to new insights for the audio/music community.

  • Therefore, additional novel technical contributions, such as sophisticated ideas to surpass the sample quality of existing text-to-music models (regardless of whether the FM-based or diffusion-based models) or techniques that enable more efficient generation while maintaining sample quality, along with sufficient experimental evaluation to demonstrate their effectiveness, should be explored.

• Considering of above, the authors' efforts to improve generation performance through architectural and training enhancements (as discussed in Section 2.5, Appendix A.2, and Table 4) could still provide some valuable insights. However, even from this perspective, both objective and subjective metrics in Table 2 suggest that, despite incorporating various techniques, the main advantage of the FM-based models lies in halving the number of sampling steps compared to diffusion-based models like Stable Audio. Thus, the contributions offered from this angle remain limited.

2. Music-editing part:

• From the perspective of ODE-based generative models, score-based and FM-based generative models share a broadly similar framework. Thus, the fundamental contribution of reformulating DDIM inversion or its extensions, such as Pix2pix-zero/ReNoise, to apply to FM-based models is not particularly significant. (Of course, I acknowledge the practical differences, so please note that I am not evaluating the proposed editing method solely from this perspective.)
• Regarding the proposed inversion technique, the experimental results do not seem to fully support its effectiveness. For example, in Table 1, considering the inherent challenges of subjective evaluation in music editing tasks and the CI95, the advantage over DDPM inversion in terms of 'AVG.' does not appear sufficiently clear. (To clarify, I am indicating that the results are not clear enough; I am not suggesting that the proposed method is ineffective. For more details on objective metrics, please refer to Details below and Questions 1~3.)

Details:

The objective metrics used for music editing and generation leave some uncertainty regarding their adequacy in validating the effectiveness of the proposed methods.

• Recent studies, such as [3], suggest that using FAD with VGG-ish to evaluate music samples is inappropriate. To better support the authors' claims in this work, I recommend using alternative pretrained models as introduced in [3] (for both editing and generation tasks).
  • For example, based on the correlation between MOS and FAD discussed in [3], it would be more appropriate to use FAD with LAION-CLAP to evaluate musical quality, and to use FAD with DAC/EnCodec embeddings to assess acoustic quality (please see more detail in [3]).
• For LPAPS, if the authors used the pretrained network from this repository [4], it was trained on the VGGSound dataset [5] (also, please review its data filtering process). This raises concerns about its validity for numerical evaluation in music editing. Additionally, the checkpoint provided in that repository [4] has other issues. As noted in this issue [6], the authors of this repository acknowledge a problem in the LPAPS training procedure itself and do not recommend using LPAPS for at least training purposes.
  • It would be appropriate to calculate the L2 distance using other audio encoders trained properly. For instance, as in [3], I recommend calculating the L2 distance based on embeddings from audio encoders like LAION-CLAP, DAC, or EnCodec.
• Considering both these points and the subjective evaluation results, the current experimental results do not seem sufficient to demonstrate the effectiveness of the proposed latent KL regularization, which contributes to the originality of this method. I recommend revisiting the evaluation protocol to provide stronger support for contributions of this paper. (I do acknowledge that the authors have demonstrated that FM-based music generative models can achieve music editing through inversion techniques to some extent.)
  • For instance, conducting additional evaluations using the objective metrics mentioned above, as well as a subjective evaluation for an ablation study on the necessity of latent KL regularization in MelodyFlow’s inversion ($\lambda_{KL}=0$ or not), would support the contribution much more in terms of the proposed editing method if their effectiveness is sufficiently demonstrated.

[1] Tal, O., Ziv, A., Gat, I., Kreuk, F., Adi, Y., 2024 Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation. The 25th International Society for Music Information Retrieval (ISMIR) Conference.

[2] Vyas, A., Shi, B., Le, M., Tjandra, A., Wu, Y.C., Guo, B., Zhang, J., Zhang, X., Adkins, R., Ngan, W. and Wang, J., 2023. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821.

[3] Gui, A., Gamper, H., Braun, S. and Emmanouilidou, D., 2024, April. Adapting frechet audio distance for generative music evaluation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1331-1335). IEEE. Here is toolkit https://github.com/microsoft/fadtk.

[4] https://github.com/v-iashin/SpecVQGAN

[5] Chen, H., Xie, W., Vedaldi, A. and Zisserman, A., 2020, May. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 721-725). IEEE.

[6] https://github.com/v-iashin/SpecVQGAN/issues/13

• Questionable quantitative results: In Table 1, the proposed method shows marginal improvement in relevance to the editing prompt, but it falls behind in terms of consistency. Additionally, a better overall sound quality does not necessarily indicate improved editing capability. Overall, the metrics presented regarding the editting are not convincing.

• Poor qualitative results: The provided samples in the supp. do not demonstrate fine-grained editing capabilities, as they fail to showcase cases where only one or two words are changed in the prompts. This makes it difficult to assess the quality of the editing. Personally, I found the samples to lack consistency with the source music clips.

• Limited novelty: Although the authors offer valuable insights into training audio FM models, the proposed editing method is primarily an adaptation of the ReNoise technique, which limits the work’s technical novelty.