MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction

We sincerely thank Reviewer cGKq for your careful reading and constructive comments. Below, we address your main concerns:

"Unfortunately, I’m not convinced by the results. The empirical study using VGGish FAD is poor, as multiple previous works have shown that VGGish is a suboptimal embedding model for music [1-3]."

Authors' Response: We appreciate your feedback and agree that VGGish-based FAD is not the ideal metric for music generation. We initially adopted it for its widespread use as in SingSong [4] and MSDM [5], but we acknowledge its limitations. We additionally calculated the FAD using CLAP-MA embeddings (LAION-CLAP checkpoint music_audioset_epoch_15_esc_90.14), which Grötschla et al. [2] have shown best correlation with human perception.

Below, Table A reports the total-generation results using FAD-CLAP-MA.

---

Table A: Total generation result with FAD-CLAP-MA $\downarrow$

---

It shows a similar relative ranking to Table 1 of the main paper, with lower absolute values. One notable difference is that our $T_1$ model achieves the lowest FAD-CLAP-MA score compared to the baselines, which is a clear contrast to the VGGish-based results in Table 1 of the main paper.

We also computed the FAD-CLAP-MA for partial generation task, where the results are shown below in Table B.

---

Table B: Partial generation result with sub-FAD-CLAP-MA $\downarrow$

---

---

---

---

The overall relative pattern of the Table B closely matches Table 2 of the main text, with MSG-LD achieving the best scores on most stem combinations. We believe these additional metrics offer a more reliable assessment of partial generation performance.

We also agree that FAD alone cannot capture musical coherence, so we also evaluated the COCOLA score [6] for partial generation (please refer to the table and discussion in our response to Reviewer W8on).

"Additionally, while I appreciate the authors having trained various models (T1-4), I don’t see how it makes sense to compare these against a baseline only trained on a single subset of the data."

Authors' Response: You are correct that only the baselines (MSDM and MSG-LD) and our $T_1$, each trained on the same $S_A$ set, are directly comparable. We included $T_2$~$T_4$ to illustrate how training on larger and more diverse datasets affects performance, since dataset-agnostic scaling is rarely evaluated in prior work and no existing baselines cover these settings. We hope these results will serve as useful reference points for future research.

For additional comparability, we also trained Hybrid Demucs on the $S_A$ set. The corresponding results are shown in Table C below to strenghten the baseline comparison.

---

Table C: Source extraction results with Hybrid Demucs. Column headers indicate the target stem and its corresponding set.

---

"The biggest issue I have is the missing human evaluation, as the FAD numbers don’t show the full picture."

Authors' Response: We agree that human evaluation is crucial for assessing generated music quality. Accordingly, we conducted a listening study on a total generation task, and the results are shown in Table D below.

---

Table D: Mean Opinion Score (MOS) for the total‐generation task, rated from 1 (bad) to 5 (excellent). Two aspects were evaluated: Quality (how much the snippet sounds like noise versus a real song) and Coherence (consistency across generated stems). A total of 24 subjects participated, each listening to 8 segments per model. Scores are reported with 95% confidence intervals.

---

The results indicate that $T_1$ is slightly less preferred than MSG-LD, whereas $T_2$~$T_4$ receive higher preference ratings, demonstrating that expanding the training dataset can benefit the perceptual quality of generated music.

Q:"Why are the authors only training on 16 kHz? As far as I know the Stable Audio pipeline supports up to 44.1 kHz stereo."

Authors’ Response: We initially chose 16 kHz mono to match the baseline models and ensure a fair comparison. We agree that 16 kHz mono is not sufficient for high-fidelity music generation (as noted in our Limitations section). Having demonstrated the validity of our framework at 16 kHz, we plan to extend MGE-LDM to 44.1 kHz stereo in future work.

Q:"Why are the authors using an outdated version of Demucs? The Hybrid Demucs is significantly better compared to the original."

Authors’ Response: Thank you for pointing this out. We have attached results for Hybrid Demucs (HDemucs) in Table C above. All models were trained to convergence. In our experiments, HDemucs results have not shown significant improvements over standard Demucs, so we initially reported the original Demucs comparison in the main paper, consistent with the MSDM paper. However, we agree that including HDemucs in the main tables is more appropriate and will update the manuscript accordingly.

Q:"Why does the demo page only have comparisons in the Slakh section but not for Moises/MusDB? Especially for the source extraction, which can be easily compared to existing methods."

Authors’ Response: Thank you for your feedback regarding the demo page. On the “Slakh Full” demo page, we initially showcased only $T_4$ results, assuming that adding other references would not be informative given the diversity of instruments. However, we agree that this limits comparability. Accordingly, we have now added Demucs inference examples alongside our model under “Source Extraction (Full) / MUSDB18 & Moises". We hope this makes the demo more informative and directly comparable.

"The authors have a section on limitations in the appendix. I suggest they move this to the main text." Thank you for the suggestion. We will move the Limitations section into the main text to ensure it is clearly visible to readers.

We appreciate your valuable feedback and will incorporate these changes in the final manuscript to enhance its clarity and rigor.

References

[1] Gui et al. "Adapting frechet audio distance for generative music evaluation." ICASSP 2024

[2] Grötschla et al. "Benchmarking Music Generation Models and Metrics via Human Preference Studies." ICASSP 2025

[3] Huang et al. "Aligning Text-to-Music Evaluation with Human Preferences." arXiv:2503.16669, 2025.

[4] Donahue et al. "Singsong: Generating musical accompaniments from singing." arXiv:2301.12662, 2023.

[5] Mariani et al. "Multi-source diffusion models for simultaneous music generation and separation." ICLR 2024

[6] Ciranni et al. "Cocola: Coherence-oriented contrastive learning of musical audio representations." ICASSP 2025.

The reviewer should re-consider their evaluation, especially after additional quantitative evaluation on COCOLA score, which complements the sub-FAD evaluation (and offers a more important metric for capturing coherence on the partial generation task). Also, using VGGish embeddings for sub-FAD is not really poor. Practically all the field of accompaniment generation (SingSong [1], MSDM [2], GMSDI [3], COCOLA [4], MSG-LD [5], JEN-1 Composer [6], STAGE [7]) uses VGGish embeddings for sub-FAD!. Also with the authors providing metrics with CLAP embeddings and giving a more detailed picture, the current experimental results are more than sufficient.

References

• [1] Donahue, Chris, et al. "Singsong: Generating musical accompaniments from singing." arXiv preprint arXiv:2301.12662 (2023).
• [2] Mariani, Giorgio, et al. "Multi-Source Diffusion Models for Simultaneous Music Generation and Separation." The Twelfth International Conference on Learning Representations.
• [3] Postolache, Emilian, et al. "Generalized multi-source inference for text conditioned music diffusion models." ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.
• [4] Ciranni, Ruben, et al. "Cocola: Coherence-oriented contrastive learning of musical audio representations." ICASSP 2025.
• [5] Karchkhadze, Tornike, Mohammad Rasool Izadi, and Shlomo Dubnov. "Simultaneous music separation and generation using multi-track latent diffusion models." ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.
• [6] Yao, Yao, et al. "Jen-1 composer: A unified framework for high-fidelity multi-track music generation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 13. 2025.
• [7] Strano, Giorgio, et al. "STAGE: Stemmed Accompaniment Generation through Prefix-Based Conditioning." arXiv preprint arXiv:2504.05690 (2025).

Dear reviewer W8on,

The provided experimental results are indeed more than sufficient, I have not said anything to the contrary. However, from how I see it, the results don't show outperformance of prior work to warrant acceptance. The results from COCOLA look promising, but the rebuttal is not a place to meticulously inspect and re-evaluate new results. For example. how can we be sure there was no inadvertent information leakage, as COCOLA trained on many of the same datasets that were also used in this work. All the provided results together give an impression of not significantly better than previous work, especially the human evaluation which is the gold-standard metric. Therefore, I politely disagree with your assessment and maintain my original score.

On the matter of VGGish. SingSong started with VGGish at a time when this drawback was not known yet, and then every other paper followed by reusing it. Does this mean it's a good embedding model? I would argue not. Especially since there is now ample evidence to support that VGGish is a poor choice for music models.

We can gladly continue this discussion during the AC-reviewer phase.

We sincerely thank Reviewer W8on for the through and insightful feedback. We appreciate your recognition of our contritbutions to multi-track music generation, and we address your comments below.

"A limitation of the paper is that partial generation metrics are computed on sub-FAD, a metric which, while having served well as an initial foray in evaluating accompaniments models (e..g, in SingSong, in MSDM, etc.), there is empirical evidence (Ciranni et al. 2025) that it is not optimal for such a task, favouring randomly mixed stems over natural combinations. The authors should also compute metrics such as COCOLA score (Ciranni et al. 2025), which are better suited for the task. Also in that paper one finds a benchmark on a conditional model based on Stable Audio Open, released publicly, and trained on MusDB and MoisesDB, which shares the diffusion architecture of the presented paper: it would be interesting to see how the presented joint model compares to the conditional one."

Authors' Response: You're correct that sub-FAD alone can bias toward random mixtures rather than musical coherence. In response to your suggestion, we computed the COCOLA score [1] on exactly the same partial-generation outputs reported in Table2 of the main paper. The results are shown below, with the highest COCOLA score per column bolded, and among models trained on the $\mathbf{S}_A$ set (MSDM, MSG-LD, or our T1), the best result underlined.

---

---

---

---

---

The results show that our MGE-LDM variants achieve the highest coherence across nearly all stem combination compared to the baselines (MSDM and MSG-LD), often matching or even approaching the ground-truth levels.

Because COCOLA emphasizes musical coherence rather than timbral accuracy to reference stems, we believe it complements sub-FAD and provides a more balanced view of partial generation quality. We will integrate these COCOLA results either alongside sub-FAD in the main manuscript or in the Appendix, as you suggested.

(For reference, the COCOLA score (Harmonic) of stable audio + control net reported by the COCOLA paper [1] is 57.34, though it is not directly comparable to our results since it uses a different evaluation set.)

Once again, thank you for this valuable suggestion.

"While the authors acknowledge that the nonlinearity of the latent space makes it difficult to constrain the sources in a mixture and they opt out for their approach, I believe that the three entries of a triple are not necessarily constrained to sum well in the data (waveform) domain. This could have a negative impact in the performance of the model. Thus, finding the best way to solve compositional music generation in the latent domain it still remains an open problem, and maybe a more constrained approach would be given, for example, by learning latent-domain likelihood functions like in (Postolache et al. 2023). The authors could discuss this in the concluding remarks of the paper or in limitations."

Authors' Response: You rightly point out that, although we train on triplets satisfying $mix = submix + source$ in waveform space, our latent-diffusion model does not enforce an explicit additivity constraint for generated triplets. This issue is critical, as we believe it directly related to the “hallucination” phenomenon, in which the model extracts sources that aren’t actually present in the mixture. Postolache et al. [2] offer an elegant solution by enforcing additivity in a discrete VQ-VAE latent space: they estimate the joint likelihood of two sources by counting exact codebook occurences (bin-counting), effectively modeling $p(z_{mix}|z_{src_1}, z_{src_2})$, where $z$ are quantized latent codes. This bin-counting approach serves as a powerful regularizer to enforce a summation constraint in the latent domain. Unfortunately, our current pipeline relies on a continuous latent space, which precludes direct application of discrete bin counts.

Nonetheless, we believe adapting this latent-domain likelihood approach to our model would be highly promising. We could either extend it to a continuous latent space by designing appropriate regularizers, or convert our encoder to a discrete representation, such as using VQ-VAE based encoder and discrete diffusion (or MaskGIT) for generation to enable bin counting in the latent space.

We thank the reviewer for this valuable suggestion, and we will discuss these directions in the Limitations and Future Work sections as promising approaches to reduce hallucinations and improve compositional consistency.

We also appreciate your helpful line-by-line suggestions and will update the manuscript accordingly. In particular, your idea of negative conditioning to supress unwanted tracks is very valuable and will be explored in our future work.

References:

[1] Ciranni, Ruben, et al. "Cocola: Coherence-oriented contrastive learning of musical audio representations." ICASSP 2025.

[2] Postolache, Emilian, et al. "Latent autoregressive source separation." AAAI 2023.

We thank Reviewer ShCW for the thoughtful and constructive feedback and are delighted that you recognize our work as a timely contribution to the field. Below, we address each of your points:

"The paper is generally well-organized, though some sections (e.g. the method) can be dense and may benefit from additional high-level intuition or summarization. / while the method is mathematically rigorous, its clarity suffers from heavy notation and long explanations without always grounding the reader in intuitive takeaways. The presentation could be tightened to emphasize conceptual contributions before diving into formalism."

Authors' response: We agree that the Method section can feel dense. To make it more accessible, we will revise the method section to include a conscise, high-level overview outlining the key ideas in plain language before describing mathematical details.

"Q: Can you quantify or mitigate the limitations of CLAP-based conditioning? CLAP is conceptually appealing for language-guided extraction, but its granularity and semantic specificity are limited. Do you observe consistent issues with ambiguous or overlapping prompts (e.g., "strings" vs. "synth pad")? Have you explored alternative or complementary embeddings, or prompt disambiguation strategies?"

Authors' response: You’re correct that CLAP’s semantic granularity can lead to ambiguity. To illustrate, we ran a small experiment on mixtures containing strings or synth‑pad and measured Log‑Mel L1 distances between the true isolated stems and the model’s output under different text prompts:

In an ideal setting, the lowest errors would lie on the diagonal, but here we see that asking for “synth pad” on a mixture with strings yields a lower L1 distance (2.25) than asking for “strings” (2.57). Listening confirms that the model often reproduces the correct pitch contour but with a synth‑like timbre, which is a hallucinatory behavior we have noted in our Limitations section. This highlights an important direction for future work in improving prompt‑based source separation (which we also discuss with the reviewer W8on).

We also intend to explore more recent embedding models such as M2D‑CLAP [1] to enhance semantic precision, investigate negative prompting strategies (e.g. “strings, not synth pad”) to explicitly suppress unwanted classes. We also can try leverage CLAP’s audio‑branch embeddings from short reference clips to anchor the model and reduce ambiguity. Although we have not yet quantified these ideas, we believe they could meaningfully improve disambiguation.

"Q: What are the primary failure modes, and how often do they occur?"

Authors' response: Empirically, we observe three primary failure modes. First, the model sometimes fails to extract a source and outputs near‑silence instead of the intended stem. Second, with ambiguous prompts the model often hallucinates sounds that aren’t present in the mixture, as discussed above. This issue seldom arises for clear prompts like “the sound of vocals,” where the model correctly outputs silence when vocals are absent. Third, the model struggles to capture the fine-grained detail of isolated single notes on piano and guitar: while chords and percussive strokes are generally well reproduced, individual notes often vanish or become blurred. We believe these issues stem from a relative scarcity of fine-detail, single-source examples in our training data. To address them, we plan to augment our dataset with additional isolated‑note samples and experiment with increased loss weighting on fine‑detail reconstruction.

Reference

[1] Niizumi, D., et al. “M2D‑CLAP: Masked modeling duo meets CLAP for learning general‑purpose audio‑language representations.” Interspeech 2024.