SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement

We are grateful to the reviewers for their acknowledgment of the significance of our work and for the thoughtful attention they have given to our manuscript. Here are answers to the questions:

Q1 & Q2: The choice of maximum length is due to the constraint of computing resources. 150s is generally enough to cover both verse and chorus sections in a song. For the ablation studies, we used a smaller model (60 seconds) to optimize training time and resource usage while still demonstrating key performance trends.

Q3: We have trained a new version of SongBloom which receives text descriptions as conditions. We will release this model together with the one used in our paper once the paper is accepted.

Q4: Our evaluation set did not specifically contain cross-lingual pairs, but according to user feedback, the model is able to handle such cases.

Q5: According to previous papers [1] and our experiments, semantic tokens typically encode high-level musical features, such as instruments, vocal content, pitch, and so on. These tokens are more abstract compared to acoustic latents, which capture lower-level audio details. We refer to these as "sketch" tokens, as they provide a rough structure for the song. These tokens are derived from pretrained self-supervised models, and while we can extract broad semantic information, decoupling this into distinct, interpretable components is still an ongoing challenge. This will be a focus of our future work as we explore how to better interpret and control these tokens.

Q6: The design of predicting different types of tokens in one sequence is motivated by previous cross-modal LLM (eg, [2,3]). In our model, the input acoustic tokens can be regarded as a special $\<sos\>$ (start of sequence) token of each patch, which contains additional acoustic information.

Q7: We have noted the table caption formatting issue and will correct it in the revised version to ensure it adheres to the submission guidelines.

[1] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al., “Musiclm: Generating music from text,” arXiv preprint arXiv:2301.11325, 2023.

[2] Wu B, Yan C, Hu C, et al. Step-Audio 2 Technical Report[J]. arXiv preprint arXiv:2507.16632, 2025.

[3] Zhou C, Yu L, Babu A, et al. Transfusion: Predict the next token and diffuse images with one multi-modal model[J]. arXiv preprint arXiv:2408.11039, 2024.

We greatly appreciate the reviewers' time and effort in reviewing our manuscript. Their constructive comments were instrumental in refining our work.

About originality:

While [7] proposes generating continuous VAE latents of each patch, it lacks an explicit sketch-prediction stage. After reproducing [7] in the speech domain, we encountered difficulties with convergence, especially when using a larger patch size. For the song generation task, we found that [7] failed to produce intelligible results, even when the patch size was reduced to 2 or 4, as shown in Table 4 (the ‘without sketch’ configuration is quite similar to [7], but it was time-consuming with small patch sizes).

In contrast, our method iteratively generates discrete sketch tokens and continuous VAE latents, and we train the model using both cross-entropy and flow-matching loss. This iterative process allows sketch tokens to guide VAE latent generation, while VAE latents enhance sketch generation by providing acoustic context. This approach not only avoids the convergence issues of [7], but also improves the intelligibility and efficiency of song generation.

About the evaluation details:

To construct the evaluation set, we randomly intercept audio prompts $A$ from Suno-generated songs $S$ across different genres (ensuring they do not overlap with our finetuning dataset). We then generate lyrics $L$ using GPT (not Suno, we’ve corrected this typo in the draft). By randomly combining these two elements, we ensure that each sample in the final evaluation set is completely unseen during training.

For computing the FAD score, we compare the MERT embeddings of the complete songs $S$ and our generated samples $G$. Even though $S$ and $G$ have different lyrics, they are expected to share similar style and genre characteristics. We assess the distance between their embeddings to evaluate the stylistic and genre consistency.

About the reproducibility:

We have already released the inference code and model weights before the rebuttal phase began. While further updates and additions are still in progress, the code currently allows users to generate songs with customized lyrics and prompts based on the model described in the paper. We will include a link to the complete repository in the camera-ready version after the anonymous review stage concludes.

About the antoencoder:

Actually we only modify the down-sample rate of the original stable-audio-vae to ensure a frame rate of 25Hz for the VAE latent. The weights of the re-trained VAE model are also available in our repository. We acknowledge that the initial description in the paper was insufficient, and we will provide a more rigorous explanation of these modifications in the camera-ready version.

About other questions:

A1: We will add these citations properly. Thanks for your suggestion.

A2: Yes, this is a typo 'batch size' -> 'patch size'. Thanks for pointing it out.

A3: The term “sketch token accuracy” refers to the average accuracy of predicting the sketch tokens during training, using a teacher-forcing strategy. Since we only modify the patch size, which affects the frequency of acoustic embeddings inserted in the preceding sequence, the sketch token accuracy serves as an indicator of the benefit of acoustic context in guiding the sketch generation.

We express our sincere gratitude to the reviewers for their detailed and thoughtful feedback. We have carefully addressed all their concerns to enhance the clarity and depth of the paper

W1: The 20 samples are randomly chosen from diverse genres, and for each sample, we further generate two items for evaluation, so the final score is the average of the 40 items. We’ll provide further details in the revised version. Additionally, the model weights have already been released, although, due to anonymous review policies, we cannot update the current submission. The released repositories allow users to provide their own prompts, which should help alleviate concerns that the model is limited to certain genres or prompts.

W2: To construct the evaluation set, we randomly intercept audio prompts $A$ from complete songs $S$ across different genres with corresponding descriptions $T$ (ensuring they do not overlap with our finetuning dataset). We then generate lyrics $L$ using GPT (not Suno, we’ve corrected this typo in the draft). Either $T$ or $A$ is used as conditions for different models. The FAD is computed between the generated samples $G$ and complete songs $S$, instead of $A$. We will add an unambiguous description in the paper. This measure ensures fairness as much as possible and avoids the potential bias introduced by using $A$ as ground truth..

W3: $h_i$ is the last hidden layer output of LM at the end of each patch. Using embeddings directly would prevent the gradient from being properly back-propagated from CFM to LM.

Q1: Actually, it is not "the acoustic content in less than a second helps for generating the semantic tokens", but "compressing acoustic content into small patches (less than a second) helps for generating the semantic tokens". All preceding acoustic content ($[0:iP]$, $P$ is the patch size and $i$ is the patch idx) is accessible, and the patch size only affects the compression rate. Previous pure AR-diffusion paper [1,2] in speech domain demonstrates that only with very small patch size ($\leq 8$), the model tends to converge. However, in SongBloom, we found that this conclusion no longer fully applies. It’s more of a trade-off: when the patch size $P$ is small, the inference speed drops and the acoustic latents generated in each iteration become too short. When $P$ is large, the most recent acoustic content ($[iP:iP+j]$, $j$ is the local idx of sketch token to be generated in current patch) will be lost. Experiments demonstrate that a patch size of 16 strikes a good balance (As shown in Figure 2, when the patch size further increases, the overall performance degrades).

Q2: This is an excellent suggestion! We plan to visualize the attention weights over prefixed tokens (including audio prompts, lyrics, and preceding semantic and acoustic tokens). This should help to better understand the interactions between different token types and provide more transparency regarding how the acoustic tokens influence the language model generation process.

L1: We will include the subjective and objective evaluation results of Mureka-O1 in this paper. This will provide a more comprehensive comparison and address the reviewer’s concern.

[1] D. Jia, Z. Chen, J. Chen, C. Du, J. Wu, J. Cong, X. Zhuang, C. Li, Z. Wei, Y. Wang et al., “Ditar: Diffusion transformer autoregressive modeling for speech generation,” arXiv preprint arXiv:2502.03930, 2025.

[2] Z. Liu, S. Wang, S. Inoue, Q. Bai, and H. Li, “Autoregressive diffusion transformer for text-to-speech synthesis,” arXiv preprint arXiv:2406.05551, 2024.

Thanks for addressing my questions. Most of my concerns have been addressed, except for the sample size and FAD calculation -- I recognized that the FAD is computed between the generated samples $G$ and complete songs $S$, instead of $A$, but my concern was the fairness of comparing to the models conditioned on text prompts. As mentioned, $A$ was obtained by randomly intercepting audio from the complete songs $S$. This means that the models conditioned on audio prompts are exposed to some common acoustic characteristics of $S$ (e.g., bpm, texture, key, etc). All in all, I decide to keep my score, leaning towards an acceptance.

Certainly, the FAD score of models conditioned on different prompts can not be directly compared. However, among models under the same prompt settings, their FAD scores can serve as a reliable and fair metric to reflect both the overall quality of synthetic samples and their similarity to real data. It is precisely based on this consideration that we included such evaluations in our paper. Thanks for your response.

We would like to sincerely thank the reviewers for their thorough review and valuable feedback.

W1: While SongBloom integrates existing components like the SSL sketch token extractor and VAE, the core model and its design—particularly the interleaved generation of discrete sketch tokens and continuous VAE latents—is our own contribution. Previous autoregressive-diffusion (AR-Diff) approaches in speech generation have primarily relied on either discrete or continuous domains, but did not successfully combine both in a way that is computationally efficient and converges reliably. Additionally, this is the first attempt to migrate AR-Diff models into song generation, which is much longer and more complex. We have tested simply training the AR-Diff model (eg, DiTAR [1]) with song data. The model struggled with convergence issues as shown in Table 4, last line, while SongBloom maintains stability and efficiency, which demonstrates the necessity and novelty of our method. Additionally, the idea of introducing acoustic context into the semantic generation process is a simple yet crucial advancement that no previous methods in song generation have tackled.

W2: We’re excited to announce that the inference code and model weights have already been released publicly after submission. Currently, users can generate songs with customized lyrics and prompts using the model as described in the paper. We continue to improve and update the repository. Please note that we are unable to share the link here due to anonymous review policies, but it will be available in the camera-ready version after the review phase.

W3: The amount of synthesized data used for fine-tuning is quite small—less than 2% of the total training data (1000 extra fine-tuning steps). The majority of our primary training data was noisy. To address this, we synthesized a small amount of clean data with more distinct structures, clearer instrumental tracks, and more precise lyrics. This fine-tuning helps the model learn to differentiate the verse and chorus and to synthesize 'cleaner' instruments, thereby improving the performance on AI-generated lyrics.

W4: Yes, as you mentioned, how to achieve interpretable and precise control is an essential yet unsolved topic for song generation. We are actively exploring solutions for this.

Q1: As discussed in W3, the primary training data is noisy and lacks consistent structure, which complicates the learning process. Additionally, the complexity of the musical accompaniment sometimes doesn’t align with standard compositional rules, leading to tanglesome performance in song generation. To overcome these challenges, we fine-tuned the model on the synthesized data that had clearer instrumental tracks and more structured compositions for a few steps. This fine-tuning helped the model learn more effectively, alleviating the negative influence of low-quality training data.

To generate the synthetic data, we first used Suno to generate a set of songs and lyrics, and then filtered these data based on metrics such as Phoneme Error Rate (PER) to ensure that only high-quality synthetic examples were included in the fine-tuning process.

Q2: We acknowledge that achieving fine-grained control over SSL-derived sketch tokens is a challenging problem. In our view, the main difficulty lies in decoupling the SSL embeddings and understanding which components directly influence the generation process, rather than the generation itself. At this stage, we don’t consider SSL tokens as the final solution. Instead, we believe that designing a new, interpretable pattern—similar to MIDI, but is both human-readable and machine-accessible—could be a better long-term approach. This would allow for greater control and customization while maintaining the flexibility and efficiency of machine processing.

[1] D. Jia, Z. Chen, J. Chen, C. Du, J. Wu, J. Cong, X. Zhuang, C. Li, Z. Wei, Y. Wang et al., “Ditar: Diffusion transformer autoregressive modeling for speech generation,” arXiv preprint arXiv:2502.03930, 2025.

Thank you for your feedback. In our experiments, we selected high-quality samples based on the following two criteria:

• low Phoneme Error Rate (PER);

• high alignment between the detected structure by All-In-One [1] and the input structure.

Besides, the instrumental composition and chord progressions of synthetic music are generally simpler and easier to learn than those of real music, which is why we refer to it as having clear instruments.

[1] Kim, Taejun, and Juhan Nam. "All-in-one metrical and functional structure analysis with neighborhood attentions on demixed audio." 2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2023.