SongCreator: Lyrics-based Universal Song Generation

We appreciate the reviewer's careful reading of our paper. We hope the following addresses all concerns mentioned.

Regarding the differences from Suno and Udio

We appreciate the reviewer’s constructive comments and will revise the introduction to better highlight our unique contributions. Since these products do not publicly disclose their methods, detailed comparisons are challenging. However, our main innovations and contributions are the design of the DSLM and the corresponding attention masking strategies. For tasks like song generation, which require generate two temporally aligned sequences such as vocals and accompaniment, DSLM offers significant advantages over independent single-stream or dual-stream modeling, and enabling independent control over generated vocals and accompaniment. We believe it is a novel solution to these tasks.

Furthermore, our model achieve diverse song generation capabilities in a unified framework by the atttention mask strategy, such as accompaniment-to-song, song editing and vocals editing in song, which Suno and Udio cannot currently achieve. Of course, due to limitations in resources and data, there are still gaps in audio quality and control through textual descriptions compared to these products. These limitations are noted in the paper, and we will provide a clearer comparison in the final version.

Regarding the introduction of Jukebox

We are thankful for the reviewer's constructive comment. Jukebox is the first and only published literature attempting song generation and should be cited in Table 1. It models vocals and accompaniment as a single entity, leading to several limitations. Our work is not an extension of Jukebox; we propose a completely different framework with DSLM and attention mask strategy specifically designed for song generation. Our approach enhances the musicality and quality of generated songs while providing flexible control, generation, and editing capabilities. As mentioned in the first response, our model achieves diverse song generation tasks in a unified framework. Most of these tasks have not been accomplished by previous models, including Jukebox. Multi-task learning further improves the model’s performance across these tasks.

Regarding the arrangement and composition tasks

As the reviewer mentioned, as an end-to-end generative model, our proposed model does not handle arrangement and composition tasks by explicitly predicting traditional music sheets or MIDI files. Instead, we train the model to directly generate natural and musical accompaniment, vocals and song based on given conditions, such as lyrics. The outputs of the model are audio, not music sheets or MIDI files. This approach enables our model to learn the knowledge for arrangement and composition and to generate songs without traditional music sheets or MIDI files.

Regarding the harmony

We define harmony as whether the vocals and accompaniment sound harmonious and pleasant together. This is crucial when generating natural-sounding songs that include both vocals and accompaniment. The works referenced in [15-25] focus on generating either vocals or accompaniment music alone, so the concept of harmony is not applicable. This perspective comes from SingSong, which involves generating instrumental music that can be naively mixed with the input vocals, and we define it as harmony.

Regarding the universal song generation

The term “universal song generation” refers to our model’s ability to perform various song generation tasks beyond lyrics-to-song, including editing, continuation, and generation from pre-determined track. This universality and flexibility are significant contributions and unique capabilities of our work. Additionally, it enables multi-task training, which further enhances the model’s generation capabilities.

Regarding the unique capabilities

As mentioned in the second response, Jukebox was the first model capable of generating both vocals and accompaniment from lyrics alone. However, our model improves the musicality of generated songs and offers more diverse song generation features, as detailed in the first and second responses.

Regarding the selection of related work

Jukebox is the only published literature on lyric-based music generation. We have provided a detailed introduction to Jukebox in the introduction section, so we chose not to repeat it in the related work.

Regarding the Figure 1 Caption

Thank you for the constructive comment. We will make the necessary revisions in the final version of the paper.

Regarding the audio quality and using the None strategy 20% of the time

We take these seriously and have provided a detailed explanation in the global rebuttal section at the top.

Regarding the structure of input and output

As the reviewer mentioned, our model supports multiple tasks, each with various combinations of lyrics, vocals, and accompaniments as input. This demonstrates the model's capabilities and flexibility. Specific input and output structures and corresponding tasks are detailed in Table 2. During training and generation, we set up various input and output combinations according to the tasks.

Regarding the implementation of song editing

We thank the reviewer's question about our editing task. In this task, users provide the edited lyrics with the start and end points of the segment to be edited. The segment following the end point is used as a prompt, while the segment preceding the start point is treated as already generated part, with a special <EDIT> token separating the two. Since the LM is trained in an autoregressive manner, the system continues generating the edited segment based on the already generated part and then seamlessly transitions into the prompt segment. To test editing performance, we manually constructed a dataset of 30 examples, encompassing songs of different styles and performed by different singers.

We thank the reviewer for recognizing our work. We appreciate the constructive comments the reviewer provided, which will help us further improve our paper. We are delighted to have the following discussion with the reviewer.

Regarding the latency discussion

The reviewer is correct that SongCreator comprises four components for successful inference, i.e., a self-supervised model (BEST-RQ), a language model (DSLM), and a latent diffusion model. However, to achieve high-quality generation, state-of-the-art music generation models such as MusicLM [1] consists of 6 components: 3 language models (semantic, coarse & fine), a self-supervised model (w2v-BERT), a prompt encoder (MuLan) and an autoencoder (SoundStream). Similarly, state-of-the-art speech synthesis models like Seed-TTS [2] consists of 4 componets: an self-supervised model (speech tokenizer), a language model, a diffusion model and a vocoder. All of these have similar or even higher complexity compared to our work.

Considering that song generation is a complex task that includes vocal composition, instrumental arrangement and harmonious generation, our primarily focus is on optimizing the musicality and quality of the generated songs, rather than real-time requirements at this stage. Therefore, we have not discussed latency. We hope to further simplify this process to achieve real-time song generation in the future.

Regarding the impact of Demucs quality and more instrumental-level controls

The reviewer’s argument is thought-provoking. We have provided a detailed discussion on the impact of Demucs quality in the global rebuttal section at the top, where we explain how our approach mitigates its impact on the overall quality of generated songs. While achieving more instrumental-level controls remains challenging at present, we believe our approach offers valuable insights and assistance in minimizing the influence of separation quality as much as possible.

Regarding supporting disentanglement control

The reviewer is correct that disentangling control for semantic tokens, which are mixtures of multiple features, is challenging. While we are not focused on disentanglement control, we believe that our proposed DSLM can be extended to address this problem. One possible approach is to disentangle the elements within the semantic tokenizer, as explored in previous work [3, 4]. Another approach is to introduce textual descriptions to control various attributes and different streams in the generated music (e.g., tempo and different instruments), which has been widely attempted in instrumental music generation. These methods are compatible with our proposed DSLM, giving it the potential to address disentanglement control challenges in the future.

Regarding aligning lyrics and Voice Activity Detection (VAD) result

We are sorry that we did not explain the detailed process clearly. Specifically, we employed an automatic speech recognition (ASR) model to provide timestamps for each sentence in the lyrics and a voice activity detection (VAD) model to detect silent segments. We then select appropriate silent segments to split the data into segments of no more than 30 seconds, ensuring the completeness of the sentences. We will include detailed explanation in the final version.

Regarding support for other languages

Due to the cost of data collection, our experiments in the paper were conducted only on the English datasets. However, as a generative model, it is not inherently bound to a specific language. With sufficient data for a specific language, the model can be adapted to support generation in other languages.

[1] Lam M W Y, Tian Q, Li T, et al. Efficient neural music generation[J]. Advances in Neural Information Processing Systems, 2024, 36.

[2] Anastassiou P, Chen J, Chen J, et al. Seed-TTS: A Family of High-Quality Versatile Speech Generation Models[J]. arXiv preprint arXiv:2406.02430, 2024.

[3] Zhang X, Zhang D, Li S, et al. Speechtokenizer: Unified speech tokenizer for speech large language models[J]. The Twelfth International Conference on Learning Representations, 2024.

[4] Ju Z, Wang Y, Shen K, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models[J]. arXiv preprint arXiv:2403.03100, 2024.

We are grateful for the reviewer's overall positive response and the consstructive comments provided. We address the concerns and questions below.

Regarding the audio quality, semantic tokenizer and using the None strategy 20% of the time

We take these concerns seriously and have provided a detailed explanation in the global rebuttal section at the top.

Regarding the control through textual descriptions

The reviewer is correct that the current model cannot control the generated songs through textual descriptions. This limitation is mainly due to the dataset. Open-source datasets only contain instrumental music with textual descriptions and lack song data with textual descriptions. Collecting and annotating song data with textual descriptions requires significant time. We look forward to addressing this issue in future work.

Regarding the baseline set

First, we would like to note that only Jukebox has implemented song generation. In section 4.2, we compared our model with Jukebox's official samples, showing that SongCreator was preferred in 60% of cases. Other baselines are SOTA models for instrumental music generation or speech synthesis, which can’t generate songs with both vocals and accompaniment. To ensure a reliable and fair comparison, we reproduced these models on our song dataset based on reliable open-source code.

Regarding the experiments corresponding to Table 15, it is important to note that the six samples provided by SingSong’s official sources are (at least partially) cherrypicked. In contrast, we used non-cherrypicked samples for all experiments. Additionally, in our reproduced results, SingSong performs comparably to our model in the subjective evaluations of the Vocals-to-song and Accompaniment-to-song tasks. We speculate that the better performance in their official demo is mainly due to their use of a larger and higher-quality dataset.

Regarding the samples used for subjective evaluation

We thank the reviewer for reminding us to introduce the samples used for subjective evaluation. During subjective evaluation, we did not use audio prompts in any experiments except for the prompt-based experiments in Tables 5 and 6, to ensure a fair comparison.

Regarding the reliability and reproducibility of the research

Due to concerns over music copyright and the potential misuse of voice cloning capabilities, we do not plan to release the checkpoints trained on the full dataset. To better assist readers in reproducing the experiments in the paper, we have provided detailed descriptions of the model structure and hyperparameter settings and plan to open-source our code in the future.

Regarding the sentence in lines 149-153

We appreciate the reviewer’s constructive comment. We will revise this part for clarity and ease of understanding.

Regarding the opposite trend of FAD compared to subjective quality in Table 3

Thanks for the reviewer's insightful comments. We believe one possible reason for this phenomenon is the inconsistency in evaluation criteria. On one hand, subjects focus mainly on the clarity and intelligibility of the songs, specifically whether the lyrics are accurately conveyed. On the other hand, FAD evaluates the overall quality and fidelity of the audio. Additionally, prior work [1] suggests that existing objective quality metrics, including FAD, fail to reliably predict the perceptual quality of generative music. We also observed a similar phenomenon in MusicGen. Given that we are assessing the combined vocals and accompaniment, we believe the perceptual quality in the listening study is more reliable.

Regarding the increase in vocal quality in Table 4

We thank this comment and apologize for the confusion caused by our presentation. In the experiments presented in Table 4, we did not provide the model with an accompaniment track. The difference between SongCreator and SongCreator (Vocal Only) lies in whether BCA and the accompaniment decoder are used during inference. SongCreator (Vocal Only) uses only the vocal decoder, while SongCreator uses the same setup as in the lyrics-to-song task, generating both vocal and accompaniment tokens before using the obtained vocal tokens to generate vocals. This means the accompaniment track in this part is still generated by the model, not artificially provided.

Furthermore, to prevent information leakage from artificially separated tracks in the accompaniment-to-song and vocals-to-song tasks, we added noise to the inputs to conceal artifacts and used only semantic tokens as inputs. This approach has been validated for its effectiveness in SingSong.

Regarding the “separate instrumental music and vocals”

This refers to non-vocal music data and a cappella performances rather than artificially separated tracks.

Regarding the omission of MusicGen from the music continuation study

First, we would like to note that MusicGen does not emphasize its capability for music continuation, focusing mainly on text-to-music generation. Therefore, we did not consider its music continuation ability. And as shown in MusicGen's paper and our experiments in Table 3, MusicGen performs worse compared to the Flattening approach. Consequently, we chose AudioLM, which uses the Flattening approach, as the baseline for music continuation.

Regarding the reason for the relatively low quality of the original data samples

Thanks for the reviewer’s insightful comments. High-quality song data from professional singers are often strictly copyrighted, so most of our data comes from performances by non-professional singers on the internet. Although these data samples have a sampling rate of 44.1kHz, their overall quality is relatively low due to the limitations of the recording environment and equipment.

[1] Vinay A, Lerch A. Evaluating Generative Audio Systems and Their Metrics[C]//Ismir 2022 Hybrid Conference. 2022.

I thank the authors for the clarifications. My concerns were adequately answered, and my score would remain unchanged.

The only thing that remained unclear to me is the quality of the "original song" samples in the demo page. Were this samples taken from the DISCO-10M ([68] in the paper), or from the in-house datasets? In both cases, the low-quality isn't fully explained by recording environment and equipment. In case the samples were either preprocessed, or processed using one of the encoder models of SongCreator, this should be mentioned both in the paper and in the demo page.

We are grateful for the reviewer's overall positive response. We will address the specific suggestions regarding Figure 2 and the information mentioned in the rebuttal in the final version of the paper. The other concerns and questions raised are addressed below.

Regarding the training dataset

We thank the reviewer for reading our paper carefully. Our data is collected from the internet, including part of the DISCO-10M and some in-house data. For the audio, we employed an Automatic Speech Recognition model to provide timestamps for each sentence in the lyrics and used a Voice Activity Detection model to detect silence segments. We chose appropriate silent segments to split the data into segments no longer than 30 seconds and ensure sentence integrity. The lyrics are tokenized by the tokenizer of BERT. Due to copyright issues, we can't open-source this dataset. To assist readers in reproducing the experiments, we have provided detailed descriptions of the model structure and hyperparameter settings and plan to open-source our code in the future.

Regarding the usage of vocal prompt and accompaniment prompt

We followed the setup from VALL-E. Specifically, the prompt audio is converted into tokens by BEST-RQ and then passed as a prefix to the DSLM. The model uses this prefix to sequentially predict the following token sequence. During training, our vocal and accompaniment prompts are taken from the previous sentence of the target audio. During inference, we randomly select unseen prompts from the test set.

Regarding the choice of masking strategy for each task

We agree with the reviewer’s suggestion and will provide detailed explanations in Appendix. Briefly, for sequences that need to be generated, we use a causal mask in SA to support autoregressive generation. For pre-determined track (e.g., accompaniment in accompaniment-to-song or vocals in vocals-to-song), we use a non-causal mask in SA to better encode the contextual representation. Regarding the BCA mask, when both vocals and accompaniment need to be generated simultaneously (e.g., in lyrics-to-song or song editing tasks), we use the BR strategy to consider the interrelationship between vocals and accompaniment. For song generation from pre-determined track (e.g., accompaniment-to-song or vocals-to-song), we use the corresponding A2V or V2A strategy to ensure that the sequence to be generated can consider the full context of the other sequence. For independent sequence generation (e.g., music continuation or vocals editing), we use the None strategy to support independent generation. Ablation experiments and results supplemented in our response to #reviewer kmxW demonstrate that our chosen masking strategies for different tasks are reasonable.

Regarding the intermediate representations

Thank you for the insightful comments. We have provided a detailed explanation about semantic token in the global rebuttal section at the top. Regarding other forms of intermediate representations, we experimented with acoustic tokens extracted from the Encodec model. In the Lyrics-to-song experiment in Table 3, we used GPT and MusicGen models, which have similar hyperparameter settings and structures (autoregressive transformer decoders). However, MusicGen’s prediction target is acoustic tokens, whereas GPT’s prediction target is semantic tokens. The results show that GPT has an advantage in terms of musicality and quality in subjective evaluations.

Regarding the impact of the Demucs quality

Noticeably, #reviewer nDSW also share a similar concern. We take these comments seriously and have provided a detailed explanation in the global rebuttal section at the top.

Regarding the training detail of the beaseline

All baselines are trained using similar strategies to those used for DSLM, includes the same dataset, training resources, optimizer settings, and similar parameter scales. Each model was trained for 500K steps. Additionally, for a fair comparison, baselines with semantic tokens as the prediction target (e.g., GPT, SingSong (Diffusion)) shared the same BEST-RQ and LDM modules as DSLM.

Regarding the training strategy for the VAE

To train the VAE, we first adopted the pre-trained model provided in DAC, then fine-tuned the encoder and decoder components (i.e., replacing the vector quantizers with a diagonal Gaussian re-sampler as in LDM). We retained the frequency-domain reconstruction loss, discriminators and adversarial loss from DAC and added a KL loss typically used for training VAEs. The VAE was trained on our prepared dataset of 100k hours of songs data, which is the same as the one used for training BEST-RQ.

Regarding the expressiveness of generated vocals

The reviewer's argument is thought-provoking, and we are pleased to share our findings. We also noticed this interesting contribution and conducted experiments to verify it. Different from works that only focus on vocal generation, SongCreator generates both vocals and accompaniment tokes before using the obtained vocal tokens to generate vocals. This means that even if the model only generates vocals, the relationships between vocals and accompaniment are also considered. As presented in the experimental results (see Table 4), we find that this approach significantly enhances the musicality of the generated vocals compared to SongCreator (Vocal Only), which only considers vocal generation. This indicates that taking the relationships between vocals and accompaniment into account helps generate more expressive vocals.

We appreciate the thorough review regarding our study. We provide detailed responses to your concerns, as summarized below.

Regarding the clarity of paper

We are thankful for the reviewer's constructive comment, which we take seriously to revise this paper for a clearer presentation. Some of the amendments are described below:

1. We will emphasize in the abstract and introduction sections that the current vocal prompts and accompaniment prompts are audio rather than text.
2. We will include pre-conditioning in the introduction and method sections to clarify the use of different attention masking strategies for different tasks. Additionally, we will provide a more detailed explanation of the basis for setting different attention masking strategies in the Appendix.
3. We will refine the style and legend of the vocal/accompaniment prompt in Figure 1 to avoid ambiguity, and further highlight the relationship between different tasks and attention mask strategies in Figure 2.

Regarding the audio quality

We acknowledge the reviewer's comment regarding the audio quality. We take this seriously and have provided a detailed explanation in the global rebuttal section at the top.

Regarding the contribution of the proposed method

We appreciate the reviewer's feedback and hope to clarify the novelty of our proposed SongCreator. The core contribution lies in DSLM and the corresponding attention masking strategies, a novel approach to achieve dual-sequence modelling such as song data including both vocals and accompaniment. In song generation, DSLM offers advantages over independent single-stream or dual-stream modeling, and enabling independent control over the generated vocals and accompaniment. By incorporating various attention masking strategies, our model can complete diverse song generation tasks, such as generation, editing, and understanding, while multi-task training further enhances the musicality of the generated songs. These advantages are beyond the capabilities of previous works, and we believe our approach provides valuable insights for other dual-sequence modeling tasks.

Furthermore, as the reviewer mentioned, our overall framework is a well-validated approach in the audio domain (the combination of LM and Diffusion). However, on the one hand, we are the first to apply this framework to the task of song generation, and its performance greatly exceeds that of the previous SOTA Jukebox. On the other hand, the framework is used to validate the effectiveness of our proposed DSLM and attention mask strategy. By comparing our method with other approaches within the same framework, we demonstrated that DSLM not only enables more flexible universal song generation but also achieves state-of-the-art or competitive performance across all eight tasks.

Regarding the ablation studies on the different masking strategies for BCA

Thank you for the suggestion. We have supplemented additional ablation studies on the different masking strategies for BCA. Specifically, we conducted AB preference tests for the lyrics-to-song and accompaniment-to-song tasks. In lyrics-to-song, we compared BR with A2V and V2A, and in accompaniment-to-song, we compared A2V with BR. The results are as follows:

​ Lyrics-to-song

​ Accompaniment-to-song

For lyrics-to-song, the comparison between BR and None has been already presented in the paper (see Figure 4). The results indicate that in lyrics-to-song, replacing the BR strategy with other strategies leads to a significant performance deterioration, demonstrating that the BR strategy is helpful for the model generate harmonious vocals and accompaniment. The None strategy, which disregards the relationship between vocals and accompaniment, performed the worst. In accompaniment-to-song, participants preferred the song generated with the A2V strategy. We believe that this is because the A2V strategy provides more context about the accompaniment sequence when generating vocals.