SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation

We are grateful for the reviewer's positive response and constructive comments. Below, we address each of the concerns and suggestions in detail.

Q1：Details of X-Codec

In our work, we use the publicly released 'xcodec_hubert_general_audio' checkpoint provided by the X-Codec authors. This model is trained on a large-scale private dataset (~200,000 hours) with a distribution similar to AudioSet, which is specifically adapted for general audio tasks.

While Encodec and DAC indeed yield better perceptual quality for audio reconstruction, we observed that in song generation, both codecs resulted in higher rates of invalid outputs—such as failure to follow lyrics and generation of noise or silence. In contrast, X-Codec consistently demonstrated more stable training, faster convergence, and higher success rates in generating coherent vocals, with notably lower PER scores. We speculate that the advantage of X-Codec may stem from its pretraining on a large amount of music data. Additionally, as emphasized in the X-Codec paper, the incorporation of not only acoustic but also semantic features from SSL representations might also contribute positively to the performance.

Although Encodec and DAC have been widely adopted in prior audio generation systems across domains such as speech, sounds, and instrumental music, song generation presents a substantially higher level of semantic complexity. Unlike speech, singing involves a broader pitch range, a rich variety of expressive techniques, and highly dynamic rhythmic patterns. Moreover, songs demand the precise coordination of vocal and instrumental components to achieve harmonic and structural coherence. Despite X-Codec operating at a relatively low sampling rate of 16 kHz, we selected it as the most suitable option available at the time. To date, high-fidelity, song-specific neural codecs tailored for generative modeling remain an open challenge in the research community.

Q2: Use of 16 kHz Sampling Rate and Its Limitations

We appreciate the reviewer’s critical insight regarding the sampling rate. We acknowledge that using 16 kHz may limit audio fidelity and overlook important high-frequency content. As suggested, we will move the discussion of this limitation to the main manuscript. As discussed in Q1, high-fidelity, song-specific codecs suitable for generation are still lacking. We thank the reviewer for this important suggestion, and we are actively working on integrating audio super-resolution modules and stereo rendering to mitigate this limitation in future work.

Regarding evaluation, we followed widely used protocols to ensure consistency with prior work. However, we recognize that many objective metrics computed at a 16 kHz sampling rate may fail to capture information above this frequency range. We agree that developing higher-resolution evaluation metrics is crucial for advancing the field of audio generation. We would also like to clarify that all ablation studies reported in the main manuscript were conducted using X-Codec at a 16 kHz sampling rate, which reflects the model’s performance under this setting. We believe these results still offer valuable insights, demonstrating the effectiveness of our proposed token pattern design for jointly modeling vocal and accompaniment tracks within a single-stage autoregressive Transformer framework.

Q3: Track Mixing

Thank you for the valuable suggestion. Currently, we mix the vocal and accompaniment tracks by simple waveform addition. We agree that developing a post-mixing parallel generation model could further improve the quality of the final output, and we plan to explore this direction in future work.

Q4：Discussion of MusicGen-Stem

Thank you for the suggestion. We agree that MusicGen-Stem provides valuable insights into the importance of disentangling different components in instrumental music generation. We will include a discussion of this work in the revised Related Work section.

Q5：Release of Evaluation Code

Thank you for the suggestion. We will release the evaluation dataset and code to support reproducibility and benefit the broader song generation research community.

We thank the reviewer for the detailed and critical feedback. Below, we provide point-by-point responses to the concerns.

Q1: Single-stage vs. Two-stage

We respectfully disagree that joint modeling should be viewed as a criticism. For text-to-song generation, single-stage models consistently outperform the two-stage pipeline in both efficiency and generation quality. Due to space limitations, we kindly refer the reviewer to Response Q1 to Reviewer F4Th for supporting experiments and detailed discussion.

Q2: Low Fidelity (16 kHz); Codec Choice

In our experiments, we tested several codecs, including higher-fidelity options like Encodec and DAC. While these codecs perform well in speech and pure music generation, they show unsatisfactory performance in song generation. For a more detailed discussion of our codec selection, we kindly refer the reviewer to Response Q1 to Reviewer 3Stt. Although X-Codec operates at 16 kHz, we selected it as the most suitable option available at the time. Currently, high-fidelity, song-specific codecs for generative modeling remain an open problem in the community. We fully acknowledge the limitations imposed by low fidelity, and we are actively working on integrating audio super-resolution modules to improve audio quality in future work.

Q3: Clarification of “Accompaniment is easier to produce”

We apologize for the confusion caused by this phrasing, and we will revise the sentence in the manuscript to avoid ambiguity. Our intent was to highlight the learning bias observed during joint modeling of vocals and accompaniment. To illustrate this more intuitively, we include a visualization of the Content Enjoyment (CE) curves for both tracks, generated by the mixed-mode model over training steps (refer to Section C of our anonymous demo page ). The figure shows that the accompaniment track improves much faster, reaching near-GT performance around 104k steps, whereas the vocal track improves more slowly, and still exhibits a noticeable gap from GT performance even after 168k steps.

Q4: Evaluation Criteria and Fairness of Comparisons

At the time of submission, there were no open-source baselines specifically for text-to-song generation (except Jukebox), which limited direct comparisons. We appreciate the reviewer’s suggestion and revise our evaluation to group baselines by input modality. We also incorporate automatic audio aesthetics metrics for a more comprehensive assessment.

For text-to-song, we compare against a two-stage pipeline and Parler-TTS*. Jukebox is excluded due to its impractical inference time (~3 hours for 20s on a V100). (refer to Section B of our anonymous demo page ).

To evaluate accompaniment quality, we separated it from the generated songs and compared it with instrumental-only models. As shown in the table below, SongGen's accompaniment achieves performance between these two, despite being trained on a much smaller song dataset without pure instrumental music data.

Q5: Quantitative analysis of the proposed data processing pipeline

Thank you for the helpful suggestion. However, building a gold-standard lyrics dataset and conducting human evaluations are time-consuming. To provide timely feedback, we instead randomly sample 5,000 examples from the training set and evaluate the filtering strategy using recently proposed automatic metrics for audio aesthetics and text-audio alignment. The results show that edit-distance filtering improves Content Enjoyment(CE) and Production Quality(PQ), while CLAP-based filtering increases CLaMP3 scores, indicating stronger audio-text relevance.

Q6: Missing Objective Metrics for GT / Suno

We noticed that prior works typically do not report objective metrics for GT, and we thus omitted them as well. We now include them for completeness： FAD 0, KL 0, CLAP 0.18, CLaMP3 0.1052, PER 21.39, SECS 76.42. As the CLAP score for GT is unexpectedly low, we also report CLaMP3 for a more robust text-audio alignment evaluation. For Suno, due to the lack of an official API, all user study samples were generated manually. With 326 samples, full objective evaluation was infeasible due to the high manual cost.

Q7: Why Does GT Sound Compressed?

In the demo page, we use X-Codec to reconstruct the GT audio for a fair comparison under consistent codec settings and to represent the model's upper bound.

Q8: Annotated Data

We only release annotations from MSD, including VAD results, aligned lyrics, and generated captions.

We sincerely appreciate your constructive comments, which are extremely helpful in improving our work. We are also grateful for your recognition of the technical soundness and practical value of our approach, as well as your acknowledgment of the substantial effort behind it. Below we provide detailed responses to your concerns.

Q1: Controlled experiments-- Single-stage Outperforms Two-Stage

Thank you for your insightful suggestion to conduct a controlled comparison. We train a two-stage Transformer stack using the same architecture and training data as SongGen. Specifically, in Stage 1, given lyrics, description, and a 3-second reference voice, the first model generates the vocal track; in Stage 2, given lyrics, description, and the generated vocal (prepended into the decoder), the second model generates the accompaniment track. The final song is then mixed from the two tracks.

We conduct both automatic and human evaluations. For automatic evaluation, we introduce recently proposed and effective metrics to provide a more comprehensive assessment, including the audio-text alignment score CLaMP3 [1] and content-based aesthetics metrics proposed by Meta [2], covering Content Enjoyment (CE), Content Usefulness (CU), Production Complexity (PC), and Production Quality (PQ). Additionally, we measured inference time on an A800 GPU for generating 30-second song samples. Generation audios are also provided in Section A on our demo page for the reviewers to review.

Table 1 Automatic Evaluation：

Table 2 Human Evaluation：

Our results demonstrate that the single-stage model outperforms the two-stage pipeline in both efficiency and generation quality:

• Efficiency: Compared to the single-stage approach, the two-stage pipeline requires more complex training and inference procedures. Experimental results indicate that the inference time of the two-stage model is more than twice that of the mixed-pro single-stage model.

• Generation Quality:

  • Unlike joint modeling of P(vocal, accompaniment), which indicates the joint probability of modeling vocal and accompaniment, the pipeline-based approach, which separately optimizes P(vocal) and P(accompaniment ∣ vocal), may fail to capture global optimality due to error accumulation across stages. This limitation is especially problematic for song generation, where harmony between vocals and accompaniment is crucial. For instance, in genres like rap, vocal rhythm is tightly coupled with the instrumental beat. Generating vocals first without considering the underlying rhythm may result in rhythm misalignment. Conversely, in expressive genres such as ballads, where vocals typically guide the emotional flow, generating accompaniment first may constrain vocal expressiveness, resulting in rigid or disconnected performances. In both cases, pipeline approaches struggle to capture the intricate interplay between vocals and accompaniment. In contrast, joint modeling in a single-stage framework enables better coordination and global optimization, resulting in more coherent and musically aligned outputs.
  • Our results further support these observations — our single-stage model consistently outperforms the two-stage pipeline across both automatic and human evaluations, particularly on the aesthetics metrics (CE, CU, PC, PQ) and subjective scores such as Overall Quality (OVL.) and Harmony (HAM.).

Q2: Statistical Rigor of Subjective Evaluation

Thank you for pointing this out. We have updated the results to include 95% confidence intervals for each subjective metric. Furthermore, we incorporate newly proposed aesthetics metrics[2] to enhance the evaluation and better reflect the improvements brought by our approach. Due to space limitations, the updated results are provided in Section B of our anonymous demo page.

[1] CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages (Feb 2025)

[2] Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound (Feb 2025)