SongComposer: A Large Language Model for Lyric and Melody Composition in Song Generation

1. The novelty is still limited because SongComposer is based on InternLM2-7B with little modification, most of the work is on data processing.
2. The experiment setting needs further discussion. Since other LLMs are not for melody or lyrics tasks, it is obvious that they will have poor performance compared with SongComposer.

1. This paper makes extensive use of open-source Large Language Models (LLMs) to process the dataset. The hallucination effect of large language models can lead to errors, such as the omission/generation of blank content and incorrect content, resulting in a lower quality dataset. The paper does not provide an expanded analysis or explanation of how to address this issue.
2. The model introduced in this paper is based on the existing capabilities of InternLM2-7B, with integration and subsequent optimization, suggesting that the model may lack in terms of innovation.
3. There are flaws in the evaluation: (1) Despite the paper's emphasis on alignment issues, it does not include a specific evaluation targeting alignment. (2) The evaluation metrics chosen in this paper are insufficient, particularly the subjective evaluation metrics.

1. The methodological innovation is limited; the paper primarily reorganizes the data and explores different initialization methods for additional embeddings. It offers little insight into out-of-domain token embedding initialization for other LLM fine-tuning processes.
2. The paper lacks a comparison with existing LLM-based melody generation models, such as ChatMusician[1], and recent melody-to-lyrics generation models, like LOAF-M2L[2].
3. The generated Chinese lyrics do not appear to be very rhymed, and it's a pity that no improvements are proposed to address rhyming, as this is something that existing LLMs cannot achieve in a zero-shot manner.

[1] Yuan, Ruibin, et al. "Chatmusician: Understanding and generating music intrinsically with llm." arXiv preprint arXiv:2402.16153 (2024). [2] Ou, Longshen, Xichu Ma, and Ye Wang. "Loaf-m2l: Joint learning of wording and formatting for singable melody-to-lyric generation." arXiv preprint arXiv:2307.02146 (2023).

Deficient Musicality in Melody Generation: Despite the novelty of the approach, the musical quality of the generated melodies falls short compared to what current advanced models can achieve.

1. In terms of end-to-end music generation, models like Jukebox and Suno deliver significantly more musically appealing results.
2. Using a pipeline approach, where a top-performing symbolic music model (music transformer or more recent development of whole song generation using hierarchical diffusion models) first generates the melody followed by adding lyrics, would also yield better musicality and coherence. The comparatively lower quality of SongComposer's melodies makes it less competitive in the rapidly advancing field of Music AI.

1. Lack of Evaluation on Music Understanding
   • The paper frequently claims that SongComposer "understands" sheet music and musical structures. For example, the abstract mentions: "Sheet music understanding, we designed a flexible tuple format to load lyric and note attributes, fostering word-level alignment between lyrics and melodies, and enabling SongComposer to generate lyrics with accompanying well-aligned melodies." However, the paper does not provide concrete evaluations to substantiate the model's music understanding capabilities. The evaluations focus solely on generation tasks, without assessing the model's comprehension of musical concepts or structures.
   • To support the claim of "understanding," throughout the paper (e.g. sheet music understanding), the authors should include evaluations on music understanding tasks. For instance, Yuan et al. in ChatMusician [https://arxiv.org/abs/2402.16153] propose a college-level music understanding benchmark to evaluate such capabilities. Incorporating similar evaluation settings would strengthen the paper's claims. Additionally, given the three-stage training strategy on data that differs significantly from the original text distribution, there is a concern about catastrophic forgetting, which could diminish the model's general understanding abilities.
2. Interpretability of the Objective Metrics
   • The paper uses Melody Distance (MD) as a key metric to evaluate the similarity between generated melodies and ground truth. However, the interpretability of the MD values is unclear. In the memorization analysis section, the paper states that a melody distance of approximately 2.4 indicates that the generated melody is "far away" from the training data. Yet, in Table 1, the lowest MD value is 2.2, which is not significantly different from 2.4. This raises questions about the usefulness of MD as an informative metric.
   • Moreover, in practical experience, the connection between lyrics and melody is often weak. Therefore, low MD values might not necessarily reflect better performance in aligning lyrics with melodies. To verify the effectiveness of MD as a metric, the authors should consider constructing a simple baseline: select random lyrics from the training set, retrieve the most similar lyrics, and use their corresponding melodies to compute MD. If this baseline MD is also between 2 and 3, similar to the values reported in the paper, it would suggest that the MD metric may not effectively measure the quality of the model's generation. If it is a large number, then it would raise question about the memorization effect of the model, and whether it is safe to be open-sourced.
   • Additionally, the large MD values observed for models like LLaMA 2 and InternLM 2 might be due to invalid output formats, as these models have not been fine-tuned on the specific notation and may not produce outputs that conform to the expected format. This could artificially inflate their MD values, making direct comparisons less informative. Therefore, the paper should provide a more detailed analysis and discuss the limitations of the MD metric, possibly including additional baseline comparisons and considering alternative evaluation metrics.
   • Not just MD, I believe the simple retrieval based baseline should be implemented to give a better understanding of all objective metrics, including PD, DD, MD, Cosine, ROUGE2, BS.
   • (Note: There is also a typo in Table 1 where "Cosine Dist." should be "Cosine Sim.")
3. Comparison with Established Music Notations
   • The paper introduces a custom notation format for encoding lyrics and melodies. However, established notation systems like ABC notation already provide capabilities to encode lyrics, comments, notes, and chords effectively. Comparing the proposed notation with ABC notation would offer insights into the advantages or limitations of the new format. Such a comparison could address aspects like expressiveness, ease of use, compatibility with existing tools, and potential for broader adoption. Without this comparison, it is unclear why a new notation was necessary and how it improves upon existing standards.
4. Fairness of Baseline Comparisons
   • The paper compares SongComposer with other large language models (LLMs) like GPT-4 and GPT-3.5 using few-shot prompting. However, these models have not been trained on the specific notation introduced in the paper. This makes direct comparisons potentially unfair and less informative, as the performance gap may be due to unfamiliarity with the notation rather than differences in underlying capabilities. The baselines might not adequately reflect the strengths of the competing models in the context of song composition tasks. The paper would benefit from fine-tuning baseline models on the notation or designing evaluation tasks where all models have equal footing.