VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling

1. The authors claim that their method is end-to-end while the MIDI-based methods are not end-to-end, but VidMuse relies on a pretrained audio decoder MusicGen to generate music from embeddings. Besides, M2UGen also uses a similar decoding strategy. The difference between VidMuse and M2UGen is explained as "conditioned solely on visual input", but M2UGen can generate music with only video inputs. It is doubtful that the performance gain mainly comes from the larger amount of training data. The comparison with Diff-BGM (Li et al, 2024) should also be discussed.
2. The authors mention efficiency and computational costs multiple times in the ablations but do not provide quantitive results like latency or throughput.
3. The writing of this paper is poor. The main paper is not self-contained as Section 5.6 relies on a figure in the appendix. There are also many typos.
4. For the qualitative result, it is improper to conclude that CMT has no high-frequency components. If the vertical axis is the frequency in Hz, 8K Hz is far beyond the frequency range of common instruments. For instance, the maximum frequency of a piano is usually 4K Hz. High-frequency components in the figure may be due to harmonic waves or overtones related to specific sound fonts. It is also difficult to conclude from the figure that M2UGen has repetitive structures.

1)I don't think this paper is well-writen and authors may write this paper in a rush, could refer to: Line 378-390 The Table 2 right side is out of the boudary of the page; Line 498-505 The Table 4 and Table 5 overlap with other in a conflict manner.

2)The evaluation is limited to the V2MBench, which is author self-proposed benchmark, and does not include any external validations on other datasets, casting doubt on its generalizability and overfit on their own dataset.

3)Metrics like FAD, KL, and ImageBind Score do not provide enough insight into real-world applications, as they lack clear explanation of their relevance to audio-visual coherence. So in the human user study should include some subjective quesions about how they rate them.

4)They didn't specify involved human objects source and how they pay them for both data quality process and user study period.

5)Overflowing tables (e.g., Tables 2, 4 and 5) affect readability and reduce the professional presentation of the work.

• While music source separation can be used to remove the vocal soundtrack, the final generated music will not contain any vocal music. It is not a wrong choice but human vocal music is missing and obviously it is still playing a critical role in music. It just needs more investigation to generate both background music and reasonable vocal music. Besides, an evaluation/analysis for sound separation is needed. i.e., how well does the music sound separation (demucs) work for the collected data?
• The audio in new collected video data actually contains more than music. Most movie trailers contain sound effects that are not necessary music. Unfortunately these sound effects will not be removed by music sound separation and it is unclear whether these sound effects will affect the quality.
• The technical contribution regarding model component is relatively limited. The design of long-short term visual feature fusion is straightforward but in general I am okay with that for an application + dataset paper.
• An analysis of music genre available in the dataset is missing. This is even more important than the video genre.
• It seems like a pre-trained Encodec model is used in this work but why not fine-tune or even training a new Encodec model on this new dataset?
• Although the effort and focus here is to generate music from video input only, it actually makes sense to incorporate additional text for better style control (like V2Meow). Since VidMuse leverages pre-trained MusicGen which is already a text-to-music model, why not maintaining its original capability while incorporating Long-short-term visual embeddings? This will potentially unlock more flexible applications.
• As authors mentioned already, the ImageBind score is definitely not the best choice for video-music relevance. Even training a video-music contrastive learning model on top of this new dataset would be a better choice.
• Some minor comments:
  • Font size in Figure 2 is too small.
  • Table 4 and Table 5 have bad overlapping
  • Several videos (e.g., movie trailers) in the demo webpage seem like copyright protected. My impression is it might be ok for paper reviewing stage but please follow the correct guidance and use them carefully.

1. I am curious about the role of the video-to-music generation task. Though some previous advances tackle the task of video-to-music generation, additional constraints are attached to these models to make them more applicable. For example, some previous works [1-6] explore the rhythm synchronization of music and video, which can generate musical soundtracks with high audio-visual rhythm correspondence [1-5], and some other previous advances generate background music with corresponding emotional responses [7] or combine the music with additional audio effects [8]. However, the proposed model seems to only be able to generate semantic-matched music, which can be easily fulfilled in a training-free way, especially considering the proposed method directly leverages the pre-trained MusicGen as the music generator. There are at least three ways to achieve a similar goal: 1). Use some video-music model (such as M2UGen [9]) to generate musical captions and then leverage MusicGen to generate semantic-matched background music. 2) Use a video captioner to generate video captions and transform them into musical captions based on its semantic information using LLM, and then leverage MusicGen to generate semantic-matched background music. 3) Use Imagebind-av, the very same model that the authors use to construct the dataset, to retrieve music with the same semantics as the visual contents, and use music captioner to generate music captions, then leverage MusicGen to generate semantic-matched background music. In other words, generating semantic-matched music, especially leveraging several existing modules, seems to be an unnecessary need, which can be solved in a training-free manner, using almost the same pretrained models. From another perspective, a good soundtrack for a given video should respond timely to the semantic change in the visual contents, yet I cannot find any explicit control module in the model architecture, nor the musical rhythm change in the provided demos. What will the music be like when the video's rhythm of the former part is rapid and enthusiastic, yet suddenly becomes slow and sad in the latter part? Consequently, the restricted applicability of the proposed model significantly diminishes the paper's contribution.

2. The model architecture is trivial. Clip is used for visual encoding, Encodec is utilized for audio codec, and MusicGen is used for music generation. That is to say, only the long-short-term visual module is the newly proposed module, while it is constructed by several attention-based integration and fusion blocks. The entire framework is more likely to be a successful industrial product rather than a highlighted research finding.

3. The experiments are insufficient. The authors only conduct experiments on some weak baseline methods. For example, VM-Net and CMT are works published 7 and 3 years ago, and M2UGen is a music-centric multi-task model that is not specifically designed for video-to-music generation. On the contrary, some newly proposed video-to-music generation methods [1-6] are not compared. Besides, have the authors tested the model's performance on other existing benchmarks, such as BGM909 [5], LORIS [3], or SymMV [6]? Experiments on more available benchmarks and comparisons with more recent advances are needed to support the authors' claim.

Reference:

[1]: Zhu, Ye, et al. "Quantized gan for complex music generation from dance videos." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.
[2]: Zhu, Ye, et al. "Discrete contrastive diffusion for cross-modal music and image generation." arXiv preprint arXiv:2206.07771 (2022).
[3]: Yu, Jiashuo, et al. "Long-term rhythmic video soundtracker." International Conference on Machine Learning. PMLR, 2023.
[4]: Su, Kun, et al. "V2Meow: Meowing to the Visual Beat via Video-to-Music Generation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.
[5]: Li, Sizhe, et al. "Diff-BGM: A Diffusion Model for Video Background Music Generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[6] Zhuo, Le, et al. "Video background music generation: Dataset, method and evaluation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
[7] Kang, Jaeyong, Soujanya Poria, and Dorien Herremans. "Video2music: Suitable music generation from videos using an affective multimodal transformer model." Expert Systems with Applications 249 (2024): 123640.
[8] Movie Gen: A Cast of Media Foundation Models, meta, 2024
[9] Liu, Shansong, et al. "M $^{2}$ UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models." arXiv preprint arXiv:2311.11255 (2023).