Investigating Memorization in Video Diffusion Models

I have some concerns:

1. The proposed GSSCD is just a naive extension of SSCD. For video copy detection (an area you may know) research, this aggregation (GSSCD) is very common. Please see the recent VSCD competition by Meta AI. In addition, SSCD is not a good metric itself for copy detection in the background of the diffusion model (Do you know why?). You should design something to substitute SSCD instead.
2. The solutions and evaluations are also naive :) I want to see some insightful methods.
3. The video diffusion models used in the paper seem a little old, for instance, you may use the SOTA like Open-Sora, CogVideo, and etc. The poor performance of the used video diffusion models prevent you from drawing useful conclusions.

1. The writing needs improvement. In my opinion, the paper reads more like an experimental report. The contribution section is overly long, and there are frequent direct comparisons with previous work throughout the paper. These comparisons are often presented in bullet points, with each point being lengthy. Additionally, in Table 1, what do the red text and bold font represent? This is not explained in the paper.
2. The paper lacks coherent expression, making it difficult to read smoothly. Each section feels disconnected from the others, affecting the overall flow.
3. In Table 1, the notations OFS-k and NMF appear for the first time, but there is no explanation provided in the table’s caption, leaving readers confused when interpreting the content.

This paper points out several problems of prior study in investigating the memorization problem of video diffusion models, which are valuable. However, there are the following concerns that make the paper less solid and convincing.

1. It is doubtful that content and motion memorization can be disentangled as in this paper. This paper uses optical flow to measure motion memorization, as indicated in Section 2.2. However, optical flow is also influenced by the content, as shown in examples with similar content in Figure 2 (like the first and third examples).

2. Memorization detection methods detect but do not solve the memorization problem. For instance, if a user wants to generate a video based on a text prompt, but the generated video is detected to have memorization issues, what should the system do to solve the memorization issue while meeting the user's requirement? Discard the video and regenerate based on the prompt?

3. Clarification is needed on metrics computation in Table 2. It would be helpful to provide specific equations or pseudocode to show how these metrics are calculated.

4. The paper organization is sometimes confusing. For example, Table 1 (left) on page 4 is first referenced on page 6, after the reference of Table 1 (right) on page 6. Additionally, table titles lack clarity, making it difficult for readers to understand the key conclusions from tables.

1. The paper introduces a new metric for evaluating content memorization in video diffusion models in a frame-level manner but does not sufficiently justify the need for these over existing methods developed for image diffusion models. It remains unclear whether the current metrics for IDMs are insufficient for assessing memorization in videos.

2. Limited Validation: The new metrics' validation primarily relies on 1K manual annotations on a single dataset (WebVid-10M), which might not reflect the diversity of real-world scenarios. More extensive validation across diverse datasets and more automated, objective evaluation methods could strengthen the findings.

3. The paper repeatedly mentions the existence of a method in the field that addresses motion memorization in video diffusion models (Rahman et al., 2024), yet it fails to provide a qualitative comparison with this method. The absence of a direct, detailed comparative analysis weakens the argument for the proposed method's novelty and effectiveness.

4. Additional minor weaknesses:

a. Figure 3 appears to be redundant, as Table 2 already sufficiently demonstrates that ModelScope’s generations replicate the image training data.

b. In Table 2, it would improve clarity if text-conditional generations (T2V) and unconditional generations (UV) were distinguished in separate column.