DIS-CO: Discovering Copyrighted Content in VLMs Training Data

Dear Reviewer,

We appreciate the time and effort invested in reviewing our paper. Below, we clarify your comments.

Why do you think using copyrighted material in the training data is copyright infringement rather than fair use?

If the paper gave the impression that we consider all copyrighted content in training data inherently infringing, that was not our intention.

We fully recognize that the legality of training on copyrighted data is complex and context-dependent. However, especially in light of recent lawsuits, we notice that training on copyrighted data without authorization can raise serious concerns, particularly when the resulting models are deployed commercially.

Our work does not take a legal position on whether a specific instance constitutes infringement or fair use. Rather, our contribution is technical: we propose a method of detecting whether specific content (copyrighted or otherwise) was included in a model’s training data.

If you think using copyrighted material in the training data is copyright infringement, why do you include the copyrighted movies in your dataset? (…) What we should do is to detect “copy” rather than “copyright”.

With respect to our own use of copyrighted movie frames in the MovieTection benchmark, we address this concern directly in our Impact Statement. Our dataset contains a small number of frames per film, is used solely for education and research purposes, and does not substitute the original work, conditions under which we believe fair use applies.

We also fully agree with the distinction made between detecting “copyright” and “copy.” The key issue is not whether a model saw the material, but whether it memorizes and reproduces it in a problematic way. As noted, “Infringement happens when model generates something very similar to the copyrighted image…”

This was also a core motivation behind our work. For example, it is probably fine, and even expected, for a model to know Notting Hill is a romantic comedy. But when it can consistently name the movie title from frames like Figure 9 (in the paper), that level of specificity suggests not just general understanding, but visual memorization, which may go beyond what’s expected from general exposure.

This is why we designed the benchmark, and we believe it plays an important contribution for the field, as it helps drawing the line between the different levels of memorization that VLMs may exhibit. In that sense, we believe our work fully aligns with your view.

Second, I am curious about why you focus on the movies. (…) Also, is your task limited to the video domain? Why not images?

Our focus on movies was mainly motivated by the fact that they are likely to be familiar to a broader audience, making the results easier to interpret and relate to than, for example, paintings or less widely consumed copyrighted material.

As for the second part of your question: no, our method is definitely not limited to the video domain. In fact, as shown in our proof-of-concept experiment using COCO, our technique works on static, single images. On top of that, we also conducted an initial experiment using a different type of copyrighted content: comic books.

For this, we assembled a small dataset of five different works: Astérix Legionary, Lucky Luke: Billy the Kid, The Amazing Spider-Man #1, Spirou and Fantasio: Comme Zorglub, and Tintin in America. As these comics come from long-running series with similar visuals, we expected models to struggle to identify specific titles. However, GPT-4o performed surprisingly well:

That said, we chose not to further develop this experiment in the paper because, unlike movie frames, comic pages often contain text within the images. This introduces an additional variable, making it harder to determine whether the model is relying on the artwork alone or also using the text to make its predictions. Since our goal was to evaluate visual memorization specifically, we felt that this mix of modalities could reduce the clarity of our findings.

Finally, I am concerned about the novelty of the proposed DIS-CO (…) It is a combination of known technologies.

While DIS-CO builds on existing components, its application to detecting visual copyrighted content in VLMs is, to our knowledge, novel. No prior work has addressed this task. And the fact that it combines known techniques is not by itself a limitation! In the end, it’s what enabled us to outperform the strongest prior method [1] and bring new insights to the field.

[1] Li Z, et al. Membership Inference Attacks against Large Vision-Language Models. NeurIPS, 2024.

Conclusion:

We hope that our answers have addressed your concerns. Please let us know if any further clarification or additional information is needed from our end.

Dear Reviewer,

We appreciate the time and effort dedicated to evaluating our paper. We understand the concerns raised and below, we address each point in detail:

Q1.1 How applicable and universal is the proposed mechanism?

We acknowledge that MovieTection’s focus on box-office hits may limit its coverage of niche films, which are also subject to copyright and may appear in pretraining datasets.

We would like, nonetheless, to make two clarifications:

• While this is indeed a limitation, it reflects a broader challenge shared by most existing studies in the field. Detecting memorization of rarely seen content is inherently more difficult than detecting memorization of content that appears frequently during training.
• We agree that it is important to assess how VLMs respond to niche content. To complement our experiments, we conducted an auxiliary study in which we tested DIS-CO on three films with little to no international projection.

It’s interesting to see that with DIS-CO, a smaller movie such as Gaiola Dourada is still partially recognized by gpt-4o, suggesting that even limited exposure during training can leave some detectable traces. However, the overall trend is clear: as a film’s popularity and likely exposure in training data decreases, so does the strength of the memorization signal. This reinforces our focus on popular films: while weaker signals may appear for niche works, box-office hits offer stronger, more consistent signals for evaluating memorization.

Q1.2 The impact of image compression and resolution changes has not been tested.

We did a small experiment to evaluate how different resolutions affect DIS-CO’s performance.

As expected, lower resolutions reduce accuracy, since the model has less visual details to work with. That said, DIS-CO remains effective overall: suspect movies are still clearly distinguishable from clean ones, which consistently score near 0% accuracy, regardless of the resolution. Although smaller frames introduce a light performance drop, they can still be a practical choice to reduce computational effort without significantly impacting detection quality.

Q2 + Q5 Does it involve ethical issues? Is it aligned with specific copyright laws?

We believe that all aspects of our work are aligned with fair use exceptions and conducted in accordance with relevant ethical and legal standards.

We would like to clarify that the dataset release was reviewed in advance by our institution’s Data Protection Officer (DPO), who provided a positive assessment regarding its compliance.

In addition, to safeguard ethical usage and prevent misuse, the dataset is being released under a restrictive CC BY-NC-SA 4.0 license, which limits its use strictly to non-commercial research and academic purposes.

Q3 The current study only simulates key forgery and fine-tuning attacks (…) This results in a potentially insufficient defense capability, thus leaving certain security vulnerabilities.

We respectfully note that our paper does not mention or simulate “key forgery” attacks. Could you clarify what was meant by this term, and how it relates to our work?

Assuming the comment refers to our fine-tuning experiment, we would like to clarify that this component serves primarily as a proof of concept to illustrate that disclosure of memorized content can be mitigated. While we do explore fine-tuning as a way to reduce the disclosure of memorization, this remains a secondary contribution. The central focus of our work is on detection, not mitigation.

Q4 The paper does not analyze the computational cost introduced by multiple frame inputs.

Following this suggestion, we evaluated the GPU memory usage when increasing the number of input frames.

On average, each additional frame increases memory by 0.46 GB (Qwen2-VL 7B) and 1.53 GB (Qwen2-VL 72B). This corresponds, for the bigger model, in less than 3.5% total additional memory to process 4 frames, showing that the main memory requirement comes from loading the model itself. The cost added by using multiple frames is small and unlikely to be a problem in most settings.

Conclusion

We hope that our responses clarify the concerns raised and demonstrate the validity and value of our work. We thank you for the insightful feedback and we are happy to present further clarifications.

Dear Reviewer,

We greatly appreciate the time and effort you invested in reviewing our paper. Below, we provide a response to your question.

My question to this paper is whether if this paradigm can be applied to other copyright content as well beyond just movie, which also states in the method review section: "The paper focuses on movie frames and captions, and it remains to be seen if the approach can generalize to other kinds of copyrighted content or other multimodal tasks.”

We believe it definitely can! In fact, we even ran DIS-CO on a small experiment with a completely different type of visual content: comic books.

The works we used for this were: Astérix Legionary; Lucky Luke: Billy the Kid; The Amazing Spider-Man #1; Spirou and Fantasio: Comme Zorglub; and Tintin in America.

Given that these comics come from series with multiple of similar-looking volumes, we expected a model to struggle. Still, GPT-4o managed to correctly identify the specific comic-title surprisingly often.

That said, we ended up leaving this part out of our main experiments because, unlike movie frames, where the visual signal is (mostly) isolated, comic books introduce an extra variable: text inside the images. This makes it harder to know whether the model’s prediction came from the visual content or if it was leveraging the text. Since our goal was to test visual memorization specifically, we felt that mixing modalities here could weaken the core message.

Conclusion:

We hope that our answer has addressed your concern. Please let us know if any further clarification or additional information is needed from our end.

Dear Reviewer,

Thank you very much for your valuable feedback and comments. Below, we address each of your questions.

W1. Lack of formal mathematical analysis.

Here, we present our mathematical analysis to support our intuition on why free-form completions (FF) are a more robust indicator of memorization than Multiple Choice (MC) formats.

Random-Chance Baselines: In an MC setting with $k$ options, the probability of a correct guess by chance is $P_{MC} = \frac{1}{k}$, wich for $k=4$ would result in 25%

By contrast, in an FF setting the model must generate the exact label from a vast output space $\Omega$ (with $|\Omega| \gg k$), so that the chance-level probability is approximately $P_{FF} \approx \frac{1}{|\Omega|},$ which is orders of magnitude lower than $P_{MC}$.

Impact of Non-Uniform Priors: It is true that models may exhibit a bias toward more popular movies. Even if this bias increases the chance for a popular movie by a factor of 100, the overall probability remains extremely small. For instance, if $|\Omega| = 10{,}000$, even with a bias, the probability $P'_{FF} = \frac{100}{10{,}000} =$ 1%, which is still 25 times lower than the MC chance of 25%.

Q1. Clarification on Dataset Contamination.

We fully agree that external sources could influence the model’s performance. To investigate this potential contamination, we would like to draw your attention to Appendix D. We found that GPT-4o (with a knowledge cutoff in October 2023) acknowledged 20 out of the 50 clean movies included in MovieTection.

We believe it's reasonable to assume these 20 titles are likely candidates for having appeared in publicly accessible sources such as trailers, promotional posters, or press coverage before the model's cutoff date, though not as full movies, since they had not been released yet.

The table below summarizes findings from Table 10 in Appendix D.

We observe that movies released closer to the model's cutoff date, and thus with more external content available, result on the highest accuracy scores, while very recent releases, for which public information was considerably sparser at the cutoff date, show no correct mappings. The highest observed accuracy for this subset of recognized movies was, however, only 6%, significantly lower than accuracy scores typically achieved for suspect movies. Therefore, we firmly believe that exposure to external content alone minimally impacts our primary results.

W2 + Q2 Could you consider conducting study to isolate the contributions of specific components of DIS-CO?

Isolating and understanding the contributions of individual components in DIS-CO is definitely essential. However, we believe to have already explored some of these aspects, for which we would like to direct your attention to Appendix I, where we analyze the effect of varying the number of frames per query. Our analysis is conducted across the different frame types (main vs. neutral) and model sizes, offering a wide view of how these variables affect performance.

As for your suggestion regarding prompt variations, we conducted an additional experiment to evaluate model sensitivity to different wordings, which can be relevant for practical applications, since user inputs will naturally have some variance in the words.

We designed a small-scale evaluation using two categories of prompts:

• Biased Prompts: These include additional cues that might assist the model.
  • Example: “What Oscar-winning movie is this frame from?”
• Paraphrased Prompts: Semantically equivalent rephrasings of the default prompt.
  • Example: “Can you identify what movie is present here?”

We evaluated model responses on a subset of the MovieTection dataset, summarized below:

Given that in real-world settings the target content may not always be a well-known blockbuster, biasing the model toward popular titles through hints may not be ideal. We believe that sticking to neutral prompt variations is a more reliable choice, as it avoids introducing external priors and better reflects the model’s actual memorization.

Flag for Ethics Review

Regarding the ethics review flag, please refer to our response provided in the reply to Reviewer V24U (Q2 + Q5), which addresses this issue as well.

Conclusion:

We hope that our answers have addressed your concerns, and thank you once again for your valuable feedback.

Please let us know if any further clarification or additional information is needed from our end.