Image Copy Detection for Diffusion Models

We sincerely appreciate your positive feedback and helpful suggestions. We hope our paper will meet your approval once we address the following concerns.

Q1. The large use of "refer to Section XXX in the Appendix".

A1. Due to limited space, we indeed have placed the expected implementation details in the Appendix. Since the camera-ready version will have one more page, we will move these details into the main paper. We also summarize them as below:

• How the model is trained, what is the architecture, what are the configurations?

We implement our PDF-Embedding using PyTorch and distribute its training over 8 A100 GPUs. The ViT-B/16 serves as the backbone and is pre-trained on the ImageNet dataset using DeiT. We resize images to a resolution of 224 × 224 pixels before training. A batch size of 512 is used, and the total number of training epochs is 25 with a cosine-decreasing learning rate. 

• How the other methods are compared, do the authors train them, how are they trained or adapted to the current task?

In Table 1, we apply the methods in a zero-shot manner. Specifically, we use these models as feature extractors and calculate the cosine similarity between pairs of image features, except for GPT-4V Turbo. We ask GPT-4V Turbo to reply with one similarity directly.

For the five different methods compared in Table 2, we ensure a fair comparison with the proposed method by using the same training schedule, network architecture, and configuration as described above.

Q2. Generalizability to other datasets or diffusion models.

A2. We provide the required experimental results in the table of the attached PDF, along with the associated analysis in the “Common Concerns” section. These experimental results validate the generalizability of our model across different diffusion models.

Q3. Table 1: Unfair comparison with other models.

A3. We apologize for causing this confusion. The purpose of Table 1 is to demonstrate that all existing models fail to solve the proposed ICDiff task, highlighting the necessity for a specialized ICDiff model rather than to compare our model against them. For details, please refer to the “Common Concerns” section.

Q4. The overlap between Sec 5.2 and Sec 5.3.

A4. We will delete the first part, which contains an inappropriate/unfair comparison, in Section 5.3.

Q5. The quality of text matching using Sentence-Bert.

A5. We are aware that relying solely on linguistic matching can lead to incorrect matching; however, this does not affect the quality of our D-Rep dataset. This is because the text matching is used as a pre-filtering step, and we engage professional labelers after identifying candidate pairs. Specifically, as shown in Line 113 ~ Line 121, these labelers assess the pairs from a visual perspective; for example, they distinguish between large shipping containers and small plastic kitchen containers, assigning a replication level of $0$. As Fig. 3 illustrates, only $15.92\\%$ of the dataset consists of image pairs at Level $0$, indicating the number of mismatched pairs is not large. In conclusion, this approach ensures that mismatching pairs also provide supervision signals (as negative samples) to train models instead of compromising dataset quality.

Q6. The definition of copy or replication.

A6. The definition of copy/replication in this paper follows [1]:

“We say that a generated image has replicated content if it contains an object (either in the foreground or background) that appears identically in a training image, neglecting minor variations in appearance that could result from data augmentation.”

According to [1], this definition “focuses on object-level similarity because it is likely to be the subject of intellectual property disputes”. After clarifying this, we provide a point-to-point reply to your questions.

• The generated Mona Lisa is not a copy, and thus this kind of detection is not helpful for downstream tasks.

We respectfully disagree with that. First, it is important to note that this paper aims to detect replication itself, providing a basis for subsequent copyright checks rather than directly detecting copyright infringement. Furthermore, such cases also cause plausible infringement problems; therefore, detecting them assists in copyright protection. Specifically, although the Mona Lisa is in the public domain and not subject to copyright, diffusion models are equally capable of generating other famous works, such as “The Persistence of Memory”, which remains under copyright by the Dalí Foundation. This is illustrated in the figure of the attached PDF. Utilizing these generated images for profit would constitute a copyright infringement against the Foundation. Therefore, it is reasonable to regard such cases as copies.

• Limiting the prompt solves the copy directly.

We respectfully disagree with that. Because: (1) Many open-source diffusion models, like Stable Diffusion, do not have such limits. Therefore, users can easily generate copies of copyrighted images. (2) Although commercial models limit the prompt text, as shown by [2], this approach does not completely prevent object-level copying. For instance, Fig. 1 in [2] shows copyrighted Superman logo can be generated by ChatGPT without mentioning Superman directly.

• Detecting the case without explicitly being mentioned in a prompt.

This case will also be solved by our method because we focus on matching visual features rather than prompting. For instance, whether the prompt is ‘Generate a yellow, cartoon-style mouse’ or ‘Generate a Pikachu,’ if the generated image resembles Pikachu, our algorithm can match it with a copyrighted Pikachu image.

[1] Diffusion art or digital forgery? investigating data replication in diffusion models. CVPR, 2023.

[2] On Copyright Risks of Text-to-Image Diffusion Models. Arxiv, 2023.

We sincerely thank you for your positive feedback and helpful suggestions. We address your questions below.

Q1. The selection rule and influence of $A$.

A1. We thank you for this insightful question. We provide the selection rule and influence of $A$ here.

Selection rule.

We select $A$ to maintain $g \left( x \right )$ as a validation PDF. Because the random variables are discrete in practice, $A$ cannot be randomly selected, and its value must lie within a certain range. The below example demonstrates how to calculate the range of $A$:

For the Gaussian function, according to Eq. 16:

$

\begin{gathered}\sum_{x \in{0,0.2,0.4,0.6,0.8,1}}\left(A \cdot \exp \left(-\frac{\left(x-p^l\right)^2}{2 \cdot \sigma^2}\right)\right)=1, \\ A \cdot \exp \left(-\frac{\left(x-p^l\right)^2}{2 \cdot \sigma^2}\right) \geq 0, \end{gathered}

$

we have:

$

6 \cdot A \geq \sum_{x\in { 0,0.2,0.4,0.6,0.8,1}} \left( A\cdot \exp \left( -\frac{\left( x-p^{l} \right)^{2}}{2\cdot \sigma^{2}} \right) \right) =1.

$

Therefore, we have $A \geq \frac{1}{6}$. As shown in Fig. 15, experimentally, we start with $A=0.167$. Similarly, we can calculate the range of $A$ for the Linear and Exponential functions.

Influence.

As shown in Fig. 15, for the Gaussian and Linear functions, a larger $A$ implies steeper supervision, while for the Exponential function, a larger $A$ implies smoother supervision. According to our intuition that “the probability of neighboring replication levels should be continuous and smooth,” the parameter $A$ controls the expected smoothness of the learned distribution.

We thank you again for this insightful question, which helps make our method more theoretically sound. We will incorporate these points into our revised paper.

Q2. The efficiency training and testing is not better than others.

A2. Thanks for this question. Since we use a set of vectors to calculate similarity instead of the original one, it is reasonable to expect little computational overhead. However, in our paper, we argue that the computational overhead introduced by our method is negligible. Specifically: (1) during training, our method is only $5.8\\%$ slower than the baseline; (2) during inference, our method is only $2.8\\%$ slower than the baseline; (3) the magnitude of matching, which is $10^{-9}$, is negligible compared to the magnitude of inference, which is $10^{-3}$. Furthermore, as shown in Lines 262 to 266, we find that in practice:

Our PDF-Embedding requires only $2.07 \times 10^{-3}$ seconds for inference and an additional $8.36 \times 10^{-2}$ seconds for matching when comparing a generated image against a reference dataset of 12 million images using a standard A100 GPU. This time overhead is negligible compared to the time required for generating (several seconds).

In conclusion, given the significantly enhanced performance, the introduction of such a minimal computational overhead is worthwhile in practice.

Q3. The details of the proposed dataset, such as the label distribution.

A3. Thanks. We have provided the label distribution of the proposed dataset on the left side of Fig. 3 in the main paper. To make this clearer, we will highlight it in the main text. Additionally, we provide more labeling details in the response to Q2 of Reviewer 2Ky2, which enriches the details of the proposed dataset.

We sincerely thank you for your positive feedback and helpful suggestions. We address your questions below.

Q1. The rationale behind using six vectors.

A1. We appreciate this insightful question. As you say, when we understand our PDF-Embedding approach separately, it seems counterintuitive: Typically, we use larger dot product values to indicate higher similarity. In contrast, in the context of PDF-Embedding, when two images are more similar, the dot product of a certain set of vectors, which indicates their similarity as zero, needs to be smaller.

However, as the name of our method, Probability-Density-Function, suggests, under ideal conditions, our method is continuous and should not be focused on local regions. From the perspective of continuity, the rationale or intuition is that the probabilities of neighboring replication levels should be continuous and smooth. For instance, if an image-replica pair is annotated as level-3 replication, the probabilities for level-2 and level-4 replications should not be significantly low either. After training, the largest-scored entry indicates the predicted replication level. This intuition is experimentally verified by comparing our method against two standard supervision signals, namely “One-hot Label” and “Label Smoothing,” as shown in Table 2.

Furthermore, even if we only focus locally, learning such features by optimizing the neural network will not cause contradictions. The neural network is capable of capturing the underlying distribution and continuity, ensuring that the learned embeddings reflect the smooth transition between different levels of similarity.

Q2. Generalizability to other datasets or diffusion models.

A2. We provide the required experimental results in the table of the attached PDF, along with the associated analysis in the “Common Concerns” section. These experimental results validate the generalizability of our model across different diffusion models.

Q3. Table 1: Somewhat unfair comparison with other models.

A3. We apologize for causing this confusion. The purpose of Table 1 is to demonstrate that all existing models fail to solve the proposed ICDiff task, highlighting the necessity for a specialized ICDiff model rather than to compare our model against them. For details, please refer to the “Common Concerns” section.

Q4. The design of Relative Deviation (RD): it does not reflect more egregiously wrong in some cases.

A4. We appreciate your deep understanding of our designed Relative Deviation metric. Indeed, the Absolute Deviation (Eq. 12) indicates that when $s^l = 5$ and $s^p = 0$, the penalty is greater than when $s^l = 3$ and $s^p = 0$. Although we notice it is difficult to design an evaluation metric suitable for all cases, both Relative Deviation and Absolute Deviation fulfill the primary purpose of proposing the second evaluation metric. Specifically, as mentioned in Lines 132 to 134:

A limitation of the PCC is its insensitivity to global shifts. If all the predictions differ from their corresponding ground truth with the same shift, the PCC does not reflect such a shift and remains large. To overcome this limitation, we propose a new metric called the Relative Deviation (RD).

Both Relative Deviation and Absolute Deviation can reflect global shifts: for the same label, a larger distance from the label results in a higher deviation/penalty score.

Furthermore, if we maintain when $s^l = 5$ and $s^p = 0$, the penalty is greater than when $s^l = 3$ and $s^p = 0$, we cannot normalize the penalty to the range $0 \sim 1$ for each pair, which may lead to overly optimistic results. This is because: assuming the maximum penalty is $1$, we have

$

1=penalty\left( s^{l}=5,s^{p}=0 \right) >penalty\left( s^{l}=3,s^{p}=0 \right).

$

In conclusion, while we acknowledge that our Relative Deviation may not reflect more egregious wrong in some cases, it empirically aligns better with human intuition. Additionally, we will also seek more perfect evaluation metrics.

Q5. How to assess the replication ratio of diffusion models.

A5. Thank you for your reminder, and we apologize for missing this important detail. When assessing the replication ratio of diffusion models, we consider image pairs rated at Level 4 and Level 5 to be replications. We will add this to the revision.

Thank you for your detailed response. You summarized my questions as Q1-Q5, which I find to be appropriate. Regarding Q2, Q3, and Q5, I have no additional queries.

For Q1, I would like to delve deeper into a hypothetical scenario. Imagine figures A and B are identical (or almost identical), and we have a vector derived from A and another from B following identical procedures. In such a case, these vectors should have a minimal dot product. How does the model address this requirement?

For Q4, I would appreciate further clarification on the statement, "we cannot normalize the penalty to the range $0 \sim 1$ for each pair, which may lead to overly optimistic results." Why will we have overly optimistic results?

Thank you again and look forward to your further response.

I appreciate the thorough responses to my questions, the revisions made to the paper, and the additional experiments detailed in the "Common Concerns" section. As a result, I have adjusted my evaluation positively.

From my current understanding, to better capture the properties of an image, multiple vectors are required instead of one. Within this set, some vectors carry a significant amount of the image's information, while others carry less. When assessing the similarity between two images, as their differences increase, the dot product of all the corresponding vector pairs will decrease, but those of vectors that carry more substantial information about the images will decrease faster. Thus, the relative scale relationship could change. Is my understanding accurate?

We sincerely thank you for your positive feedback and helpful suggestions. We address your questions below.

Q1. The use of continuous labels 0-1 will be more useful in practice.

A1. Thanks. According to your suggestion, we implement this idea and find that it improves the original performance while allowing for continuous predictions. For instance, the PCCs for three variants of our PDF-Embedding change from ($53.7\\%$, $54.0\\%$, and $56.3\\%$) to ($55.1\\%$, $55.9\\%$, and $57.7\\%$), respectively. We will also include this more useful implementation when we release the code.

Q2. Details of the labeling procedure are missing.

A2. Thanks. Currently, we have 10 professional labelers and 40,000 image pairs. Initially, we assign 4,000 image pairs to each labeler. If labelers are confident in their judgment of an image pair, they will directly assign a label. Otherwise, they will place the image pair in an undecided pool. On average, each labeler has about 600 undecided pairs. Finally, for each undecided pair, we vote to reach a final decision. For example, if the votes for an undecided pair are 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, the final label assigned is 3. As a result, we do not calculate the variance for each image, and we believe this process ensures high-quality labeling.

Q3. Evaluation with a separate dataset.

A3. We appreciate this constructive suggestion and provide the required experimental results in the table of the attached PDF, along with the associated analysis in the “Common Concerns” section. These experimental results validate the generalizability of our model across different diffusion models.

Thanks for your reply and glad to see that my suggestion improved the original performance. It would be helpful to include the details you mentioned in Q2 in the paper.