An Efficient Framework for Crediting Data Contributors of Diffusion Models

6. We thank the reviewer for this comment. As suggested, we have also compared different retraining approximations, including fine-tuning (FT), gradient ascent (GA), influence unlearning (IU), and their sparsified version, and show that sFT provides the best performance. For GA and IU, we found that their retraining approximations, both with and without sparsification, were ineffective (refer to Table 12 in Appendix F.4).

Building on this, we conducted an ablation study across different attribution kernels—Shapley, leave-one-out (LOO), and Banzhaf—evaluated using FT, sFT, and retraining (Tables 5, 8, and 11 in Appendix). Among these kernels, the Shapley value consistently demonstrated superior performance. Furthermore, within the Shapley framework, sFT outperformed other retraining approximations when operating under the same computational budget.

4. We thank the reviewer for raising this question. The Shapley value is a general game-theoretic framework that works across various scenarios, as it only requires sampling subsets of "players" (data contributors in this context) and evaluating their impact on a given model behavior. This flexibility allows the Shapley value to be adapted for different task settings, including both conventional prediction tasks and generative tasks. The key difference between these tasks lies in how model behavior is defined. In prediction tasks, model behavior is typically captured by each dimension of model output, such as class probabilities, or by using metrics like classification accuracy, precision, or recall. In contrast, generative tasks present unique challenges: the generation process is inherently stochastic (as models produce different outputs based on varying initial Gaussian noise inputs) and the outputs are high-dimensional (e.g., 224 × 224 = 50,176 pixels), with individual dimensions lacking direct semantic meaning. To address these challenges, we propose defining model behavior in terms of global model properties. Specifically, we evaluate a batch of generated data using application-relevant metrics that capture key aspects of generated outputs, such as image quality and diversity. This enables us to align the Shapley value framework with generative tasks, and we show it remains effective via quantitative and qualitative evaluations.

5. We thank the reviewer for suggesting this ablation study. We have conducted an analysis summarizing the results of retraining-based attribution methods from existing works, including the Shapley value, the Banzhaf value, and Leave-One-Out (LOO), evaluated under three scenarios: (1) retraining from scratch, (2) fine-tuning (FT), and (3) sparsified fine-tuning (sFT). The following table presents the results for CIFAR-20.

Table 5 of Appendix F.1: LDS (%) results for retraining-based attribution across Shapley, Leave-One-Out (LOO), and Banzhaf distributions with α = 0.25, 0.5, 0.75 on CIFAR-20. The global model behavior is evaluated using the Inception Score of 10,240 generated images. Means and 95% confidence intervals across three random initializations are reported. For sparsified fine-tuning (sFT) and fine-tuning (FT), the number of fine-tuning steps is set to 1000.

Our findings across datasets demonstrate that Shapley value consistently outperforms both LOO and Banzhaf value across all training scenarios. Among the training procedures, retraining (exact unlearning) from scratch expectedly delivers the best performance across all three attribution methods. For unlearning approximation, sparsified fine-tuning (sFT) achieves superior performance compared to fine-tuning (FT), highlighting its effectiveness within our framework. Please refer to the updated manuscript for the results of CelebA-HQ (Table 8 of Appendix F.2) and ArtBench (Table 11 of Appendix F.3).

Here we address the weaknesses raised point by point.

1. We thank the reviewer for raising this point to improve our paper. We have updated our manuscript to address this point with both theoretical and empirical results.

In Proposition 1 of the revised manuscript, we show that the approximation error for Equation (6) is bounded when the number of sparsified fine-tuning steps increases asymptotically. To summarize more formally, Proposition 1 shows that $E[|F(\tilde{\theta}^{\text{ft}}_{S, k}) - F(\theta^*_S)|] \le B$ for some constant $B > 0$ when the number of fine-tuning steps $k \to \infty$.

In Proposition 2 of the revised manuscript, we show that the approximation error for Shapley values is also bounded when we increase the number of sparsified fine-tuning steps asymptotically. To summarize more formally, Proposition 2 shows that $E[\lVert\tilde{\beta}^{\text{ft}}_k - \beta^*\rVert_2] \le 2\sqrt{n} C$ for some constant $C > 0$ when $k \to \infty$. Here, $\tilde{\beta}^{\text{ft}}_k$ and $\beta^*$ denote Shapley values evaluated with global properties from sparsfied fine-tuned models and full-parameter models retrained from scratch, respectively.

We also provide empirical results to assess the insights gained from our theoretical results (Propositions 1 and 2). As shown in Figure 5 in Appendix D of the revised manuscript, the approximation in Equation (6) empirically improves with more sparsified fine-tuning steps. As shown in Figure 6 in Appendix D of the revised manuscript, the similarity between the Shapley values estimated with retraining vs. sparsified fine-tuning indeed improves with more sparsified fine-tuning steps.

2. We thank the reviewer for this comment. Since computing global model behaviors can require generating a large batch of samples (e.g., 10,240 generated images are required for the Inception Score in CIFAR-20), sparsification offers speed-ups for both unlearning and inference, making it a more suitable choice for our framework. While LoRA fine-tuning from a full model is a viable alternative, our results (Table 2 of Appendix F in the revised manuscript) show that although LoRA achieves a training speed-up comparable to sFT, its overall computational time remains longer than sFT due to the lack of inference speed-up. This highlights the advantage of sparsified fine-tuning. Additionally, we have experimented with using LoRA to calculate Shapley values on CIFAR-20 (Figure 8 of Appendix F in the revised manuscript) and CelebA-HQ (Figure 10 of Appendix F in the revised manuscript). The results demonstrate that sFT consistently outperforms LoRA under the same computational budget. Table 2 Average runtime for data subset in minutes (training + inference) for Shapley values estimated with retraining, fine-tuning(FT), sparsified FT, and LoRA fine-tuning

3. To calculate sparsified-FT Shapley values, $F(\theta^*_{S_j})$ (which denotes the global property evaluated with a full-parameter model retrained on the subset $S_j$) in Equation (5) is replaced with $F(\tilde{\theta}^{\text{ft}}_{S_j, k})$ (which denotes the global property evaluated with a pruned model fine-tuned on the same subset with $k$ steps). Concretely, if we introduce the shorthand notations:

$A = \frac{1}{M} \sum_{j=1}^M 1_{S_j} 1_{S_j}^T$

and

$b = \frac{1}{M} \sum_{j=1}^M 1_{S_j} (F(\tilde{\theta}^{\text{ft}}_{S_j, k}))$

$c = \frac{1}{M} \sum_{j=1}^M 1_{S_j}( F(\theta_{\emptyset}))$

Then the Shapley values have the closed-form expression:

$\beta = A^{-1} \left( (b -c) - 1 \frac{1^T A^{-1} (b -c) - F(\theta^*) + F(\theta_{\emptyset})}{1^T A^{-1} 1} \right)$.

Here we address the weaknesses raised point by point.

1. We thank the reviewer for this suggestion while recognizing that this is not a weakness of our paper. Indeed, data attribution methods for supervised models have been shown capable of detecting data poisoning, while data attribution methods for diffusion models have not focused on this capability [R1-R2]. We believe a comprehensive evaluation against data poisoning across different data attribution methods for diffusion models would be an impactful work and leave that for future research.

[R1] Zheng et al. - Intriguing Properties of Data Attribution on Diffusion Models

[R2] Georgiev et al. - The Journey, Not the Destination: How Data Guides Diffusion Models

2. We thank the reviewer for this valuable suggestion! In response, we have included a tutorial.md file to provide more detailed instructions for replicating our experiments, including steps for retraining, various unlearning methods, and LDS evaluation. We hope this will make it easier for others to test and utilize our method.

Here we address the questions raised one by one.

1. We thank the reviewer for this insightful question. Varying data quality can be an important factor to consider when attributing credit. Our experiments take this into account because there is inherent variability in the datasets. In the updated manuscript, we show variations in image quality within and across data contributors. In Figure 17 of Appendix F.6, the entropy distribution for each data contributor’s images is shown. For example, while the first data contributor shows a high median entropy, there are individual images with low entropy, indicating varying data quality within the contributor's provided data. Similarly, in CelebA-HQ (Figure 18 of Appendix F.6), when measuring the embedding distance to the majority cluster for images of each celebrity, we observe significant variability in data quality both within a single celebrity’s images and across different celebrities. In Figure 19 of Appendix F.6, inter- and intra-artist variations in image aesthetic scores are also observed for ArtBench (Post-Impression). Given these variations, our experiment results show that our framework can still perform well.

Here we address the weaknesses raised point by point.

1. We thank the reviewer for raising this point to improve our paper. We have updated our manuscript to address this point with both theoretical and empirical results.

In Proposition 1 of the revised manuscript, we show that the approximation error for Equation (6) is bounded when the number of sparsified fine-tuning steps increases asymptotically. To summarize more formally, Proposition 1 shows that $E[|F(\tilde{\theta}^{\text{ft}}_{S, k}) - F(\theta^*_S)|] \le B$ for some constant $B > 0$ when the number of fine-tuning steps $k \to \infty$. In Proposition 2 of the revised manuscript, we show that the approximation error for Shapley values is also bounded when we increase the number of sparsified fine-tuning steps asymptotically. To summarize more formally, Proposition 2 shows that $E[\lVert\tilde{\beta}^{\text{ft}}_k - \beta^*\rVert_2] \le 2\sqrt{n} C$ for some constant $C > 0$ when $k \to \infty$. Here, $\tilde{\beta}^{\text{ft}}_k$ and $\beta^*$ denote Shapley values evaluated with model properties from sparsfied fine-tuned models and fully retrained models, respectively.

We also provide empirical results to assess the insights gained from our theoretical results (Propositions 1 and 2). As shown in Figure 5 in Appendix D of the revised manuscript, the approximation in Equation (6) empirically improves with more sparsified fine-tuning steps. As shown in Figure 6 in Appendix D of the revised manuscript, the similarity between the Shapley values estimated with retraining vs. sparsified fine-tuning indeed improves with more sparsified fine-tuning steps.

2. We appreciate the reviewer for giving us the opportunity to elaborate on this point, as well as on the related Question 3. You are correct that the proposed approach for accelerating the computation of Shapley values for data contributors is not inherently restricted to diffusion models and could be broadly applied to other large-scale deep learning models. We chose to focus on diffusion models for this paper as they are state-of-the-art models for generating realistic images and they have been widely used for real-world applications, including artwork generation. This has sparked widespread discussions around issues such as data attribution and royalties, making diffusion models an ideal testbed for our framework. This targeted focus allowed us to conduct rigorous experiments across diverse datasets, demonstrating the utility and efficiency of our approach. That said, we agree that the methodology could be extended to other models, such as large language models, where similar computational challenges arise. In such cases, appropriate global model properties would need to be defined, and parameters for sparsified finetuning would require tuning to optimize the retraining and inference process. We believe extending our method to other model types is indeed an exciting avenue for future research, and we have noted this in the revised manuscript as a potential avenue for further exploration.

3. We thank the reviewer for the feedback. To clarify our contribution in the updated manuscript, we rephrase the first claim to: "we are the first to investigate how to attribute global properties of diffusion models to data contributors.” This rephrasing aims to clarify that our contribution focuses on crediting data contributors instead of assessing how data affect model performance. In the updated manuscript, we include both theoretical and empirical results to demonstrate that our method with sparsified fine-tuning can approximate Shapley values estimated via retraining. In Proposition 2 of the updated manuscript, we show that the approximation error for Shapley values is bounded when the number of sparsified fine-tuning steps increases asymptotically. In Figure 6 in Appendix D of the updated manuscript, it is empirically shown that increasing the number of sparsified fine-tuning steps indeed corresponds to improved similarity between the Shapley values estimated via retraining vs. sparsified fine-tuning. In Figure 7 in Appendix D of the updated manuscript, it is shown that the average runtime increases linearly with the number of sparsified fine-tuning steps. With Figures 6 and 7, the empirical trade-off between approximation accuracy and computational cost are shown. With these theoretical and empirical results, the second contribution claimed in the introduction is now more substantiated.

4. We thank the reviewer for this suggestion to improve our paper. In the updated manuscript, Section 2 is shortened to remove non-essential details. More importantly, Section 3.2 is now updated to include an in-depth discussion with theoretical justifications (Propositions 1 and 2) for sparsified fine-tuning.