Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problems

We thank the reviewer for their valuable feedback and insights.

Comment: Experimental results on 10 figures are not enough to report statistically meaningful results

Response: In Table 14 we added comparisons across a test dataset of 25 images. Additionally, we added Figures 25 and 26 to the paper that show comparisons between the PSNR of the whole image model and our proposed patch-based model for each individual image in the test dataset. The plots show that for all but one test image in the CT case and for all test images in the deblurring case that the patch-based model outperforms the whole-image model, illustrating the consistency of our method.

Comment: 3.1 is difficult to follow

Response: The bordering region is added so that there are many possible ways to tile the central image with patches. If we were to use the same method of tiling the central image with patches for every diffusion iteration during the reconstruction process, there would be boundary artifacts between the patches. By varying the random patch location in Figure 1 for each diffusion iteration, we change the tiling method and eliminate boundary artifacts.

There was duplicate notation used in the original submission; originally, in Section 3.1, $M^2$ was stated to be the number of ways to choose the random patch location in Figure 1 (so that $M$ is the number of ways to choose the horizontal and vertical locations), and $M$ was also set to be the number of conjugate gradient iterations to be run in Section 3.2 and Algorithm 1. We have fixed the second use of $M$ so that now, $C$ represents the number of conjugate gradient iterations. Also, $k$ is the floor of $N/P$, so that $k+1$ patches in both directions are needed to fully cover the central image, hence the sum in eq. (8) ranges from 1 to $(k+1)^2$. In eq. (8), $i$ denotes an integer that indexes all the possible ways to select a random patch offset.

In eq. (8), the $x$ on the LHS denotes the whole image with zero padding, as we are modeling the distribution of the entire image. However, in eq. (9), the $x$ denotes extracting a random patch from an image in the training dataset. In all subsequent places where $x_t$ occurs in Section 3.2, it refers to the whole image. Finally, both the $x$ and $y$ coordinates of the patches are concatenated as inputs to the network. We modified the corresponding statement in the paper to make this more clear. We have also provided additional explanations in Section 3.1 to make the algorithm and probability model more clear.

Comment: What do the authors mean by the outermost product is computationally very expensive

Response: The outermost product refers to the outer product that iterates from $i=1$ to $M^2$ in eq. (8). When using this form of the distribution to compute the score function, the log in the score turns the product into a sum with $M^2$ terms. Since $M=64$ for most of our experiments, this would involve computing a sum over 4096 terms. As this computation would need to be performed for each of the 1000 diffusion iterations, this would be undesirably slow. Thus, instead of computing all 4096 terms and summing over them, at each diffusion iteration we randomly choose one of the terms and compute only this term.

Comment: How many samples were used in Table 1?

Response: We used average metrics over a test dataset of 10 images in Table 1. We now provide average metrics over a larger test dataset of 25 images in Table 14.

Comment: I recommend reporting perceptual metrics such as LPIPS

Response: We added Tables 12 and 13 to the revised paper which show the LPIPS scores for the deblurring and superresolution experiments in both the single measurement and small dataset settings. These results show that our proposed method (with the same parameters as in all previous results) obtains the images with the best visual image quality. Appendix A.7.2 provides further details.

Comment: How is it possible that the proposed method without training outperforms diffusion approaches that leverages training data

Response: It is indeed true that in Table 2, where a small sample of training data is available, the results from using the patch-based model are generally worse than those of Table 1 where only a single measurement is available. The main reason for this is that after fine-tuning a diffusion model using a small dataset of in-distribution data, the inverse solver assumes that the model has learned the correct prior for the test images. Therefore, when using this trained network to solve inverse problems, we simply use an existing diffusion inverse solving method without further refining the network through self-supervised learning based on the measurements data. On the other hand, the proposed method in the single measurement setting first takes a network that is known to have learned a prior that does not match the images in the test dataset, so we perform on the fly network refining via Algorithm 1. This difference in algorithm is likely to have contributed to this performance discrepancy in some tasks.

We further explored this point through the new Table 18. We conducted additional experiments by first fine-tuning (FT) the network on the small dataset and then applying the self-supervised learning (SS) based on measurements (Algorithm 1). The results show that this hybrid method (FT+SS) significantly improved performance over the FT-only or SS-only methods. We also note that the second best method may be either FT-only or SS-only method depending on the specific task. Hence, since the performance gap between training data (ellipse phantom images) and testing data (CT images or facial images) is significant, with only 10 “in-distribution” data samples to fine-tune the network, it is hard to adapt to the correct prior for testing domain. We conjecture this is because the distribution of CT images or facial images in question is very high dimensional and may contain many modes. Hence, refining it to fit a specific measurement can still have significant benefits.

Comment: Which dataset has been used for Figures 4 and 5?

Response: Figures 4 and 5 are the results of CT reconstruction with 60 views, so we used the AAPM 2016 CT challenge data. We provided more detailed information about this dataset in the experimental setup part of Section 4, and provided additional clarifications in the revised paper.

Comment: Discussion on diversity of generated samples is very vague

Response: Since the dataset used to fine-tune the networks is very small (10 images), it is difficult to meaningfully compute quality-of-sample metrics such as FID (Frechet inception distance) or IS (inception score) or diversity metrics. However, we can quantify the degree to which the networks are memorizing the training datasets. We generated 100 images using both the patch-based model and the whole image model after fine-tuning with the small dataset of CT images. For each generation, we compared the NRMSD (normalized root mean square difference) of the image with each of the 10 CT training images to find the one with the smallest NRMSD. We now summarize these results in the new Table 19, which illustrates that the whole-image model tended to memorize the training dataset much more than the patch-based model.

Comment: Table 3 is in the appendix and should be moved

Response: Table 3 provides the same information as Figures 4 and 5 but in a table format. The message of Figures 4 and 5 is that when fine-tuning the whole image model for an excessively long duration of time, it will overfit to the data and obtain worse image reconstructions, but patch-based models can avoid this issue. This is easily seen from Figures 4 and 5 from the trend of the curves, but is less clear when presented in the table format of Table 3. Hence, to avoid redundancy of information, in light of the page limit, we put Table 3 in the appendix.

We thank the reviewer for reading our response and providing additional feedback. We further rephrased much of section 3.1 to make the notation more clear. In particular, while [1] used $P$ for the patch size and $M$ for the padding, we found that it was sufficient to consider the model where the padding and patch size were equal (both set to 64 for the main experiments), so we used $P$ to denote both of these quantities. We have removed references to $M$ in the revision as this should be equal to $P$ in our model and also restated the definition of $k$. Finally, for the batch offset, we added the clarification that each offset should indeed be specified by two indices. However, in eq. (8) and onwards in the paper, we used one index $i$ to denote the patch offset so that it would be consistent with the single index $r$ (also in eq. (8)) which corresponds to the particular $P \times P$ patch. (For example, the index 1 could represent the offset (0,0), 1 could represent the offset (0,1), etc.) It would have been possible to use two indices for both the patch and the offset, but this would have led to an excessive amount of notation and subscripts. We also updated Figure 1 to make this more clear.

We believe that our experiments on test datasets of 25 images sufficiently demonstrates the superiority of using the patch-based model for the following reasons. Firstly, the test datasets were randomly drawn from the entire AAPM and CelebA datasets and are representative of their respective datasets. Secondly, to confirm that the results we obtained for these models are statistically significant, we ran two sample t-tests comparing the results from using the patch-based model and whole image model. In particular, using the 25 image test dataset experiments shown in Table 14, we compared the sample PSNR obtained by the patch-based model versus the whole image model for each of the four inverse problems. In each case, a two sample t-test found that the mean PSNR when using patches was higher than the mean PSNR when using the whole image, with a p-value less than $10^{-7}$. We repeated this test for the SSIM in Table 14 and similarly found a p-value less than $10^{-7}$ in all cases. This finding is intuitively backed up by Figures 25 and 26 where the patch-based model outperformed the whole image model in nearly all of the test images. Therefore, the experiment provides statistically significant evidence that patch-based models outperform whole image models in this setting. Finally, diffusion models are a computationally expensive method that trade off reconstruction speed in exchange for higher quality images. Table 11 shows the reconstruction runtime for a single image of different methods; running the diffusion based methods over thousands of images would take many days for a single inverse problem.

Comment: Missing baselines

Response: [1], [2], and [3] are all “general purpose” inverse solvers, which are all assumed to learn the prior from in-distribution training data and then conduct sampling on testing samples to obtain restored images from the trained model. Thus, these methods are not designed to solve OOD problems when the testing sample is out of the training distribution, and they do not provide ways to refine the network on the fly during testing sampling. Furthermore, [2] was applied to solve 3D image reconstruction problems while our submission focuses on 2D inverse problems. The general method that [2] uses for enforcing data consistency is conjugate gradient descent, which is the same as in our approach for both the whole-image model and patch-based model experiments. Hence, the “whole image, naive” and “patches, naive” experiments in Table 1 can be thought of as 2D versions of [2].

Although these methods are not designed for OOD problems so they may not be a strong baseline in the OOD setting, to address the reviewer’s comments, we conduct additional experiments to apply these methods to test samples that are out of the training distribution, the setting used in this work. Table 16 shows the results of new comparison experiments. We also showed the visual results in Figures 29 and 30, where many artifacts are clearly visible as expected. In particular, smooth artifacts are clearly visible since the ellipse phantoms that were used to train the networks are generally smooth.

[4] directly trained a network to learn the posterior distribution of the image data from the measurement given by $p(x_0|y)$. This can be seen from Algorithm 1 in [4] which requires paired data between $x_0$ and $y$ and trains the network $I_\phi$. Therefore, for different inverse problems, new networks must be retrained even if the underlying dataset is the same. On the other hand, unconditional generative methods like our proposed method (in both settings) are flexible and the same network can be used for different inverse problems. Therefore, a direct comparison between using our method for a specific inverse problem and using [4] with a network trained specifically for that inverse problem would be unfair. Moreover, Algorithm 1 in [4] requires an approximation of the score function at different timesteps $s_\psi(x_t, t)$, which in turn requires a large quantity of in-distribution training data that is not available in the single measurement setting.

In the small dataset setting, we assume that, after fine-tuning, our network is sufficiently in distribution so that no refining on the fly is needed. Then we directly applied [1], [2], and [3] with the fine-tuned network and reported the results in Table 17. For all the experiments we see that DDS [2] performed the best, and hence we use a similar method of conjugate gradient descent in our main experiments.

We thank the reviewer for their valuable feedback and insights.

Comment: Limited novelty…contribution builds on existing concepts

Response: Compared to previous patch-based model works, the main difference is that the problem setting is different. Past works studied patch-based diffusion models in a traditional generative model setting: given a dataset of images, learn the distribution of those images. In our work, we studied diffusion models in an OOD setting, where no or very limited data belonging to the test data distribution is available, so the goal is to adapt a pretrained diffusion model to a new distribution, either from a single measurement or a very small dataset. Therefore, while the networks in previous patch-based works are always trained from scratch, the networks in our work are adjusted from an existing distribution, and we show how this method drastically reduces the data requirement.

Furthermore, we provide novel experimentation and analysis of how patch-based diffusion models can avoid memorization and overfitting when being adapted to OOD data. While previous works showed that using patch-based models to solve inverse problems can lead to better results in settings of limited data, they did not explicitly illustrate that patch-based models can avoid issues of memorization and overfitting that whole-image models run into. Our work tackles these issues directly through Figure 7 which illustrates that whole image models memorize the training data when fine-tuning from a very small dataset while patch-based models have greater generalizability. Figures 4 and 5 also illustrate this point: when fine-tuning the networks for a longer period of time, the quality of reconstructed images drops substantially for the whole image models while that of the patch-based models remains relatively steady.

In the single measurement setting, DIP-based self-supervised models tend to overfit to the data. Therefore, past works have used LoRA to limit the expressiveness of the network and prevent overfitting. We show in Table 10 that patch-based models can avoid overfitting to the measurement and that early stopping is not necessary when refining the network at each diffusion iteration, even when LoRA is not applied.

In the revised paper, we also now provide novel theoretical analysis of the proposed algorithms not present in previous works. This theoretical analysis serves two purposes. Firstly, we show how in theory, using the patch-based model allows us to perform a type of data augmentation in the single measurement setting, which reinforces the point that patch-based models help guard against overfitting. Secondly, we provide more theoretical grounding for the self-supervised network refining method used in Algorithm 1 and show that this network refining process converges.

Comment: Lack of theoretical justification of Algorithm 1

Response: We added theoretical analysis of Algorithm 1 to Section A.7.1. This provides a theoretical argument for why patch-based diffusion models should outperform whole-image models in the single measurement setting. Furthermore, we show that the network refining process used in each diffusion iteration converges.

Comment: Experimental setup lacks clarity

Response: The “whole-image, correct” model indicates training a diffusion model to learn the prior of the whole image using a large dataset of in-distribution data (either CT data or CelebA facial data). Then, since it is assumed that the network has learned the correct prior, we use a traditional diffusion inverse solving algorithm that does not involve network refining. To maintain consistency with the experiments where network refining is used, we used Langevin dynamics for the sampling algorithm with conjugate gradient descent to enforce data consistency. We added the pseudocode for the reconstruction algorithm in Algorithm 2 of Appendix A.7.2.

In Table 2, “best baselines” refers to the best baseline out of the non-diffusion baselines shown in Table 1, i.e., ADMM-TV, PnP-ADMM, and PnP-RED. We did this because these experiments would have been identical for Table 1 and Table 2, so we chose to only repeat the results of the best baseline in Table 2 to avoid redundancy.

Comment: The finding of the paper is already clarified in [1]

Response: Compared to [1], the main difference is that the problem setting is different. The authors of [1] studied patch-based diffusion models in a traditional generative model setting: given a dataset of images, learn the distribution of those images. In our work, we studied diffusion models in an OOD setting, where no or very limited data belonging to the test data distribution is available, so the goal is to adapt a pretrained diffusion model to a new distribution, either from a single measurement or a very small dataset. Therefore, while the networks in [1] are always trained from scratch, the networks in our work are adjusted from an existing distribution, and we show how this method drastically reduces the data requirement.

Furthermore, compared to [1], we provide novel experimentation and analysis of how patch-based diffusion models can avoid memorization and overfitting when being adapted to OOD data. While [1] showed that using patch-based models to solve inverse problems can lead to better results in settings of limited data, they did not explicitly illustrate that patch-based models can avoid issues of memorization and overfitting that whole image models run into. Our work tackles these issues directly through Figure 7 which illustrates that whole image models memorize the training data when fine-tuning from a very small dataset while patch-based models have greater generalizability. Figures 4 and 5 also illustrate this point: when fine-tuning the networks for a longer period of time, the quality of reconstructed images drops substantially for the whole image models while that of the patch-based models remains relatively steady.

In the revised paper, we also now provide novel theoretical analysis of the proposed algorithms that was not present in [1]. This theoretical analysis serves two purposes. Firstly, we show how in theory, using the patch-based model allows us to perform a type of data augmentation in the single measurement setting, which reinforces the point that patch-based models help guard against overfitting. Secondly, we provide more theoretical grounding for the self-supervised network refining method used in Algorithm 1 and show that this network refining process converges.

Comment: Lack of theoretical analysis

Response: We added theoretical analysis of Algorithm 1 in Section A.7.1. This provides a theoretical argument for why patch-based diffusion models should outperform whole image models in the single measurement setting. Furthermore, we show that the network refining process used in each diffusion iteration converges.

Comment: Errors in Eq. 2 and 3 and repaint Fig. 1

Response: We fixed the errors in eq. 2 and 3 and remade Fig. 1. These changes are highlighted in blue in the revision.

Comment: Can the authors provide some figures of training data

Response: We added more examples of the phantom training data in Appendix A.7.6 in Figures 31 and 32.

Comment: Why Fig. 6 does not have same notation as Fig. 3

Response: Fig. 3 shows the results of the single measurement setting while Fig. 6 shows the results of the small dataset setting. To avoid redundancy, we did not show the reconstructions in Fig. 6 that were also shown in Fig. 3. For instance, the reconstructions done using “whole, naive”, “patch, naive”, “whole, correct”, and “patch, correct” are the same for the single measurement setting and the small dataset setting (with the first two of these being done with the network trained on the out-of-distribution training data (phantom ellipse) and the latter two using the network trained on a large quantity of in-distribution data (CT images). To further reduce redundancy we opted not to show the results of every non-diffusion baseline that we ran in Fig. 6, with the baselines being ADMM-TV, PnP-ADMM, and PnP-RED. However, we still showed the best reconstruction out of these three baselines, labeled “Best baseline” in Fig. 6. The figure still shows that our proposed method obtains the best visual results in the small dataset setting.

Comment: Can the proposed method scale up

Response: To show that our method scales to larger images, we have added experiments on 512×512 images for 60 view CT reconstruction and deblurring with a 17×17 uniform kernel. We used the AAPM dataset for the CT experiments and the FFHQ dataset for the deblurring experiments. More details and the results can be found in Table 15 and Figures 27 and 28 of section A.7.3.

We thank the reviewer for their valuable feedback and insights.

Comment: Not clear why eq. (11) can address…

Response: Taking the specific example of our work where the network has initially been trained on ellipse phantoms, and we are trying to perform CT reconstruction. Thus, the testing data distribution (CT images) for the target task is not the same as training data distribution (ellipse phantoms images). At some diffusion time step t, the quantity $D(x_t|y)$ is roughly the expectation of the clean image given the measurement. Since the prior was learned over ellipse phantoms, this expectation will likely be close to an image consisting of such phantoms. However, such an image will not be close to satisfying data consistency, i.e., $y$ will not be close to $A \cdot D(x_t|y)$. Hence, by minimizing the loss function measuring the difference between $y$ and $A \cdot D(x_t|y)$ to update the network parameters, we force the prior learned by the network to be adjusted in such a way that the resulting expected clean image is closer to satisfying data consistency.

Comment: Proposed method is similar to DDIP3D

Response: Compared to DDIP3D, we provide novel experimentation and analysis of how patch-based diffusion models can avoid memorization and overfitting when being adapted to OOD data. In particular, DDIP3D uses 2D whole-image models to solve 3D image reconstruction problems, which tends to lead to overfitting. Our work elucidates these issues directly through Figure 7 which illustrates that whole-image models memorize the training data when fine-tuning from a very small dataset while patch-based models have greater generalizability. Figures 4 and 5 also illustrate this point: when fine-tuning the networks for a longer period of time, the quality of reconstructed images drops substantially for the whole image models while that of the patch-based models remains relatively steady.

In the single measurement setting, DIP-based self-supervised models such as those used in DDIP3D tend to overfit to the data. Therefore, DDIP3D used LoRA to limit the expressiveness of the network and prevent overfitting. We show in Table 10 that patch-based models can avoid overfitting to the measurement and that early stopping is not necessary when refining the network at each diffusion iteration, even when LoRA is not applied.

In the revised paper, we also now provide novel theoretical analysis of the proposed algorithms that was not present in DDIP3D. This theoretical analysis serves two purposes. Firstly, we show how in theory, using the patch-based model allows us to perform a type of data augmentation in the single measurement setting, which reinforces the point that patch-based models help guard against overfitting. Secondly, we provide more theoretical grounding for the self-supervised network refining method used in Algorithm 1 and show that this network refining process converges.

Comment: What would the results be like when applying the method to the black hole imaging problem where the true prior is unavailable

Response: In our work, for the single measurement setting, we are assuming the true prior is unavailable and we are only given a measurement from an unknown test distribution. This is similar to the problem setting of [1]. The approach of [1] is different in that it examines the reconstructed image under a variety of different possible assumptions for the prior, whereas we do not make any assumption on the true prior and simply refine our pretrained network based on the measurement. After our code is released, those with more domain expertise would be able to apply our method to the black hole imaging problem of [1]. We modified the introduction of the paper to clarify this point.

[1] Wu, Zihui, et al. "Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors." arXiv preprint arXiv:2405.18782 (2024).

We thank the reviewer for their valuable feedback and insights.

Comment: I am unsure about the contribution of the paper…the robustness to OOD data is important but the relation to the experiments on small datasets presented by [1] is unclear

Response: Compared to [1], the main difference is that the problem setting is different. The authors of [1] studied patch-based diffusion models in a traditional generative model setting: given a dataset of images, learn the distribution of those images. In our work, we studied diffusion models in an OOD setting, where no data (or very limited data) belonging to the test data distribution is available, so the goal is to adapt a pretrained diffusion model to a new distribution, either from a single measurement or a very small dataset. Therefore, while the networks in [1] are always trained from scratch, the networks in our work are adjusted from an existing distribution, and we show how this method drastically reduces the data requirement.

Furthermore, compared to [1] and SCD/DDIP, we provide novel experimentation and analysis of how patch-based diffusion models can avoid memorization and overfitting when being adapted to OOD data. While [1] showed that using patch-based models to solve inverse problems can lead to better results in settings of limited data, they did not explicitly illustrate that patch-based models can avoid issues of memorization and overfitting that whole-image models run into. Our work tackles these issues directly through Figure 7 that illustrates that whole-image models memorize the training data when fine-tuning from a very small dataset while patch-based models have greater generalizability. Figures 4 and 5 also illustrate this point: when fine-tuning the networks for a longer period of time, the quality of reconstructed images drops substantially for the whole image models while that of the patch-based models remains relatively steady.

In the single measurement setting, DIP-based self-supervised models such as those used in SCD/DDIP tend to overfit to the data. Therefore, SCD/DDIP used LoRA to limit the expressiveness of the network and prevent overfitting. Table 10 shows that patch-based models can avoid overfitting to the measurement and that early stopping is not necessary when refining the network at each diffusion iteration, even when LoRA is not applied.

The revised paper also now provides novel theoretical analysis of the proposed algorithms that was absent from previous works. This theoretical analysis serves two purposes. Firstly, we show how in theory, using the patch-based model allows us to perform a type of data augmentation in the single measurement setting, which reinforces the point that patch-based models help guard against overfitting. Secondly, we provide more theoretical grounding for the self-supervised network refining method used in Algorithm 1 and show that this network refining process converges.

Comment: The main performance gains are due to using SCD/DDIP and using the patch-based DMs increases the PSNR by at most 1db

Response: We acknowledge that SCD/DDIP already brings a performance improvement in whole-image models. But firstly, this is expected, as the main advantage of using diffusion models for solving inverse problems is that they provide a strong prior when the measurements are compressed and/or lossy. When the prior is incorrect, we lose this advantage completely and it is unsurprising that the reconstructed image quality is poor. Thus methods such as SCD/DDIP which adjust the network are expected to yield better results. Secondly, the advantage of using patch-based models is learning a better and more robust prior. In addition, patch-based diffusion models can avoid memorization and overfitting when being adapted to OOD data, as shown in Figure 7. Figures 4 and 5 also illustrate this point: when fine-tuning the networks for a longer period of time, the quality of reconstructed images drops substantially for the whole image models while that of the patch-based models remains relatively steady. Thus, in practice, early stopping is required for whole-image models to avoid a performance drop.

Comment: Using the patch-based DMs already increases the PSNR

While [1] indeed showed that the patch-based model improves performance over the whole image model, this was in settings with limited in-distribution training data in the scale of hundreds to thousands of samples, where the models are assumed to be trained from scratch. This is a different setting from our work, where we push the scale of the limited dataset to just 10 samples which makes it extremely hard to train any model from scratch. Thus, we also assume a pretrained diffusion prior is available from a different training domain where a large-scale dataset is available such as synthetic ellipsoid images. This also matches practical scenarios where a model pretrained on data-abundant domains can be utilized and transferred to help with model training in data-scarce domains.

In conclusion, our work shows that the patch-based model is more readily adapted to a different test distribution either via training on a very small dataset or a single measurement. Finally, the last two rows of Table 1 show the results of using the patch-based model and the whole-image model when a large amount of in-distribution training data was available (and no network refining was done on the fly). The results are very similar for these two models, which shows that when a large-scale dataset is available, both these models are able to learn a strong prior and may achieve similar performance for inverse problem solving.

Comment: Some parts of the argumentation are missing

Response: We have added more clarifications in Section 3.1 with the changes being highlighted in blue, so that the text is more self-contained. We also added some explanations behind the motivation for using the probability model.

Comment: How do the experiments and observations relate and complement those of [1]

Response: For Table 5 of the appendix of [1], the authors trained networks from scratch using datasets that consisted of images drawn from the same distribution as the test distribution. Hence, there was no distribution mismatch in that case, and the experiment showed that the patch-based model is more readily trained using limited data than the whole-image model. In our work, for the small dataset setting, we first trained both networks using a large quantity of data from the (typically synthetic) training distribution, and then fine-tuned the networks using an extremely small dataset from a different test distribution.

In addition, since the networks in [1] were trained from scratch, more data was required: the smallest datasets on which experiments were performed in [1] contained 144 images. In this work, we push the limit of the number of samples to only 10 images to fine-tune the network from a pretrained out-of-distribution prior. Consequently, due to the pretrained out-of-distribution prior and the reduction of training data, our model can converge much faster with significantly reduced training time: Figure 4 shows that we are able to fine-tune a patch-based model in only about 2 hours, while [1] required 12-24 hours to train the patch-based models. Thus, our results complement the work of [1] in the sense that: [1] shows patch-based diffusion models are easier to train from scratch in settings of limited data; this work shows that patch-based diffusion models are also easier to fine-tune from a pretrained model with few data samples. We clarified these points in section A.3 of the revised paper.