Few-Shot Diffusion Models Escape the Curse of Dimensionality

Thanks for your valuable comments and suggestions. We provide our response to each question below.

W1: The theoretical guarantee of the fully fine-tuned method.

As shown in our real experiment and [1], when fine-tuning all parameters with a small target dataset, models tend to overfit and lose the prior information from the pre-trained model. In our theorem, this phenomenon means that in the fine-tuning phase, the model does not use $\hat{\theta}$ learned by the pre-trained model and achieves a $n_{ta}^{-2/d}$ error bound, which suffers from the curse of dimensionality. From an intuitive perspective, the probability density function (PDF) of a distribution learned by an overfitting model is only positive at the interval around the target dataset, which is far away from the PDF of true distribution and leads to a large error term. We will add this proof and discussion in the next version.

We also note that it is possible to avoid this phenomenon by using a specific loss [1] or carefully choosing the optimization epochs [2]. We leave them as interesting future works.

W2: The discussion on the curse of the dimensionality.

With a very limited target dataset (such as 5-10 images), if a method can fine-tune the pre-trained model (trained with a large source dataset) and generate novel images with the target feature, we say that this models escape the curse of dimensionality. As shown in W1, the fully fine-tuned method has a large approximation error (theoretical guarantee) and suffers the memory phenomenon (experiments). On the contrary, our method can escape the curse of dimensionality from the theoretical (Thm. 4.3) and empirical perspective.

From the empirical perspective, as shown in Appendix E (dataset part), our experiments use $6400$ source data (CelebA64) to train a pre-trained model. Then, we only use $10$ target images to fine-tune this model and generate novel images with the target feature, which indicates our few-shot models escape the curse of dimensionality. For the cat dataset, we have a similar augmentation (4200 source data and $10$ target data). We note that $10$ target images are very limited compared to the source dataset (including CelebA64 and Cat dataset). Hence, our experiments support the theoretical results under each dataset.

We will add the above discussion and description of the source and target dataset in Sec. 6 to avoid confusion. Thanks again for the concerns and valuable comments on our real-world experiments.

Q1: A clearer discussion of the results of Thm. 4.3 and Table 1.

We note that the goal of the fine-tuning phase is to achieve the same order error bound compared with the pre-trained models, which means that we consider the relative relationship between $n_{ta}$ and $n_s$. Hence, if the coefficient of $n_{ta}$ and $n_s$ has the same order, we can only consider $1/\sqrt{n_{ta}}$ and $n_s^{-\frac{2-2\alpha\left(n_s\right)}{d+5}}$. To support the above augmentation, we first recall the results and calculate the coefficients:

The dominated term of coefficient for $n_{ta}$ and $n_s$ are $Dd^3/\delta$ and $\frac{d^3}{\delta^2c_0}$, respectively. The classic choice for the early stopping parameter $\delta$ and forward time $T$ are $10^{-3}$ and $10$, respectively [3]. Then, with $D=256\*256\*3$ as an example (Since smaller $D$ is more friendly to $n_{ta}$, our discussion holds for all datasets in Table 1.), $Dd^3/\delta = d^3\*20\*10^6$ and $\frac{d^3}{\delta^2c_0} = d^3\*10^6/c_0$, which has the same order. Hence, we consider the relative relationship between $1/\sqrt{n_{ta}}$ and $n_s^{-\frac{2-2\alpha\left(n_s\right)}{d+5}}$ and require

which indicates $n_{ta}^{\frac{d+5}{4(1-\alpha(n_s))}}\ge n_s$.

We now discuss the reason why the requirement on $n_{ta}$ is smaller when $d$ is larger. We note that the error bound of the pre-trained model is heavily influenced by the latent dimension $d$. More specifically, when $d$ is large (e.g. ImageNet), the error bound of pre-trained models has a large error even with large-size source data. We only need a few target data to achieve the same error. When $d$ is small (e.g. CIFAR-10), pre-trained models have a small error, and we need a slightly large target data size to achieve a small error.

We will add the above discussion in our next version to make it clearer.

[1] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR 2023.

[2] Li, P., Li, Z., Zhang, H., & Bian, J. (2024). On the generalization properties of diffusion models. Advances in Neural Information Processing Systems, 36.

[3] Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35, 26565-26577.

Thanks for your valuable comments and suggestions on our assumptions. We discuss each assumption in detail below and will make it clearer in our next version.

Q1: The discussion on each assumption.

(a) The linear subspace and shared latent assumption. (Assumption 3.1).

Since the diffusion models can find and adapt to the low-dimensional manifold [1], many theoretical works assume the linear low-dimensional manifold as the first step to understanding this phenomenon [2] [3], and we make exactly the same assumption compared with them.

For the analysis of few-shot learning, it is a standard, natural, and necessary assumption to assume the shared latent representation [4].

(a) The subgaussian latent variable (Assumption 4.1).

In this assumption, we assume the latent variable is a subgaussian variable. Since a bounded variable is a subgaussian one, this assumption is naturally satisfied by the real-world image datasets. We note that this assumption is more realistic than the Gaussian assumption and is widely used in many theoretical works of diffusion models [1] [2].

(b) The $\beta$-Lipschitz assumption on the score function (Assumption 4.2).

We note that the $\beta$-Lipschitz assumption is a standard assumption on the score function for the theoretical works [2] [5]. As shown in Sec. 3.2 of [5], for a subgaussian variable (Assumption 4.1), we can replace $\beta$ with $C_Z/\delta^2$ and remove this assumption. We will add the above discussion to our paper.

[1] Tang, R., & Yang, Y. (2024, April). Adaptivity of diffusion models to manifold structures. In International Conference on Artificial Intelligence and Statistics (pp. 1648-1656). PMLR.

[2] Chen, M., Huang, K., Zhao, T., & Wang, M. (2023, July). Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning (pp. 4672-4712). PMLR.

[3] Yuan, H., Huang, K., Ni, C., Chen, M., & Wang, M. (2024). Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. Advances in Neural Information Processing Systems, 36.

[4] Chua, K., Lei, Q., & Lee, J. D. (2021). How fine-tuning allows for effective meta-learning. Advances in Neural Information Processing Systems, 34, 8871-8884.

[5] Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., & Zhang, A. R. (2022). Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215.

Thanks for your valuable comments and suggestions. We provide our response to each question below.

W1: The discussion on the existing fine-tuning method.

In this part, we discuss the fully fine-tuned method and explain why we need data-efficient fine-tuning methods.

In earlier times, fully fine-tuned methods, such as DreamBooth [1], provided an important boost for developing few-shot models. However, they also show that the diffusion models suffer from the overfitting and memory phenomenon when fine-tuning all parameters (Fig. 6 of their paper). To deal with this problem, DreamBooth designs a prior preservation loss as a regularizer, which is needed carefully designed. Furthermore, a fully fine-tuned method is both memory and time inefficient [2].

To avoid the above problem, many works fine-tune some key parameters and achieve success in many areas such as text2image and medical image generation [3] [4]. These works not only preserve the prior information but also have a significantly smaller model size, which is more practical for real-world applications. Hence, we focus on these models in our work.

We will add the above discussion to our introduction part.

W2: The discussion on each assumption.

We note that Assumption 3.1 is the most important assumption for our analysis, and we discuss this assumption in detail below. The other assumptions are standard, and we discuss them one by one.

(a) The linear subspace and shared latent assumption. (Assumption 3.1).

Since the diffusion models can find and adapt to the low-dimensional manifold [5], many theoretical works assume the linear low-dimensional manifold as the first step to understand this phenomenon [6] [7] and we do exactly the same assumption compared with them. We note that since our analysis depends on the formula of score function under the linear subspace assumption, this assumption is important for our paper. An interesting future works is to extend our analysis to the nonlinear low-dimensional manifold and we will discuss it in our future work part.

For the analysis of few-shot learning, it is a standard, natural and necessary assumption to assume the shared latent representation [8] [9].

(b) The subgaussian latent variable and $\beta$-Lipschitz score function (Assumption 4.1 and 4.2).

Since a bounded variable is a subgaussian one, this assumption is naturally satisfied by the real-world image datasets and is widely used in many theoretical works [6] [7]. For the Lipschitz score, this assumption is common for the theoretical works of diffusion models [6] [10].

(c) The isotropic Gaussian latent variable assumption (Sec. 5).

When considering the few-shot optimization problem, we assume the latent variable is isotropic Gaussian, which is stronger than subgaussian one in Assumption 4.1. We note that this assumption is necessary for the closed form solution in Lem. 5.1. However, as discussed in future work paragraph, it is not necessary if we use some optimization algorithm instead of directly obtaining a closed-form.

Thanks for the concerns on our assumption. We will discuss our assumption in detail in the next version.

[1] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR 2023.

[2] Xiang, C., Bao, F., Li, C., Su, H., & Zhu, J. (2023). A closer look at parameter-efficient tuning in diffusion models. arXiv preprint arXiv:2303.18181.

[3] Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., & Yang, F. Svdiff: Compact parameter space for diffusion fine-tuning. ICCV 2023.

[4] Dutt, R., Ericsson, L., Sanchez, P., Tsaftaris, S. A., & Hospedales, T. Parameter-efficient fine-tuning for medical image analysis: The missed opportunity. arXiv preprint arXiv:2305.08252.

[5] Tang, R., & Yang, Y. (2024, April). Adaptivity of diffusion models to manifold structures. In International Conference on Artificial Intelligence and Statistics (pp. 1648-1656). PMLR.

[6] Chen, M., Huang, K., Zhao, T., & Wang, M. (2023, July). Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning (pp. 4672-4712). PMLR.

[7] Yuan, H., Huang, K., Ni, C., Chen, M., & Wang, M. (2024). Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. Advances in Neural Information Processing Systems, 36.

[8] Du, S. S., Hu, W., Kakade, S. M., Lee, J. D., & Lei, Q. (2020). Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.09434.

[9] Chua, K., Lei, Q., & Lee, J. D. (2021). How fine-tuning allows for effective meta-learning. Advances in Neural Information Processing Systems, 34, 8871-8884.

[10] Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., & Zhang, A. R. (2022). Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215.

Thanks for your valuable comments and suggestions. We provide our response to each question below.

W1 & Presentation.

Thanks again for the valuable comments. Before using the approximation error to represent the score matching loss with finite datasets, we will make a clear definition in the notation paragraph to avoid confusion. We will also polish our presentation according to your suggestions.

W2 (a) A clearer discussion of the results of Thm. 4.3.

We note that the goal of the fine-tuning phase is to achieve the same order error bound compared with the pre-trained models, which means that we consider the relative relationship between $n_{ta}$ and $n_s$. Hence, if the coefficient of $n_{ta}$ and $n_s$ has the same order, we can only consider $1/\sqrt{n_{ta}}$ and $n_s^{-\frac{2-2\alpha\left(n_s\right)}{d+5}}$. To support the above augmentation, we first recall the results and calculate the coefficients:

The dominated term of coefficient for $n_{ta}$ and $n_s$ are $Dd^3/\delta$ and $\frac{d^3}{\delta^2c_0}$, respectively. The classic choice for the early stopping parameter $\delta$ and forward time $T$ are $10^{-3}$ and $10$, respectively [1]. Then, with $D=256\*256\*3$ as an example (Since smaller $D$ is more friendly to $n_{ta}$, our discussion holds for all datasets in Table 1.), $Dd^3/\delta = d^3\*20\*10^6$ and $\frac{d^3}{\delta^2c_0} = d^3\*10^6/c_0$, which has the same order. Hence, we consider the relative relationship between $1/\sqrt{n_{ta}}$ and $n_s^{-\frac{2-2\alpha\left(n_s\right)}{d+5}}$. We will add the above discussion and provide a clearer explanation.

W2 (b) The discussion of Table 1.

As shown in W2 (a), the fine-tuning phase aims to achieve the same error bound compared with source data. To achieve this goal, the first step is to determine the error bound of the pre-trained model. We note that this error bound is heavily influenced by the latent dimension $d$. More specifically, when $d$ is large (e.g. ImageNet), the error bound of pre-trained models has a large error even with large-size source data. We only need a few target data to achieve the same error. When $d$ is small (e.g. CIFAR-10), pre-trained models have a small error, and we need a slightly large target data size to achieve a small error.

W2 (c): Directly training for small $d$.

For $d\leq 4$, the models achieve $1/\sqrt{n_{ta}}$ without the prior information from the source data. However, $d$ of common image datasets is larger than $20$ (Table 1). Hence, it is meaningful to consider our few-shot fine-tuning process, which fully uses prior information.

W3 (b): The discussion on fully fine-tuned methods.

In earlier times, fully fine-tuned methods, such as DreamBooth [2], provided an important boost for developing few-shot models. However, they also show that the diffusion models suffer from the overfitting and memory phenomenon when fine-tuning all parameters (Fig. 6 of their paper). To deal with this problem, they design a prior preservation loss as a regularizer, which needs to be carefully designed. Furthermore, a fully fine-tuned method is both memory and time inefficient [6].

To avoid the above problem, many works fine-tune some key parameters and achieve success in many areas, such as text2image and medical image generation [3] [4]. These works not only preserve the prior information but also have a significantly smaller model size, which is more practical for applications. Hence, we focus on these models in our work. We will add the above discussion to our introduction.

W3 (a): The bound for fully fine-tuned methods.

As discussed above, the fully fine-tuned method tend to overfit and lose the prior information from the pre-trained model. In our theorem, this phenomenon means that in the fine-tuning phase, the model does not use $\hat{\theta}$ learned by the pre-trained model and achieves a $n_{ta}^{-2/d}$ error bound, which suffers from the curse of dimensionality. From an intuitive perspective, the probability density function (PDF) of a distribution learned by an overfitting model is only positive at the interval around the target dataset, which is far away from the PDF of true distribution and leads to a large error term.

We also note that it is possible to avoid this phenomenon by using a specific loss [2] or carefully choosing the optimization epochs [5]. We leave them as interesting future works.

Q1: The optimization problem with a fixed $t$.

When considering a linear subspace, the diffusion process happens at the latent space, and $A_{ta}$ is independent of $t$. Hence, we fix a $t$ to solve the optimization problem. As discussed in Sec. 5.1, the objective is more flexible than traditional PCA since it can choose suitable $t$ to avoid the influence of large $\lambda^2$.

[1] Karras, T., Aittala, M., Aila, T., & Laine, S. Elucidating the design space of diffusion-based generative models. NeurIPS 2022.

[2] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR 2023.

[3] Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., & Yang, F. Svdiff: Compact parameter space for diffusion fine-tuning. ICCV 2023.

[4] Dutt, R., Ericsson, L., Sanchez, P., Tsaftaris, S. A., & Hospedales, T. Parameter-efficient fine-tuning for medical image analysis: The missed opportunity. arXiv preprint arXiv:2305.08252.

[5] Li, P., Li, Z., Zhang, H., & Bian, J. On the generalization properties of diffusion models. NeurIPS 2023.

[6] Xiang, C., Bao, F., Li, C., Su, H., & Zhu, J. (2023). A closer look at parameter-efficient tuning in diffusion models. arXiv preprint arXiv:2303.18181.