Scaling Laws for Diffusion Transformers

Q6: Model size

A6: Regarding the model size, our experiments focus on models around 1B parameters. This aligns with the majority of open-sourced diffusion models, which are typically under 10B parameters, placing our model within the same order of magnitude. The primary goal of our paper is not to train a SOTA-sized diffusion model but rather to demonstrate the existence of scaling laws for diffusion models and provide a guideline on how to compute and utilize the scaling laws.

To address the reviewer’s concerns, we suggest drawing a parallel to the development of scaling laws in the LLM domain. Scaling laws are a developing field, and the exploration of their existence and limitations has historically been incremental. For example, the initial verification of scaling laws for LLMs was conducted with models at the 600M parameter scale [1]. Over time, this expanded to tens of billions of parameters and eventually to the current 400B or more scale [5, 6]. Notably, when the first paper on scaling laws for LLM[1] was published, the largest model, GPT-3[7], had 175B parameters, creating a size ratio of 175B/0.6B ≈ 291 between SOTA models and the scaling law benchmarks.

According to most reviewers’ comments and the best of our knowledge, our work is the first work to verify the existence of scaling laws in Diffusion Transformers. Most open-sourced diffusion models today are below 10B parameters, and our 1B-scale models are much closer to SOTA in this domain compared with the first few works in LLM (size ratio 291), falling within the same order of magnitude. Scaling law validation is a step-by-step process, and the scale of models used increases over time as the field progresses.

Besides, training a 3B diffusion model would require approximately 676B tokens and close to a month of training, while a 10B model would require 2.2T tokens and several months of training. This level of resource demand far exceeds the affordance of a research project. Our work focuses on proving the existence of scaling laws and providing a guideline, not on training SOTA diffusion models. Larger-scale scaling law validation, like in the LLM field, will require further exploration in future studies.

Q7: Data assessment

A7: We conducted several experiments on a dataset with sparse captions to demonstrate how scaling laws can be used to assess data quality. The images in this dataset are identical to those in the dataset used in our main study, with the captions being the only variable. Following the procedure outlined in the main paper, we fit scaling laws for FID on the sparse-captioned dataset. The results are presented in Supplementary Material Section G.6 of the supplementary materials, where the scaling laws exhibit a clear linear trend. The equation for the fitted line is 1.059 * 10^6 * x^{-0.216}. To compare data quality, we analyzed the scaling exponents. In the main text, the exponent for the dense-captioned dataset is -0.234, which is lower than -0.216. This indicates that the FID for the dense-captioned dataset decreases more rapidly, reflecting better performance. Thus, we conclude that dense captions are superior to sparse captions, consistent with our empirical observations.

[1] Scaling Laws for Neural Language Models. Kaplan et al. https://arxiv.org/abs/2001.08361

[2] PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. Chen et al. https://arxiv.org/abs/2310.00426

[3] Training Compute-Optimal Large Language Models. Hoffmann et al. https://arxiv.org/abs/2203.15556

[4] DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. Bi et al. https://arxiv.org/abs/2401.02954

[5] GPT-4 Technical Report. OpenAI. https://arxiv.org/abs/2303.08774

[6] The Llama 3 Herd of Models. Dubey et al. https://arxiv.org/abs/2407.21783

[7] Language Models are Few-Shot Learner. Brown et al. https://arxiv.org/abs/2005.14165

We thank the reviewer for detailed comments and feedback. We respond to your concern related to our paper in the following part.

Q1: Narrow compute budget range.

A1: Compared to [1], the range of our compute budget is indeed smaller. However, as referenced in [3], spanning nearly three orders of magnitude is sufficient to fit accurate scaling laws. Additionally, as demonstrated in our experiments, larger computing budgets require significantly more computational resources. For example, training a model corresponding to a compute budget of 1e23 would likely require over 10 billion parameters, which is beyond the scope of a typical research project due to the excessive computational demands. A broader range of compute range will result in more points for scaling laws estimation and yield more accurate estimation. However, it will not affect the existence of the scaling laws and the correctness of our methods on how to estimate scaling laws and how to use them.

Q2: Insufficient experiments compared with [1]

A2: We acknowledge that the experiments in our paper are not as extensive as those in [1]. The primary focus of our work is to demonstrate the existence of scaling laws for diffusion transformers and provide a guideline on how to predict the scaling laws and utilize the scaling laws. The experiments in our paper are sufficient to support these claims. Additionally, we have supplemented our study with experiments on different model architectures (Flux Supp Section G.1, Pixart Supp Section G.2), image resolutions (Supp Section G.3), and captions (Supp Section G.4), showing that the existence of scaling laws does not depend on specific transformer designs or data configurations. While [1] includes numerous experiments that provide a more comprehensive view of scaling laws, these additional studies do not affect the fundamental conclusion of the scaling laws’ existence and the guideline, which is the core focus of our paper. We plan to explore these aspects further in future work.

Q3: Some images in the paper only have one or two curves. Does this imply that it is already experimenting with the optimal model parameters calculated by the scaling law?

A3: Yes, some figures indeed use the optimal models at each budget to create the curves. This is because the purpose of each figure varies, and the visualized model results differ accordingly. In Figure 1 (left), we test models of different sizes under each compute budget to determine the optimal model size and token number for that budget. So, we plot the results of all models and pick the optimal one. For Figures 2 and 3, our goal is to predict the optimal performance at each budget. The optimal performances are achieved by the optimal models. Therefore, the results shown correspond to the loss achieved by the optimal model at each budget, and only these values are plotted.

Q4: Model architecture.

A4: First, based on prior research on large language models (LLMs)[1](Figure 5.), the existence of scaling laws for transformer-based models is generally independent of the specific architectural design. While the model architecture may influence the coefficients and exponents of the scaling laws, it does not affect their existence. To address the reviewer’s concern, we conducted a series of experiments to validate our conclusions. In the main text, we have already demonstrated this through experiments on PixArt [2], and we further provide predictions of the loss in Supplementary Section G.2. Additionally, we extended our validation to the latest DiT architecture by evaluating its performance under the Flux budget. The results are included in Supplementary Section G.1 as well. In both cases, as shown in the figures provided, there is a clear linear relationship between the compute budget and the loss, which strongly supports the presence of scaling laws.

Q5: Resolution

A5: All experiments in our paper use images with a resolution of [256, 256]. To address the reviewer’s concern regarding resolution, we also conducted experiments using images with a resolution of 512. The results are presented in Supplementary Section G.3. From the figure, we observe that at each training budget, the performance of models with varying sizes can be accurately fitted using a parabola. Moreover, the loss of the optimal model under each training budget also follows a linear relationship, further confirming the existence of scaling laws.

Dear Reviewer CvMz,

We have responded to your concerns and would be happy to clarify further if needed. Please let us know if you have any additional questions or points for discussion.

Thank you for your time and feedback. Wishing you a great day!

Best regards,

The Authors

Thank you for your valuable feedback. We appreciate the reviewer’s concerns and agree that the conclusions presented in our paper require refinement. Specifically, we acknowledge that our statement suggesting "in-context learning is worse than cross-attention" is problematic and stems from imprecise wording. To clarify, the in-context method used in our study should be referred to as vanilla in-context, which concatenates all images, text, and time tokens and processes them using standard transformers. This approach is distinct from more advanced methods such as FLUX or MMDIT, as correctly highlighted by the reviewer.

We recognize that our language in the paper was not sufficiently precise. The appropriate conclusion should state that vanilla in-context learning is less effective than cross-attention in terms of scalability, rather than broadly asserting that in-context learning is inferior to cross-attention. This correction will be made in future revisions.

Another issue is related to the wording used in the comparison presented in Section 4, which focuses on the scalability of different model architectures. By analyzing the coefficients of the FID and loss scaling curves, our intent was to evaluate which architecture performs better when scaled up. However, we acknowledge that the conclusions may appear oversimplified. For instance, as shown in Figure 5 of our paper, there is a crossing point where performance changes depending on the model architecture and available training budget. Without specifying training budgets, such conclusions can be misleading. Generally, as we increase training budgets—a common scenario of interest—cross-attention outperforms vanilla in-context learning. This finding aligns with results from both the original DiT paper [1] and PixArt [2], which demonstrate that cross-attention achieves superior performance when trained with very large budgets. We will address and clarify this in subsequent versions of our paper.

We also agree with the reviewer that modern in-context learning architectures, such as FLUX or MMDIT, may perform better. However, comparing these advanced designs was outside the scope of our current study. Vanilla in-context learning and FLUX/MMDit in-context learning differ substantially in their architectural designs, including dual-branch versus single-branch structures, additional text conditions, and other advanced techniques. The design space for DiT remains vast and continues to evolve. For instance, while PixArt [2] illustrates the advantages of cross-attention over vanilla in-context learning, advanced in-context approaches like MMDIT (SD3) and FLUX have shown improved performance. Looking forward, it is possible that future iterations of DiT will see enhanced cross-attention designs reclaiming the advantage. Therefore, at this stage, we believe that comparisons should focus on specific model architectures, evaluated individually. A definitive conclusion about the relative merits of in-context learning versus cross-attention is premature.

In Section 4 of our paper, our goal was to demonstrate how scaling laws can be used to assess the scalability of different architectures on a case-by-case basis. The conclusions drawn are specific to the architectures analyzed—namely, that vanilla in-context learning is less effective than cross-attention when scaling up—and should not be interpreted as a general critique of in-context mechanisms. We will ensure that this distinction is clearly expressed in future revisions.

[1] Scalable Diffusion Models with Transformers. Peebles et al. https://arxiv.org/abs/2212.09748

[2] PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. Chen et al. https://arxiv.org/abs/2310.00426

We thank the reviewer for detailed comments and feedback. We respond to your concern related to our paper in the following part.

Q1: Different setting

A1: In our main paper, following [1] we chose a basic Transformer model to validate our conclusions, aiming to ensure the simplicity of our experiments. [1] demonstrates that the existence of scaling laws does not rely on model architecture or dataset setup. In response to the reviewer’s comments, we further validated the scaling laws on the latest DiT architecture. As demonstrated in Supplement Material Section G, we tested our findings on PixArt (Supp Section G. 2), and Flux (Supp Section G. 2), and also explored the impact of different resolutions (Supp Section G. 3) and captions (Supp Section G. 6). Across various settings, our experiments show that the existence of scaling laws is independent of the Transformer architecture design and dataset characteristics. This paper provides a guideline on how to compute the scaling laws and how to use them. The specific model and data will not affect the scaling laws for diffusion transformers. However, the exponent and coefficient will be affected by the model and data and thus can not be directly transferred to other datasets or model architecture. The exact scaling laws on other datasets or models should be re-computed.

Q2: Model size

A2: Regarding the model size, our experiments focus on models around 1B parameters. This aligns with the majority of open-sourced diffusion models, which are typically under 10B parameters, placing our model within the same order of magnitude. The primary goal of our paper is not to train a SOTA-sized diffusion model but rather to demonstrate the existence of scaling laws for diffusion models and provide a guideline on how to compute and utilize the scaling laws. To address the reviewer’s concerns, we suggest drawing a parallel to the development of scaling laws in the LLM domain. Scaling laws are a developing field, and the exploration of their existence and limitations has historically been incremental. For example, the initial verification of scaling laws for LLMs was conducted with models at the 600M parameter scale [1]. Over time, this expanded to tens of billions of parameters and eventually to the current 400B or more scale [2, 3]. Notably, when the first paper on scaling laws for LLM[1] was published, the largest model, GPT-3[4], had 175B parameters, creating a size ratio of 175B/0.6B ≈ 291 between SOTA models and the scaling law benchmarks.

According to most reviewers’ comments and the best of our knowledge, our work is the first work to verify the existence of scaling laws in Diffusion Transformers. Most open-sourced diffusion models today are below 10B parameters, and our 1B-scale models are much closer to SOTA in this domain compared with the first few works in LLM, falling within the same order of magnitude. Scaling law validation is a step-by-step process, and the scale of models used increases over time as the field progresses. To put this into perspective, training a 3B diffusion model would require approximately 1.8T data tokens and close to a month of training, while a 10B model would require 6T data tokens and several months of training. This level of resource demand far exceeds the affordance of a research project. Our work focuses on proving the existence of scaling laws and providing a guideline, not on training SOTA diffusion models. Larger-scale scaling law validation, like in the LLM field, will require further exploration in future studies.

[1] Scaling Laws for Neural Language Models. Kaplan et al. https://arxiv.org/abs/2001.08361

[2] GPT-4 Technical Report. OpenAI. https://arxiv.org/abs/2303.08774

[3] The Llama 3 Herd of Models. Dubey et al. https://arxiv.org/abs/2407.21783

[4] Language Models are Few-Shot Learner. Brown et al. https://arxiv.org/abs/2005.14165

Dear Reviewer CVDQ,

We have responded to your concerns and would be happy to clarify further if needed. Please let us know if you have any additional questions or points for discussion.

Thank you for your time and feedback. Wishing you a great day!

Best regards,

The Authors

Dear Reviewer CVDQ,

As the rebuttal deadline approaches, we would greatly appreciate hearing from you regarding any additional concerns or feedback you may have. We are more than willing to address them and engage in further discussions during the rebuttal phase.

Thank you for your time and consideration.

Best regards,

The Authors

Dear reviewer CVDQ,

Today is the last day for discussion. We look forward to getting feedback from you regarding our response. We would love to try our best to address your concerns and do further discussion.

Best,

The authors

We thank the reviewer for detailed comments and feedback. We respond to your concern related to our paper in the following part.

Q1: Model size

A1: Regarding the model size, our experiments focus on models around 1B parameters. This aligns with the majority of open-sourced diffusion models, which are typically under 10B parameters, placing our model within the same order of magnitude. The primary goal of our paper is not to train a SOTA-sized diffusion model but rather to demonstrate the existence of scaling laws for diffusion models and provide a guideline on how to compute and utilize the scaling laws. To address the reviewer’s concerns, we suggest drawing a parallel to the development of scaling laws in the LLM domain. Scaling laws are a developing field, and the exploration of their existence and limitations has historically been incremental. For example, the initial verification of scaling laws for LLMs was conducted with models at the 600M parameter scale [1]. Over time, this expanded to tens of billions of parameters and eventually to the current 400B or more scale [2, 3]. Notably, when the first paper on scaling laws for LLM[1] was published, the largest model, GPT-3[4], had 175B parameters, creating a size ratio of 175B/0.6B ≈ 291 between SOTA models and the scaling law benchmarks.

According to most reviewers’ comments and the best of our knowledge, our work is the first work to verify the existence of scaling laws in Diffusion Transformers. Most open-sourced diffusion models today are below 10B parameters, and our 1B-scale models are much closer to SOTA in this domain compared with the first few works in LLM, falling within the same order of magnitude. Scaling law validation is a step-by-step process, and the scale of models used increases over time as the field progresses. To put this into perspective, training a 3B diffusion model would require approximately 1.8T data tokens and close to a month of training, while a 10B model would require 6T data tokens and several months of training. This level of resource demand far exceeds the affordance of a research project. Our work focuses on proving the existence of scaling laws and providing a guideline, not on training SOTA diffusion models. Larger-scale scaling law validation, like in the LLM field, will require further exploration in future studies.

Q2: Inaccurate reference work.

A2: We thank the reviewer for pointing out the inaccuracy of our reference work. We have modified the paper and fixed the discussion on masked diffusion models. (MDT v1 & v2). The discussion can be found in Supplementary Material Section G. 4. We will merge the discussion into the related work part later.

Q3: Scaling Properties.

A3: The primary scope of this paper is to demonstrate the existence of scaling laws in diffusion transformers outline how these laws are computed how we can use these laws as a benchmark to evaluate the model/data design. Instead of providing the scaling properties for a specific model architecture, we provide a general method for users of scaling laws to find the scaling properties of their models. The scaling properties are normally coupled with specific model designs and should be analyzed case-by-case. Exhausting all of them are computationally intensive and beyond our scope. We leave it to future work.

[1] Scaling Laws for Neural Language Models. Kaplan et al. https://arxiv.org/abs/2001.08361

[2] GPT-4 Technical Report. OpenAI. https://arxiv.org/abs/2303.08774

[3] The Llama 3 Herd of Models. Dubey et al. https://arxiv.org/abs/2407.21783

[4] Language Models are Few-Shot Learner. Brown et al. https://arxiv.org/abs/2005.14165

Dear Reviewer ssc5,

We have responded to your concerns and would be happy to clarify further if needed. Please let us know if you have any additional questions or points for discussion.

Thank you for your time and feedback. Wishing you a great day!

Best regards,

The Authors

Dear Reviewer ssc5,

As the rebuttal deadline approaches, we would greatly appreciate hearing from you regarding any additional concerns or feedback you may have. We are more than willing to address them and engage in further discussions during the rebuttal phase.

Thank you for your time and consideration.

Best regards,

The Authors

Dear reviewer ssc5,

Today is the last day for discussion. We look forward to getting feedback from you regarding our response. We would love to try our best to address your concerns and do further discussion.

Best,

The authors

Q5: Model size

A5: Regarding the model size, our experiments focus on models around 1B parameters. This aligns with the majority of open-sourced diffusion models, which are typically under 10B parameters, placing our model within the same order of magnitude. The primary goal of our paper is not to train a SOTA-sized diffusion model but rather to demonstrate the existence of scaling laws for diffusion models and provide a guideline on how to compute and utilize the scaling laws.

To address the reviewer’s concerns, we suggest drawing a parallel to the development of scaling laws in the LLM domain. Scaling laws are a developing field, and the exploration of their existence and limitations has historically been incremental. For example, the initial verification of scaling laws for LLMs was conducted with models at the 600M parameter scale [1]. Over time, this expanded to tens of billions of parameters and eventually to the current 400B or more scale [10, 11]. Notably, when the first paper on scaling laws for LLM[1] was published, the largest model, GPT-3[12], had 175B parameters, creating a size ratio of 175B/0.6B ≈ 291 between SOTA models and the scaling law benchmarks.

According to most reviewers’ comments and the best of our knowledge, our work is the first work to verify the existence of scaling laws in Diffusion Transformers. Most open-sourced diffusion models today are below 10B parameters, and our 1B-scale models are much closer to SOTA in this domain compared with the first few works in LLM, falling within the same order of magnitude. Scaling law validation is a step-by-step process, and the scale of models used increases over time as the field progresses.

Training a 3B diffusion model would require approximately 1.8T data tokens and close to a month of training, while a 10B model would require 6T data tokens and several months of training. This level of resource demand far exceeds the affordance of a research project. Our work focuses on proving the existence of scaling laws and providing a guideline, not on training SOTA diffusion models. Larger-scale scaling law validation, like in the LLM field, will require further exploration in future studies.

Q7: Training loss and FID?

A7: Training loss and FID are indeed different. They focus on different aspects of evaluations but are strongly correlated. In our paper, we demonstrate the existence of scaling laws both for loss and FID. We can predict the large model’s FID using small-scale models. Training loss serves as both a reliable predictor and an evaluation metric. As demonstrated in SD3 [4] Figure 8, validation loss strongly correlates with overall model performance, showing a significant relationship between loss values and holistic image evaluation metrics. In our study, Figure 2 illustrates that training loss and validation loss exhibit similar trends and shapes, confirming that scaling laws can accurately predict validation loss. Consequently, training or validation loss can be used as an indicator of a model's quality. Scaling laws also provide guidance for identifying the optimal model size, allowing us to determine the appropriate parameter range. However, model parameters are typically discrete, making it impractical to precisely match the optimal size suggested by the scaling laws. From Figure 1, we observe that loss does not change significantly in the vicinity of the optimal point, ensuring that selecting a size close to the predicted optimal still yields near-optimal performance. However, for diffusion models, even a small difference in loss, such as 0.05, can result in substantial performance variations. Therefore, if the chosen model size deviates too far from the optimal, the seemingly minor numerical loss differences can translate into significant practical performance disparities.

Q8: Writing quality.

A8: We thank the reviewer for pointing out the writing issues in our paper. We will improve the writing to make the paper more clear.

We thank the reviewer for detailed comments and feedback. We respond to your concern related to our paper in the following part.

Q1: The use of FLOPs and tokens.

A1: We appreciate the reviewer’s feedback and understand the preference for GPU hours and image counts as intuitive indicators for computing budgets and data. While these metrics are indeed related to FLOPs and token counts, there are several reasons why we chose to use the latter:

1. Consistency with Previous Work: In prior studies on scaling laws for large language models (LLMs)[1][2][3], FLOPs and token counts are commonly used as standard metrics. By adopting these measures, we ensure consistency with the established literature and enable a direct comparison with related works.

2. Practical Limitations of GPU Hours as a Metric: While GPU hours can provide a sense of the computational effort, they are not a precise indicator of training capacity due to their strong dependency on the specifics of the infrastructure. GPU hours depend on factors such as the type of GPUs used, the optimization level of the compute cluster, and the utilization efficiency. For example, to translate FLOPs into GPU hours accurately, one would need to account for the Model FLOPs Utilization (MFU), which reflects how efficiently GPUs are used in practice. Since MFU is highly dependent on engineering implementations and often varies across systems, GPU hours alone cannot universally represent the compute budget in a reproducible way.

3. Tokens as a Generalized Data Representation: Tokens provide a universal representation of the data processed by transformer-based models, as all input modalities (text, visual content, audio) are eventually tokenized. Therefore, token count serves as a more comprehensive and consistent metric for scaling compute analysis.

4. Tokens are not equal to image count. While image count may be a straightforward metric for fixed-resolution image datasets, it does not fully capture the scale of training data in many scenarios. In our case, we also include text tokens, which contribute significantly to the total compute cost during training. Moreover, in industrial-scale text-to-image models[4][5], training typically involves a mix of images at different resolutions. For video generation models, videos of varying lengths. These variations make it difficult to directly equate image or video counts to the total data processed. Our paper serves for a broad range of diffusion tasks including videos and images. Therefore tokens are more suitable to represent the amount of compute.

Q2: Validation on modern DiT architecture.

A2: First, based on prior research on large language models (LLMs)[1], the existence of scaling laws for transformer-based models is generally independent of the specific architectural design. While the model architecture may influence the coefficients and exponents of the scaling laws, it does not affect their existence. To address the reviewer’s concern, we conducted a series of experiments to validate our conclusions. In the main text, we have already demonstrated this through experiments on PixArt [6], and we further provide predictions of the loss in Supplementary Material Section G.2. Additionally, we extended our validation to the latest DiT architecture by evaluating its performance under the Flux budget. The results are included in Supplementary Material Section G.1. In both cases, as shown in the figures provided, there is a clear linear relationship between the compute budget and the loss, which strongly supports the presence of scaling laws.

Dear Reviewer 2p8z,

We have responded to your concerns and would be happy to clarify further if needed. Please let us know if you have any additional questions or points for discussion.

Thank you for your time and feedback. Wishing you a great day!

Best regards,

The Authors

Dear Reviewer 2p8z,

As the rebuttal deadline approaches, we would greatly appreciate hearing from you regarding any additional concerns or feedback you may have. We are more than willing to address them and engage in further discussions during the rebuttal phase. Besides, we have some more discussions with other reviewers, and perhaps some of them can also help solve your concerns.

Thank you for your time and consideration.

Best regards,

The Authors

Dear reviewer 2p8z,

Today is the last day for discussion. We look forward to getting feedback from you regarding our response. We would love to try our best to address your concerns and do further discussion.

Best,

The authors