ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models

Q2. Since the dispersed convolution is a dense operation with a larger receptive field, it introduces additional computational costs. How significant is the computational overhead of the method?

Please refer to our response to reviewer URVR Q2. Thank you!

Q3. Is the re-dilation described in Sec. 3.2 actually used in the method?

Yes, the re-dilation is used in all the settings. However, when the resolution exceeds 1280x1280, we need to further incorporate the noise-damped classifier-free guidance (ndcfg), and the dispersed convolution. We have documented all these details in the hyperparameters section of the supplementary. We also clarify this in the revision of the main paper.

Q4. Is it possible to find a closed-form solution for the optimization problem in Eq. 4?

There is a closed-form solution for dispersed convolution. One only needs to solve a least square problem and no training is needed. The detailed method is as follows.

We follow the notation in our main paper. Given a convolution layer with kernel $\boldsymbol{k} \in \mathbb{R}^{r \times r}$ and a target kernel size $r'$. We find a dispersion transform $\mathbf{R} \in \mathbb{R}^{r'^2 \times r^2}$ to get a dispersed kernel $\boldsymbol{k}'$. Consider an input feature map $\boldsymbol{h} \in \mathbb{R}^{n \times n}$, we ignore the bias in convolution, since it will not influence our results.

Structure-level calibration and pixel-level calibration can form an equation set

\tag{1}$$ $ interp_d (f_k(\boldsymbol{h})) \in \mathbb{R}^{nd \times nd}, f_{\boldsymbol{k}} (\boldsymbol{h}) \in \mathbb{R}^{n \times n} $. The equation set has $(nd)^2 + n^2$ equations. Each equation is made up of the sum of terms $k_{ij}h_{ij}$ (elements in $\boldsymbol{k}$ and $\boldsymbol{h}$) where the coefficient of the terms are linear combinations of $R_{ij}$ (elements in $\mathbf{R}$) and constants. For example, when $n=3, r=3$ and $r'=5$, the 5-$th$ equation in pixel-level calibration is

\begin{matrix} k_{11} h_{11} + k_{12} h_{12} + k_{13} h_{13} + \cdots + k_{33} h_{33} = \\ (R_{7,1} k_{11} + R_{7,2} k_{12} + R_{7,3} k_{13} + \cdots + R_{7,9} k_{33})h_{11} + \\ (R_{8,1} k_{11} + R_{8,2} k_{12} + R_{8,3} k_{13} + \cdots + R_{8,9} k_{33})h_{12} + \\ \cdots + \\ (R_{19,1} k_{11} + R_{19,2} k_{12} + R_{19,3} k_{13} + \cdots + R_{19,9} k_{33})h_{33} \end{matrix} \tag{2}$$ We rearrange the equation and get

$ (1 - R_{7,1}) k_{11} h_{11} + (-R_{7,2})k_{12} h_{11} + \cdots (1 - R_{19, 9})k_{33}h_{33} = 0 \notag. $ \tag{3} $$ To ensure this equation for all $k_{ij}h_{ij}$, one can let every coefficient be zero. Getting a linear equation set for the 5-$th$ equation in pixel-level calibration: $$ \left\\{\begin{matrix} R_{7, 1} = -1 \\\ R_{7, 2} = 0 \\\ \cdots \\\ R_{19, 9} = -1 \end{matrix} \right. \tag{4}$$ We do this for every equation in Eqn.(1) to derive a larger set of linear equations of $R_{ij}$. Note that the equation sets are linear equations of $R_{ij}$ and have no $\boldsymbol{h}$ or $\boldsymbol{k}$, making it applicable to all conv kernels and any input feature. Lets back to general case where $\mathbf{R} \in \mathbb{R}^{r'^2 \times r^2}$. Let $A_{\mathrm{structure}}$ denote the coefficient of linear combination for $R_{ij}$ in structure-level calibration and let $b_{\mathrm{structure}}$ denote the right-hand-side constants in the equation set of structure-level calibration. Similarly, we define $A_{\mathrm{pixel}}, b_{\mathrm{pixel}}$. We construct a least square problem $$A\mathbf{x} = b, \quad A = \left[\begin{matrix}A_{\mathrm{structure}} \\\ \eta A_{\mathrm{pixel}}\end{matrix} \right], b = \left[\begin{matrix}b_{\mathrm{structure}} \\\ \eta b_{\mathrm{pixel}}\end{matrix} \right], \mathbf{x} = \left[\begin{matrix} R_{1, 1} \\\ R_{1, 2} \\\ \cdots \\\ R_{r'^2, r^2}\end{matrix}\right]. \tag{5}$$ The solution is $\mathbf{x} = (A^T A)^{-1} A^T b$. This can be easily solved by math software, i.e., MATLAB. We have included this part in the supplementary materials.

Thank you for providing useful feedback. Below are our responses for each point.

Q1. The section on "re-dilation" mentions the use of a predefined dilation schedule function D(t,l). How is this function defined, and have any experiments been conducted to validate or optimize its selection?

We define this function via both manual effort and automatic hyperparameter search. Once the hyperparameters are settled on a very few images, they can be used for all other samples.

We introduce the details of the re-dilation schedule as follows: We first test a series of hand-crafted hyperparameters and figure out some empirical rules:

• Using re-dilation/dispersion in inner UNet blocks and using original convolution in outer UNet blocks produces good results. Because outer blocks are responsible for the low-level denoising rather than keeping the geometry structures.
• Sharing the same re-dilation scale in a block instead of using different re-dilation scales for every convolution layer in a block produces better results.

Then, we search the hyperparameters using an automatic strategy. For each target resolution, we sample 50 examples from LAION-5B to build up an evaluation set. The MSE loss of noise estimation used in the training diffusion models is used to serve as a metric for the hyperparameter search at each diffusion timestep. The hyperparameter that achieves the least loss is chosen for the corresponding timestep.

We have included this part in detail in the supplementary materials.

Q2. In the "convolution dispersion" section, a linear transformation R and structural-level calibration are used to enlarge the convolution kernel. Does this step add to the model's computational complexity? If so, by how much?

Dispersed convolution leads to slight computation overhead which is due to the quadratic complexity $O(N^2)$ of the convolution operator where $N$ is the kernel size. Therefore, we suggest using dispersed kernels as small as possible. In our case, we enlarge $3\times 3$ convolution kernels to $5 \times 5$. Theoretically, it leads to 2.7x more convolution computation in UNet. However, in practice, not all convolution layers need dispersion, and the parallel computing of GPUs will alleviate that overhead. We compare the overhead with and without $5\times 5$ dispersed convolution on $2048\times 2048$ text-to-image with SD v1.5 on one A100 40G GPU.

Q3. In the "noise-damped classifier-free guidance" section, a guidance scale w is mentioned. How is this parameter chosen? Does it require adjustments for different tasks or datasets?

The choice of the scale parameter, w, follows the same scale used in the classifier-free guidance (cfg) of the original model. Therefore, there is no need for us to adjust the value of w ourselves. Different tasks and datasets may require different values of w for the cfg. The w in our noise-damped classifier-free guidance (ndcfg) can just remain the same as in the standard cfg.

Q4. The authors used FID and KID as the metrics that require downsampling images to 229 × 229, does this imply that these metrics are not very suitable for evaluating high-resolution images?

FID and KID can only be used to measure the overall generation quality such as the global structure. Due to the downsampling issue, they are not able to reflect the texture details and definition. Thus we add four extra metrics pFID, pKID, sFID, and sKID to address this problem. Please refer to the general response for the results, which show that our method achieves significantly better sFID, and sKID scores and highlight the superiority of our method on the texture details.

Q5. Has the paper considered comparisons with other recently published methods for high-resolution image generation? Would such comparisons further validate the effectiveness of the model?

Thank you for your suggestion. We compare our method with SD XL for high-resolution image generation. Please refer to the general response. The results show that our method via SD 2.1 surpasses the SD XL on the subset of Laion 5B and has a faster inference speed and fewer model parameters.

Thank you for your time to review our paper and the encouraging comments!

W1. Fig. 5-7 and Table 2-4 could be presented in a former manner.

Thank you for your suggestion! We have made revisions to the illustration and also reorganized the order of these components accordingly.

W2. More ablation studies could be presented in the main text, and limitations are suggested.

Thanks for your suggestion! We have replenished more ablations and limitations in the supplementary due to the page limit of the main text. Please refer to Sections D and E in the supplementary materials for detailed information on the ablations and limitations.

Thank you for providing valuable feedback. We have carefully considered your questions and made corresponding revisions and experiments.

Q1. The significance of the proposed method is still unclear after the presentation of the experimental results.

Let us first clarify the significance of our work:

Our main focus is to uncover the reason behind the structural distortion observed when directly inferencing SD at higher resolutions, which is attributed to the convolutional receptive field. The key significance is the observation of the problem and the findings of its essential cause.

Based on these crucial observations, we surprisingly found altering the network receptive field, only during inference, can effectively generate higher-resolution images and videos. We believe our insights may inspire future research in designing more effective and efficient super-resolution or high-resolution image and video models.

From the application view, our approach does not require any training and big datasets, and can be directly applied to the video generation as a free solution. Furthermore, following your suggestions, we provide a comprehensive study on the general generation quality, texture details, inference time, and model parameters in Table 1 of the general response (also documented in Supplementary Table 10), showing better texture details, better efficiency, and better generation performance (when using low-resolution as guidance during the inference) of our method compared with SD+SR.

It is worth mentioning that the significance of our findings has also been acknowledged by other reviewers, who recognized that our work addresses a significant challenge and tackles a novel and important problem in image synthesis (reviewer 6a3j). It was also highlighted by reviewer 3ixD, who considered it quite important to the research community.

W1. The major issue is with the experiments. It is unsure whether the results in Table 1 tell about the differences among SD solutions or the differences between the "Method" in the second column. And when the resolution changes, how did the results vary accordingly? Are the metrics still acceptable if a higher-resolution image is generated? What do the metrics mean for the application in addition to their use for comparison?

1. Purpose of Table 1: Table 1 aims to compare the performance of our method with the baseline methods for the problem of sampling higher-resolution images than the training resolution, rather than comparing different SD solutions. We have made the necessary modifications in the caption of Table 1 to convey our intent.
2. Impact of resolution changes: The metrics deteriorate as the desired resolution increases as the setting becomes more and more challenging. However, our method consistently outperforms all other baselines. For the visual results, please refer to Section D in supplementary materials for the results with increasing resolutions.
3. Acceptability of metrics: The metrics are acceptable and can be checked via the visual results provided on our anonymous website. Note that due to the long inference time of high-resolution images, we calculate the metrics on 10k images on the settings of 6.25x, 8x, and 16x, and 30k images on the settings of 4x. Thus, we do not intend to compare the performance across different resolutions.
4. Meaning for the application: The lower scores of FID_real and KID_real mean better general generation quality and FID_base and KID_base mean the performance gap when inferencing higher-resolution images than the training resolution, indicating the scale-generalization ability of the model. Additionally, we have also incorporated two additional metrics (patch-FID/KID and sFID/KID) to evaluate the quality and sharpness of texture details in Table 1 of the general response to better evaluate our method from the application aspect.

W2. As the authors mentioned the results in Table 2 were not better than the SR+SR method, what is the justification to use the proposed solution? How could the user balance the efficiency and the quality in terms of the application's needs?

Justification for using our proposed solution compared with SD+SR:

 1. Please check Table 1 in the general response where we compare our method with SD+SR comprehensively including both efficiency and quality.

 2. Moreover, there is currently no open-domain video SR model available for high-resolution video generation. Applying the image SR model on video leads to the temporal jitter issue because the details generated for each frame are independent. In contrast, our method addresses this issue by generating higher-resolution videos while leveraging the temporal modeling ability inherited from the lower-resolution pretrained model. This allows us to generate higher-resolution videos via a free solution.

We sincerely appreciate your valuable feedback and hope that our response addresses your concerns.