Thank you for the constructive criticism. We welcome any comments that can help improve the quality of our work.

W1. While the authors several times mention that "the imaginary part in the frequency space contains most of the low-frequency information of the image" (line 187) and "imaginary part provides more structural information than the real part" (line 249), they do not provide any references for this statement or supporting experiments. To me, this fact does not look evident enough. I am sure this needs more thorough explanation since otherwise it looks just like an empirical trick.

Please refer to the global response.

W2. From the text, it is unclear which Slider value was used to obtain the numbers from Table 2. Was it the value of 30, as defined in line 244?

Yes, the position of the Slider was 30. This is also indicated by the metrics in Table 1 and Table 2 which are identical. Nevertheless, we understand that it should be specifically mentioned in the Comparison section, as such Line 264 is expanded as follows "... refer to Appendix B. The Slider position is set to 30.". We note here that the position is not cherry picked but chosen randomly. Different Slider positions will yield different results (as shown in Table 1), and there may be other positions that could potentially perform even better.

W1. Also, what exactly is "using next inference step (NIS)" (see line 253)? Is this about checkerboard mask mixing?

Yes, this is correct. The "next inference step" mentioned in the Ablation study refers to the use of the chess-like mask. As noted in line 253, "... in Appendix G, incorporating the next inference ...". Appendix G provides further explanation of this process. We recognize that this may have been confusing, so the issue has been addressed as follows:

• The line 241 is changed to "... information from the next inference step using masking and the impact ..."
• The paragraph starting with line 252 is changed to " Masking (column No Mask): Analysis in column No Mask shows that not using the chess-like mask yields better scores on two metrics. However, as shown in Appendix G, incorporating the mask is crucial for removing artifacts, so we continue to use it despite the metrics." Please refer to our reply to reveiwer wkBK for the fully revised Ablation study subsection.

W3. The presentation of Fig. 3 is not easy for understanding, I advise considering its redesign. For example, forward diffusion and taking imaginary part of FFT are denoted in the same way.

We have revised Figure 3 to enhance clarity. Please refer to the attached PDF for the updated version.

W4. It is unclear from the text what the guidelines are for selecting the size for the zone of averaging for overlapping patches (line 805).

This is a set number (not a hyperparameter) that we set to 10 and never change. From our experiments, the exact number is not critical, which is why line 805 refers to a "few pixels." We have set this number to 10, and it remains fixed, as variations like 15 or 20 pixels do not produce noticeable differences. This can also be seen in the provided code (smoothed_time_mask = create_gradient_border(time_mask, gradient_width=10)). We understand that specifying the exact value is preferable, so line 804 has been updated to read: "... a tolerance of 10 pixels in the overlap ...".

Thank you for the constructive criticism. We welcome any comments that can help improve the quality of our work.

W1. Is there any literatures to support that the imaginary part of the fourier transform corresponding to the low-frequency of the signal, or is there any deeper analysis beyond the ablation results in Tab.1 to illustrate this?

Please refer to the global response.

W1. By the way, it is suggested to provide the visual ablations of this part.

Including visual ablations is indeed a good suggestion. They will be part of the final version.

Q1. How to determine the patch orders in high-resolution generation, is it random orders or follow some principles.

The patch orders are random, as described in Lines 143 and 183. However, this randomness is controlled. We track which areas have been denoised (since each pixel is denoised only once per timestep) and avoid selecting those areas again. Therefore, while the selection of patches is random, it is constrained to areas that still require denoising. For example, if 90% of the latent space has already been denoised for timestep t, the coordinates for the next patches will be chosen from the remaining 10%.

If we selected coordinates purely at random without considering the denoised areas, the inference time would increase dramatically, rendering the process infeasible, especially for very high resolutions such as $32768^2$.

This explanation will be incorporated into the "Patch Sampling" subsection for added clarity.

Q2. From Tab. 2, it is not clear why Pixelsmith slower than Scalecrafter at 2048 resolution, while nearly 2x faster at 4096 resolution. Is there any trend that the bigger, the more efficient?

We extract patches of the same size, specifically $128²$, regardless of the resolution. Pixelsmith requires 130 seconds for $2048^2$ (4194304 pixels) and 549 seconds for $4096^2$ (16777216 pixels). This represents a 4x increase in pixels and a 4.22x increase in inference time. Pixelsmith’s inference time scales linearly with the number of pixels because the denoising UNet consistently uses patches of the same size. In contrast, Scalecrafter scales differently as it does not use patches. Unfortunatelly, Scalecrafter's official code provides scripts for specific resolutions and no further experiments can be conducted to examine how they scale at different resolutions and most importantly there is also the constraints of the required memory. Pixelsmith apart from not being constrained by memory is very flexible. It is posible to generate an image with resolution 4096x1024 using a $1024^2$ base image. The final ratio is not constrained by the base one. To accomplish this we only change the final resolution (ie just one number), no script or otherwise optimization is needed.

Q3. Is the content of the final scaled high resolution output significantly differ from the base output, whether the new objects will be generated?

Several factors influence the comparison between the final image and the base image. Key factors include the Slider position, the number of intermediate steps, the resolution of the final image, and the specific text prompt used. The Slider is designed to offer more control over the generation process. For instance, Figure 11 demonstrates how different Slider positions produce varying results, with position 49 being closest to the base image and position 1 showing excessive repetitions. However, in certain contexts, such as generating the surface of a planet with craters, having more craters can be beneficial. Thus, the final content can vary significantly or minimally, and this can be easily adjusted using the Slider.

Thank you for the constructive criticism. We welcome any comments that can help improve the quality of our work.

W1. (full question)

We have restructured the content based on the suggestions. The Related Work and Foundations sections have been shortened. The Method section has been revised, as detailed in the global response. Additionally, the Experiments section has been expanded to enhance comprehension as follows:

Ablation study

We conduct a qualitative examination of the framework on $2048^2$ image resolution. Specifically, we assess the effects of the Slider position, the importance of the imaginary part, the significance of incorporating information from the next inference step using masking and the impact of averaging overlapping patches.

Slider Position (columns SP0, SP24, SP49): Our findings indicate that the Slider position significantly influences the results. The proposed model with a Slider position of 30 (Table 1, column Proposed) outperforms positions 0, 24, and 49 (Table 1, columns SP0, SP24, and SP49). A position of 0 introduces numerous artifacts, while a position of 49 lacks fine detail. Position 24 is close to the proposed but not optimal for the random subset. Position 30 was chosen randomly and is not cherry-picked. Appendix F demonstrates the effect of the position with a qualitative example.

Imaginary Part (columns Re, Re&Im): The proposed model (Table 1, column Proposed) averages the imaginary parts of the guidance latents and the current latents, then uses the real part of the current latents as described in the Method section (see Figure 3). This setup is chosen based on experimental results demonstrated here. We compare this with averaging the real parts of the two latent spaces and using the imaginary part of the current latents to invert back to the pixel space (Table 1, column Re), as well as averaging both the imaginary and real parts from the two latent spaces (Table 1, column Re&Im).

Masking (column No Mask): Analysis in column No Mask shows that not using the chess-like mask yields better scores on two metrics. However, as shown in Appendix G, incorporating the mask is crucial for removing artifacts, so we continue to use it despite the metrics.

Averaging (column No Aver.): Finally, we show that using patch averaging (Table 1, column Proposed) improves scores compared to not using it (Table 1, column No Aver.). This is because patch averaging eliminates patch artifacts, as seen in Appendix C.

Table1: A quantitative examination of our framework through ablations.

Metric	SP0	SP24	SP49	Re	Re&Im	No Mask	No Aver.	Proposed
(unchanged)

W2. (full question)

We have revised the Method description to better align with Figure 3 (please see the global response). Additionally, we have updated Figure 3 (please see the attached PDF) to enhance readability and coherence. Regarding the second part of your concern, please refer to Appendix G for a qualitative comparison and the Experiments section for a quantitative comparison. Every time a denoising step finishes, there is a chance for artifacts to appear. The chess-like mask is applied after the denoising, replacing some information with the guidance where no artifacts exist.

W3. Errors and Mismatches in the Experiment Section: According to Table 1, using the real part is better, which contradicts Line #248.

In Lines #247-248 we state that "Using the imaginary part to average the guidance latents and the current latents is more effective than using the real part or...". This statement is consistent with the results shown in Table 1, where all four metrics demonstrate better performance when using the imaginary part (last column) compared to using the real part (column Re).

W3. Line #252 describes the wrong column for "the next inference step".

You are correct, this was an error. Thank you for bringing this to our attention. We have now corrected it.

W3. Also, the term “base model” in Table 2 is unclear—does it refer to the same model as Table 2’s Pixelsmith and Figure 1’s base model?

Throughout the paper we refer to SDXL as the base model (one of the references is in Line #258: "SDXL, which serves as the base model"). In the Experiments section, we also referred to our final proposed model as the base model, which caused confusion. This inconsistency has now been corrected by replacing the base model in the Experiments with proposed.

W4. Flaws in Figure Illustrations: In Figure 1, the base model image for “x256 (16384x16384)” is not displayed.

The higher resolution images are in scale with those generated by the base model. In the “x256 (16384x16384)” case, the base model image is not missing but hard to see due to the significant difference in resolution. As with the other images in Figure 1, the base model image is located at the bottom-right, overlaid on the higher resolution image. We understand this may confuse readers, so we have added a white box around all base model images to make them more distinguishable. The legend of Figure 1 has been expanded as follows: "... Pixelsmith and the base model. The higher resolution images are in scale with the images generated by the base model. The base model generations are enclosed in a white frame. Some cut-outs ..."

W4. Is this base model the same as the one mentioned in Table 1?

Please refer to our response to W3.

W4. In Figure 3, the illustration should be more concrete and easier to understand; however, some variable names are too small and there are too many blank areas. The fusion part is also unclear, with no mention of the real part of the Fourier transform and no explicit explanation of.

We have revised Figure 3 (please see the attached PDF). For a detailed explanation, please refer to the global response.

Thank you for the constructive criticism. We welcome any comments that can help improve the quality of our work.

W1. Poor presentation. The writing of this paper needs substantial improvement to be published in a venue like NeurIPS. First, the language of the paper needs to be improved. Second, the paper spends a lot of space on unnecessary content. The method in this paper is not difficult, but it is very difficult to understand the method in one reading. In terms of writing, the authors did not succeed in emphasizing the core concept of their proposed method. In addition, the length of the paper is too long. Such a simple method, but the experiments are not introduced until page 8, which should not happen.

We carefully review and change the language to make it easier for readers to follow. The method section has been refined to highlight the core concept effectively. Additionally, we have edited the method figure to enhance the clarity. See 'global rebuttal' and the pdf with the Figure. Could you kindly provide specific examples where the language was not up to the NeurIPS level as it will help us better revise our paper? The "Related work" and "Foundations" are shortened so that the "Experiments" can be introduced at page 7. Given the nature of the paper images take a lot of space (for example Figure 1 takes a full page) and as a result sections are pushed back.

W2. The core idea of this paper seems to be to generate once and then upsample the generated image by patch-wise processing. (By the way, Figure 3 is difficult to understand and Eq. 4 is not explained). This method is essentially not much different from other operations that use diffusion for upsampling. So what is the advantage of this method? Why is this method feasible?

We have revised Figure 3 to make it easier for readers to understand. As mentioned in the global response, the revised "Method" section and updated Figure 3 are now better aligned, ensuring improved coherence. Our method is fundamentally different from diffusion models used for upsampling. While diffusion models trained specifically for super-resolution generate high-resolution images from low-resolution inputs, our approach generates high-resolution images directly from text prompts. Diffusion models for upsampling primarily focus on the input image and largely ignore the text. In contrast, our method relies heavily on the text prompt, which drives the changes. As the patches denoise different areas of the latent space, the content of the text prompt becomes apparent. Without our proposed method, numerous duplications would appear across the final image. Our framework restricts these duplications, and depending on the position of the Slider, some or a lot of new information will appear in the higher resolution that is not present in the lower resolution. For example, in Figure 1 (top right), the low-resolution cut-out box for the necromancer woman resembles a metallic connection without a distinct shape. Our method's patch denoising, considering the text prompt, reveals a skull, which makes sense for a necromancer. The patches attempt to denoise based on the text prompt but are constrained by the proposed key components to avoid, for example, denoising multiple necromancers all over the image. Current methods attempting similar results often end up with artifacts, which is not the case with Pixelsmith, as the Slider’s position can eliminate them. In conclusion, diffusion models for upsampling lack this generative freedom. The downside of current works is the introduction of duplications and strange artifacts, but Pixelsmith provides control (via the Slider’s position) to avoid them.

This explanation will be reflected in the final version.

W3. The role of FFT Transformation is not well demonstrated. No exploratory experiments are used to demonstrate the motivation and effect of this method.

The FFT transformation is used to fuse information between the guidance latent space and the current (higher resolution) latent space. The decision to use the imaginary part is based on empirical results. This is now reflected in our paper as mentioned in the global response. The effect is demonstrated in Table 1.

In Table 1:

• The column labeled "Re" represents the case where we averaged the real parts of both latents and used the imaginary part of the current latent.
• The column labeled "Re&Im" indicates that we averaged both the real and imaginary parts of the two latents.
• The final column shows the proposed method, where we averaged the imaginary parts of the two latents and used the real part of the current latent.

As shown, the proposed method in the last column performs better.

L. The authors claim they discussed the limitation but I didn't find it in the text. Correct me if I am wrong.

Please refer to "6 Discussion and Considerations." One of the main limitations in generating higher-resolution images is the availability of effective metrics. More research is needed to develop methods that can accurately evaluate images without reducing their resolution to much smaller sizes, which results in the loss of detail. Additionally, there is no penalty for artifacts and duplications commonly found in many recent papers (see Figure 4 and Figure 6).

In the same section, we discuss Pixelsmith and the trade-off between preserving fine detail and suppressing artifacts. As noted, achieving true higher-resolution detail becomes increasingly challenging as the resolution increases, making it difficult to eliminate artifacts completely.

I have read the author's response, as well as the comments and discussions with other reviewers. The author has partially addressed my concerns. However, I still think that the presentation of the paper is lacking at this stage. I will improve my score. However, since I cannot see the revised paper, I cannot judge whether the final presentation meets the requirements of NeurIPS.