One-step Image-function Generation via Consistency Training

W5: To ... DISTS.

R5: Thank you very much for bringing our attention to these image quality assessment metrics. We mainly follow the evaluation process of current generative models, which is to evaluate the difference between the distribution of training images and generated images. We also follow your suggestion to evaluate different methods in terms of the image quality assessment metrics on two datasets. Since LPIPS and DISTS are full reference metrics, the generative models do not have ground truth, so we cannot evaluate such two metrics. We use the pyiqa package to evaluate the no reference metrics mentioned by you, i.e., NIQE for distorted images, CLIPIQA for visual consistency and comprehensibility of image content, MUSIQ for image structure and distortion in images, and MANIQA for naturalness and structural information of images. Specifically, we follow the setting mentioned in line 401 in the paper to evaluate NIQE, CLIPIQA, and MUSIQ on 50000 generated images and evaluate MANIQA on 1000 generated images for efficiency. The results are provided in the following two tables. The best results are marked as bold and the second results are marked with underline.

We find that these quantitative results are highly consistent with our conclusions in the paper (lines 499-504), which is 1) our model has a better performance than the original U-Net in terms of all metrics. 2) the performance of our function generator is comparable with the Transformer-based generators.

[1] Yin, Tianwei, et al. "One-step diffusion with distribution matching distillation." CVPR. 2024.

[2] Song, Yang, et al. "Consistency Models." ICML, 2023.

[3] Song, Yang, and Prafulla Dhariwal. "Improved Techniques for Training Consistency Models." ICLR, 2024.

W1: The paper ... applications.

R1: Our method fundamentally diverges from other Transformer-based models in diffusion. The major contribution of this paper is much more than introducing Transformers in consistency models but delivering Transformers to generate an image function, which is parameterized as the MLPs, as presented in Figure 1. We have provided a detailed discussion of the difference between our model with U-Net and other Transformer-based models in lines 479-504 and Table 2. Specifically, the traditional consistency pipelines suffer from fixed-resolution image generation, therefore if they are required to generate images with different resolutions, they cannot generate consistent images across different resolutions. And they are required to train separate models at different resolutions to generate images at different resolutions. The inference process is independent across different resolutions and therefore is very costly (See the Multi-Resolution Sampling Sampling FPS in Table 2). In contrast, our method delivers a Transformer to generate image functions, which can then be rendered into any-resolution images with nearly negligible time cost. Therefore, our method is quite suitable for the scenario where images with multiple resolutions are required.

W2: The pre-training ... approaches.

R2: Thank you very much for your suggestion. We need to emphasize that the training time for pretraining is much less than the training time for consistency training, which is so resource-costly. Take the CelebA-64 dataset (with a total of 202,599 images) as an example. Pretraining for 30 epochs (with a batch size of 32) requires approximately 190,937 iterations. This is only about one-fifth of the 900,000 iterations required for consistency training. In addition, the pretraining is applied solely to the function prediction module, which accounts for less than half of the total model parameters. As a result, the time spent on pretraining is significantly lower than that of consistency training, amounting to about 5%. The exact training time for different models is presented here:

W3: The ... DMD.

R3: Thank you very much for bringing our attention to these interesting works. We admit that several one-step methods have been proposed. However, most of these works need to finetune or distill from an existing diffusion model, e.g. DMD [1] mentioned by you and the CD mode from consistency models. The distillation-based methods may suffer from the bias of the pre-trained diffusion model. In contrast, our method relies on consistency training that can train the generative model in isolation and only from data, which should be a more flexible and convenient training mechanism as we mention in lines 40-42 and "as an independent family of generative models" by [2][3]. More importantly, our model generates an image function that supports any-resolution image generation, and this is not achieved by other one-step diffusion methods.

W4: Given ... synthesis.

R4: Thank you very much for your suggestion about training on higher-resolution images. However, it is quite unaffordable to train consistency models in such a high resolution. By now, we have showcased that even though we train our model in a 64 or 128-resolution image dataset, our model can generate consistent any-resolution images, e.g., images with 512 resolution in Figure 9. Since we generate an image function that can be quired with arbitrary coordinates within a continuous range, our model can theoretically guarantee image generation with arbitrary resolution. Training with high resolution can improve the performance of our model on high-resolution image generation (a question about better or worse), but it does not affect the ability that our model can generate arbitrary resolution images (a question about yes or no).

Dear Reviewer N1ca,

We appreciate the time and effort that you have dedicated to reviewing our manuscript. We have carefully addressed all your queries. Could you kindly spare a moment (approximately 2 minutes) to review our responses? Have our responses addressed your major concerns? If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Paper 9675

Dear Reviewer N1ca,

We appreciate the time and effort that you have dedicated to reviewing our manuscript. We have carefully addressed all your queries. Could you kindly spare a moment (approximately 2 minutes) to review our responses? Have our responses addressed your major concerns? If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Paper 9675

Dear Reviewer N1ca,

We appreciate the time and effort that you have dedicated to reviewing our manuscript. We have carefully addressed all your queries and uploaded a new PDF version. Have our responses addressed your major concerns? If you have further concerns, please discuss them with us. We will address it further. We look forward to your feedback.

Best regards,

Authors of Paper 9675

W4: (4) Computation...model.

R4: Thank you very much for your suggestion. We have shown in lines 847-849 that we implemented the DiT and UViT with a close amount of learnable parameters with our function generator. So the parameter amount and GPU cost of DiT and UViT are very close to our function generators. We will highlight this setting in our future version. The major difference in the computation cost comparison between DiT, UVit, and our function generator is the multi-resolution sampling FPS, which has been shown in Table 2.

W5: (5) It ... equations.

R5: Thank you very much for pointing out our typo.

[1] Any-resolution training for high-resolution image synthesis, ECCV 2022

W1: (1) Novelty ... incremental.

R1: We need to emphasize that we deliver INR in consistency models mainly for one-step any-resolution image generation, which mainly targets the efficient generation problem when generating images with multiple resolutions or unknown arbitrary resolutions.

Our paper shows several differences and advantages with other image generation frameworks with INRs, and the greatest advantages are the flexible and efficient one-stage end-to-end training pipeline and the one-step generation process. As we have discussed in "diffusion models based on implicit neural representations" in the related work part of the paper (lines 162-169), most works that apply INR in their works are two-stage training pipelines. They consider the process of converting signals to the INR (encoding signals to the INR space) and the process of diffusion on the INR space as two processes that are totally independent of each other. The two-stage approach helps to ease the difficulty of generating INRs but the training process is inflexible and the error in the first representation stage would greatly affect the performance of the second diffusion stage, e.g. only 40.40 FID reported by Functa in CelebA-HQ 64^2 dataset. In contrast, our method enjoys the one-stage end-to-end training process and optimizes all modules from pure data.

Naively implementing a one-stage training pipeline for diffusion on INRs requires evaluating the denoiser network in the image space and will face two challenges. 1) it is unaffordable to evaluate the denoiser network in the image space for diffusion models because they need to render the generated INRs into images for each diffusion step, which makes the inference process very costly. 2) It is very hard for a denoiser network to generate INRs for images with so much noise because INRs prior to fit low-frequency signals and are hard to fit high-frequency noises. Therefore, they deliver a two-stage training pipeline to avoid these two challenges: firstly converting all signals to their INRs and directly applying diffusion on the INR space.

In contrast, our method relying on consistency training has many insights and solves these two challenges in a much clever way. 1) The consistency training enables our model to generate INRs with just a single diffusion step, therefore it is affordable for us to directly evaluate the denoiser network in the image space. 2) The target of consistency training is to train a denoiser network to map all points in the PF-ODE trajectory into the original image, therefore our denoiser network only needs to generate INRs for images with little noise, which greatly improves the training efficiency of the denoiser network.

As a result, we believe our method is novel and should be distinguished from other INR-based diffusion models.

W2: (2) Related work... beneficial.

R2: Thank you very much for the suggestion about additional discussion with high-resolution generation strategies. Most high-resolution generation methods are based on generation in a patch-by-patch manner, e.g. [1] suggested by Reviewer 8PPP. However, our method focuses on generating the entire image function in a single step, and the image function is represented by a global function and can be rendered into any-resolution images. Basically, the patch-by-patch approach for generating high-resolution images is orthogonal to our method of generating the entire image function as a whole. The former focuses more on generating local regions iteratively, while our method emphasizes the global representation. Additionally, it is entirely feasible to combine our method with the patch-by-patch strategy. Specifically, we can generate a global signal within each patch to enable fast generation with flexible resolutions at the patch level, and then use the patch-by-patch approach to assemble a high-resolution image.

W3: (3) Performance ... Table 2.

R3: We need to clarify that our model is comparable with Transformer-based generators, e.g. better IS metric than DiT. Table 2 is mainly for the ablation study. It shows that DiT should be a better denoiser for consistency models than the original U-Net, which motivates us to deliver a DiT-like encoder as our feature extraction module that handles input noisy images and modulates noise level embedding. However, as we have discussed in Lines 479-504, our method generates image function rather than directly generating fixed-resolution images. The architectures (U-Net, UViT, DiT) that generate fixed-resolution images suffer from multiple training and inference processes when generating images with multiple resolutions, therefore they are less flexible and have a much lower multi-resolution sampling FPS than our method.

Dear Reviewer NMKa,

We appreciate the time and effort that you have dedicated to reviewing our manuscript. We have carefully addressed all your queries and uploaded a new PDF version. Have our responses addressed your major concerns? If you have further concerns, please discuss them with us. We will address it further. We look forward to your feedback.

Best regards,

Authors of Paper 9675

W1: In comparison ... sizes.

R1: We have shown that the relatively high FID values are due to the low training batch size (32) that allows training with a single GPU. In fact, we have tried training with more GPUs and larger batch sizes (64 and 128). The claim that the consistency training with UNet will oscillate is consistent as the training batch size reaches 128, which harms the performance of consistency training. In contrast, our CM-func can have a more stable consistency training process which leads to better generation quality. It is unaffordable for us to train the model exactly following Song et al.’s experimental setup which has a batch size of 2048 or 4096.

W2: In Figure 8, ..., efficient.

R2: We need to emphasize that the training time for our pretraining step is much less than the training time for consistency training, which is so resource-costly. Take the CelebA-64 dataset (with a total of 202,599 images) as an example. Pretraining for 30 epochs (with a batch size of 32) requires approximately 190,937 iterations. This is only about one-fifth of the 900,000 iterations required for consistency training. In addition, the pretraining is applied solely to the Function Prediction Module, which accounts for less than half of the total model parameters. As a result, the time spent on pretraining is significantly lower than that of consistency training, amounting to about 5%. The exact training time for different models is presented here:

Q1: Total Training Time.

R3: Please check the above table.

Q2: Evaluation in Table 2.

R4: We apologize for the unclear caption for Table 2. It should be "Table for efficiency and accuracy of different generators on CelebA dataset". The Accuracy is evaluated on the CebebA dataset with resolution of 64.

And the multi-resolution sampling FPS is described in lines 483-485 and line 495. We use this metric to evaluate the efficiency of different models if they are used to generate images with different resolutions. In our setting, it is calculated as the FPS that generates 10000 image functions, each of which requires 3 different resolutions ($32 \times 32$, $64 \times 64$, $128 \times 128$, therefore a total of 30000 images). The traditional models (CM-UNet, CM-UViT, CM-DiT) cannot sample any-resolution images, so they are required to train three separate models for three different resolutions. The multi-resolution sampling FPSs for these three models are calculated as $\frac{30000}{\sum_{i=1}^{10000}{T_{32}^i}+\sum_{i=1}^{10000}{T_{64}^i}+\sum_{i=1}^{10000}{T_{128}^i}}$, where $T_k^i$ is the average time that generates $i^{th}$ image with resolution $k$. For our CM-Func, we only need to generate one image function for each image and then can be rendered as any-resolution images, so the multi-resolution sampling FPS for CM-Func can be calculated as $\frac{30000}{\sum_{i=1}^{10000}{(T^i}+T_{R32}+T_{R64}+T_{R128})}$, where $T^i$ is the average time that generates $i^{th}$ image functions, and $T_{R32}, T_{R64}, T_{R128}$ are the time to render the image function to image with 32/64/128 resolutions (which is nearly negligible compared to $T^i$). This metric reflects the efficiency of sampling any-resolution images of our method compared to traditional models.

Q3: Effectiveness with Larger Batch Sizes.

R5: Yes. In fact, we have tried training with more GPUs and larger batch sizes (64 and 128). The claim that the consistency training with UNet will oscillate is consistent as the training batch size reaches 128, which harms the performance of consistency training. Our model continues to perform better than the original CM at these batch-size settings. Please check the R1 response.

Q4: Related Works.

R6: Thank you very much for bringing our attention to this interesting work. We will add this reference to our related works. Different from our work, this work delivers explicit field characterization to model signals with different modalities (images, shapes, and spherical data). Their target is to provide a unified framework that can denoise on the field with a single training stage compared to Functa and GEM. Due to the explicit field characterization, their model cannot achieve any-resolution image sampling as their coordinate is fixed during training. In contrast, the models relying on implicit fields have the potential for any-resolution image sampling, such as Functa and our method. As discussed in related work in our paper, Functa requires two-stage training while our model can be trained within a end-to-end manner. In addition, our model can achieve one-step generation due to the consistency training while DPF and Functa still require multiple-step sampling because their models are implemented in DDPM.

Dear Reviewer ftNQ,

We appreciate the time and effort that you have dedicated to reviewing our manuscript. We have carefully addressed all your queries. Could you kindly spare a moment (approximately 2 minutes) to review our responses? Have our responses addressed your major concerns? If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Paper 9675

Dear Reviewer ftNQ,

We appreciate the time and effort that you have dedicated to reviewing our manuscript. We have carefully addressed all your queries and uploaded a new PDF version. Have our responses addressed your major concerns? If you have further concerns, please discuss them with us. We will address it further. We look forward to your feedback.

Best regards,

Authors of Paper 9675

W1: It is not very clear that why modeling as INR can help improve the stability in low-resource training. With the Transformer generator and INR representation, is the input noisy image/target in training at fixed resolution or varied resolutions? More discussions about the intuition for the improvement might be helpful. Can the reconstruction pre-training also be applied for the UNet consistency model?

R1: We attribute the stable training to the design of the Feature Extraction Module which is based on DiT that delivers an adaLN-Zero layer to modulate the Transformer encoder. The original DiT for diffusion models has already been shown to exhibit superior scalability compared to U-Net. We adapt it to consistency models and the ablation study for CM-DiT (Table 2) shows that DiT has a consistent performance on consistency models (better than CM-UNet). The input noisy image and the target are of a fixed resolution, and the model outputs an image function, which can then be sampled at arbitrary resolutions. In Appendix C.3.2 and Table 4, we have discussed scenarios where the input noise resolution varies for the CelebA dataset. We find that increasing the resolution of the input noise slightly improves the FID of the generated images due to larger noise space, though this comes at the cost of longer runtime. The reconstruction pre-training cannot be applied to the UNet consistency model as the reconstruction pre-training is to train a model that can predict its INR for an input image, while the UNet cannot perform such a function.

W2: Despite showing many metrics, the FID values for both the baseline and proposed method are very high (though it is due to the training budget). The results will be more convincing and solid when the methods can achieve a generally better quality.

R2: Thanks for your suggestion. We have shown that the relatively high FID values are due to the low training batch size (32) that allows training with a single GPU. In fact, we have tried training with more GPUs and larger batch sizes (64 and 128). The claim that the consistency training with UNet will oscillate is consistent as the training batch size reaches 128, which harms the performance of consistency training. In contrast, our CM-func can have a more stable consistency training process which leads to better generation quality.

W3: The claim the advantage of any-resolution generation, it is better to discuss and compare to more recent works that specifically works on any-resolution image generation, for example [1, 2].

R3: Thank you very much for bringing us to these two interesting works and we will include them in our related work. [1] tries to generate any-resolution images with the manner of patch-by-patch with the supervision of GAN, which is totally different from our setting. The output of their generator is still of a fixed resolution, specifically $𝑝 \times 𝑝$, and must be square-shaped. In other words, to generate larger images, their model requires multiple runs to generate patches, which are then concatenated together. In contrast, our model only needs a single run to produce a continuous function of the corresponding image, which can then be sampled at any resolution. Furthermore, their model cannot generate images at lower resolutions, as the patches they generate are at least $𝑝 \times 𝑝$. Our model, however, allows for sampling at any desired resolution, offering greater sampling flexibility.

[2] is an interesting work that should be included in the "Diffusion Models Based on Implicit Neural Representations" section of our related work. Same as other diffusion models based on INRs, they first train an INR converter as an encoder module for represetations, and then separately train a diffusion model on these INR representations. It is a two-stage training and inference pipeline, causing inflexible training and potential error accumulation for each stage. In contrast, our paper proposes a novel unified architecture that is trained in an end-to-end manner and can generate image functions from noise in a single stage.

Q1: It is shown in the supplementary that the generated INR has better quality than bilinear interpolation when decoding to high resolutions. Is the high resolution higher than the resolution in training? If it is the case of resolution extrapolation, is any artifact observed in high resolutoins?

R4: Yes, the high resolution sampled from our generator can be higher than the resolution in training. No artifact is observed. Here, we follow [3] to apply variational coordinates to eliminate artifacts.

[1] Any-resolution training for high-resolution image synthesis, ECCV 2022

[2] Image Neural Field Diffusion Models, CVPR 2024

[3] Attention Beats Linear for Fast Implicit Neural Representation Generation, ECCV 2024

Dear Reviewer 8PPP,

We appreciate the time and effort that you have dedicated to reviewing our manuscript. We have carefully addressed all your queries. Could you kindly spare a moment (approximately 2 minutes) to review our responses? Have our responses addressed your major concerns? If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Paper 9675

Dear Reviewer 8PPP,

We appreciate the time and effort that you have dedicated to reviewing our manuscript. We have carefully addressed all your queries. Could you kindly spare a moment (approximately 2 minutes) to review our responses? Have our responses addressed your major concerns? If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Paper 9675

Dear Reviewer 8PPP,

We appreciate the time and effort that you have dedicated to reviewing our manuscript. We have carefully addressed all your queries and uploaded a new PDF version. Have our responses addressed your major concerns? If you have further concerns, please discuss them with us. We will address it further. We look forward to your feedback.

Best regards,

Authors of Paper 9675