ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization

the authors have not solved a key issue of the approach of noise optimization but just ignored it entirely in favor of a more limited problem setting where the results they achieved is rather expected.

While we agree that solving the issue of computational efficiency for multi-step generation with noise optimization is a very interesting research question, we never claim to solve this problem generally with ReNO. We sidestep this challenge by considering one-step diffusion models. Note that this is still a major insight because applying noise optimization in the form of DOODL/D-Flow is not practical for general T2I generation, as mentioned in the global rebuttal.

Furthermore, the reliance on an ensemble [...] ungeneralizable [...]

We report the performance of all the combinations of reward models in Table 1 & 7. As can be seen, also without an ensemble with only HPSv2 (better image quality) or ImageReward (better prompt following), ReNO achieves significant performance improvements. We are unsure if we understand why this would make ReNO not generalizable. In the general setting we are considering, the goal is no one specific preference but a general improvement of the used model.

If there were a novel preference that it were important to optimize a text-to-image model to respect, this approach would not be able to be applied successfully.

On the contrary, our approach is designed to be flexible and adaptable to various objectives. There's no inherent limitation preventing the application of ReNO to novel preferences or optimization goals. In fact, we demonstrate in Figure 3 how "personalized" objectives can be effectively incorporated. With just 10 optimization iterations, we show significant increases in specific color attributes (redness/blueness) of generated images.

Furthermore, the user study was conducted for SD-Turbo + ReNO against default models like SD-Turbo without any optimization; of course, there should be an improvement!

In the user study, we specifically compare SD-Turbo + ReNO against SDXL-Turbo, SD2.1 (50-step), and SDXL (50-step). All of these models are usually preferred over SD-Turbo. SD-Turbo + ReNO is significantly preferred over all of them, even the next-generation SDXL with 50 steps! Additionally, HyperSDXL + ReNO is preferred over SD3 (8B).

Are there ways to optimize multiple noises through ReNO to generate a batch of preference-respecting images without essentially optimizing each noise separately?

Theoretically, this could also be incorporated into ReNO as a further Criterion function. However, the problem we are tackling is general T2I generation, which is by default only per single prompt and noise. Thus, this is out of the scope of our work but an interesting future direction.

What are the ways ReNO can enable or address the backpropagation through multiple denoising levels, and provide interesting implications for general multi-step diffusion models? Are there ways this can be demonstrated in the rebuttal?

While we agree that solving the challenges of noise optimization for multi-step generation is a very interesting research question, we never claim to solve this problem generally with ReNO, and it is also out of the scope of this work. We show how the noise optimization network can be leveraged to effectively arrive at a very significantly better model. We leave how to best adapt our findings to multi-step diffusion models to future work.

Equation 7 was referenced before it was even written

Thanks for this pointer, it was supposed to refer to Equation 4 and we will update it in the paper accordingly.

one-step diffusion models were only mentioned in two sentences (lines 116-117).

The paragraph [lines 112-125] completely serves to introduce the different one-step diffusion models we employ in this work and how they work in general.

reads almost as a background/related works section (most clearly demonstrated in lines 118-125)

In this part, we introduce the four different one-step models we use to benchmark ReNO. We consider this as background for ReNO, as the section is also titled.

Then, there is a separate Related Works section. The paper could benefit from severe reorganization to improve its clarity.

Thank you for this comment; we will consider how to improve the clarity. With the separate related work section after the introduction of ReNO we aim to contextualize ReNO within the scope of all related work. Which part of the way we introduce ReNO exactly was unclear? We would be happy to incorporate any specific suggestions to enhance the clarity.

We would like to clarify the contributions of ReNO to ensure the problems ReNO is aiming to solve are understood. We tackle the question of whether we can generally enhance T2I models without any fine-tuning at test time. We propose to tackle this through the Noise Optimization framework by considering human-preference reward models as the optimization criterion. This is not a task-specific framework, nor are we proposing a new general method for noise optimization. Instead, we are specifically aiming to generally enhance a trained model at test time through our approach. This has not yet been considered or discussed in previous work, and how this generally performs was unclear. Additionally, it poses challenges that we propose to tackle through the use of one-step models (for computational efficiency), the use of multiple reward models (to prevent reward hacking), and by using noise regularization (for the noise to stay in distribution).

We thoroughly evaluate ReNO across four one-step T2I models (i.e SD-Turbo, SDXL-Turbo, Pixart-$\alpha$-DMD, and HyperSDXL), across 3 benchmarks (T2I-Compbench, GenEval, Parti-Prompts), human preference evaluations (nearly 10k comparisons), and now also the diversity of the outputs. Our results show generally enhanced performance without optimizing for any specific task. For example, SD-Turbo is elevated to prompt-following levels close to DALLE-3 and is significantly preferred over 50-step SDXL, while HyperSDXL + ReNO is preferred over SD3 (8B). This substantial performance increase is a major contribution of ReNO, and we argue it is a very significant finding. It elucidates the importance of initially sampled noise and demonstrates its effective manipulation within a reasonable timeframe. We believe it is fascinating and unexpected that such a lightweight approach can significantly enhance the performance of image generation. This also differentiates ReNO from previous noise optimization work by showcasing the framework's power in a new, more general setting. Lastly, it motivates further research into understanding and controlling initial noise and provides a novel way of benchmarking future T2I reward models.

Comparison to SeedSelect (Samuel et al.)

SeedSelect tackles rare-concept generation, a fundamentally different problem from the one ReNO is solving. I.e. the goal of SeedSelect is, given 3-5 reference images, to generate images of rare concepts represented in these images. They propose to tackle this based on the noise optimization framework and reduce computational time, as mentioned in our related work section: "To mitigate this, Samuel et al. [64] propose a bootstrap-based method to increase the efficiency of generating a batch of images. However, this method is limited to settings where the goal is to generate samples including a concept jointly represented by a set of input images". On the other hand, we consider a more general setting, where each image is generated separately, given noise and text as input. Therefore, adapting SeedSelect to the general setting considered in our paper is not straightforward.

The authors demonstrate that the performance of this optimized one-step text-to-image model is comparable to diffusion models that utilize multiple levels of denoising on preference benchmarks.

We disagree with this statement. We clearly show across multiple benchmarks and user studies that ReNO-optimized one-step models significantly outperform their corresponding multi-step models in all benchmarks and even their next-generation multi-step ones (See Tables 2, 3 & 5 and Figures 4 & 5).

Optimizing the initial source noise has been explored before, and the authors are reusing existing preference classifiers/constraint.

As mentioned above, while we are repurposing existing reward models, as far as we are aware, the optimization criterion of human-preference reward models for noise optimization has not been considered or discussed before.

The results seem expected, and do not provide deep insight (they essentially verify that optimization of the the initial noise helps performance)

We would argue that the magnitude of the performance increase makes it a very significant insight that is not expected at all. One-step models are known to have significantly worse visual quality and prompt following than the base multi-step models, therefore it is not expected that purely optimizing the initial noise would make them outperform even next generation multi-step models. Further, increasing the diversity of the generated images compared to one-step models is also unexpected and a valuable finding as less diversity is also a common weakness of one-step models.

First and foremost, the comparisons showcased within this paper are flawed. In Table 2 and Table 3, the authors compare their approach against the performance of default text-to-image models. It should be almost expected that the performance of ReNO, by virtue of task-specific optimization, should have improved performance over default text-to-image models.

As mentioned, we are not doing any task-specific optimization. We are proposing one optimization that generally improves a T2I model, and we thoroughly evaluate each ReNO-enhanced model over different benchmarks as well as human evaluation.

This is therefore not an interesting comparison; in fact, it even raises suspicions on the approach in the cases that it does not outperform default models (e.g. Attribute Binding in Dall-E 3).

We respectfully disagree. Pushing SD-Turbo to a performance close to DALL-E 3 just by changing the initial noise is not expected and is actually a very interesting comparison as also acknowledged by Reviewer UuFQ.

We thank the reviewer for engaging with our rebuttal. We would like to first clarify our motivation for approaching this work: Our goal was to obtain the best possible/extremely high-quality text-to-image generation using open-source models and tools. To this end, models such as the 50-step SD2.1/SDXL models would be the first choice. However, the shortcomings of these models are well-documented and are further corroborated in our paper. Alternately, one could train bigger models on larger datasets with higher-quality data (as is the trend with SD2.1 -> SDXL -> SD3). Unfortunately, these require large-scale GPU resources unavailable to most research groups (e.g. DALL-E 2 itself uses 40000+ A100 GPU days) and additionally, current SOTA models are paid services and not open source (DALLE-3, SD3 (8B)). Therefore, test-time optimization methods are a compelling alternative to enhance the generation quality of existing multi-step T2I models. These methods (e.g. DOODL) work with multi-step models and enhance the quality of the generated images. However, not only are they computationally expensive, they also provide limited improvements in the metrics that they optimize for(on average CLIPScore increases by 0.03 with DOODL on 50 step SD2.1). Our hypothesis was that the lack of effective optimization was due to exploding gradients and other challenges of optimizing multi-step diffusion models. Therefore, we made the unconventional decision of optimizing the initial noise of a one-step model. Apriori, it was unclear if these models could even match the corresponding multi-step model even after noise optimization, let alone surpass them. To our surprise, not only was it 60x faster to optimize, but the gains in CLIPScore were 4x (0.12) that were obtained from optimizing multi-step models (e.g., DOODL). Enhancing this further, we incorporated other models that could provide complementary signals to improve both visual quality (e.g., HPSv2) and prompt following (e.g., ImageReward). As a result, we obtained image generation results that were the highest reported results among any open-source method on T2I-Compbench and GenEval. Further, even for the same time that a 50-step SDXL takes for generation, we show better prompt following by performing noise optimization with SD-Turbo (Fig 5). We believe that providing a recipe to enhance the quality of text-to-image generation (noise optimization of the best-distilled models) in a cost-effective manner is our key contribution.

In Tables 2 it is pretty apparent that DALL-E 3 outperforms ReNO, while being a multi-step denoising diffusion model. There is some confusion on why the authors disagree with the initial statement, and suggests that “ReNO-optimized one-step models significantly outperform their corresponding multi-step models in all benchmarks”. [...] In fact, it is troubling that even with optimization it does not outperform DALL-E 3;

Our finding is for one-step models and their corresponding multi-step models. SD-Turbo is based on SD2.1 with an 800M U-Net and a 336M params CLIP ViT-H text encoder, which makes it the multi-step model to compare to. SD-Turbo and DALLE-3 are models trained with very different resources and architectures. While specific details for DALLE-3 are not available, e.g. SD3 leverages a DiT of 8B params with a T5-XXL (4B params) text encoder. We show that in all experiments conducted, SD-Turbo outperforms 50-step SD2.1 and even 50-step SDXL, which is the next-generation multi-step model. Based on our findings, a one-step model based on SD3/DALLE-3, like SD3-Turbo, enhanced with ReNO should outperform multi-step SD3/DALLE-3. Unfortunately, these models are proprietary, and thus, we could not benchmark with SD3-Turbo. To summarize, DALLE-3 and SD3 are "two generations" after SD-Turbo and thus, do not constitute a fair comparison between one-step and multi-step models. Similarly, a method improving LLaMa2-7B isn't expected to outperform GPT-4 to be a meaningful research contribution.

Regarding multiple reward models: it still appears that ReNO depends on an ensemble of reward models. And that performance does not increase significantly without the ensemble. Relying on an ensemble suggests that this method may not perform well for a specific, novel criteria (in situations where other existing criteria in the ensemble are not as important)

As mentioned in our previous answer, ReNO can be flexibly used based on given preferences. While we benchmark all current T2I reward models we are aware of, adapting this to new models is straightforward as long as the novel criteria are expressed in a differentiable function. See, for example, the color example in Figure 3, or given a new, more robust, and stronger T2I reward model, ReNO can be employed with just this reward. Table 1 does show that already a single reward model achieves significant improvements.

We thank the reviewer for their detailed comments and are especially glad they emphasize the significant enhancement achieved by ReNO. Below, we address the concerns raised in the review.

Therefore, optimizing time is a crucial factor that requires attention. The paper points out that the optimization time takes 20-50 seconds, and it says that one-step t2i is used. What if multi-step reasoning is directly used? Is the effect better than optimizing noise?

We would like to point out that ReNO actually provides an efficient formulation to enhance T2I models even compared to existing multi-step T2I models. For instance, in Figure 5, we show that ReNO outperforms existing open-source models such as multi-step SDXL and PixArt-$\alpha$ with the same compute budget. Even with 10-15 iterations (4-10 seconds depending on the one-step model), we see significant improvements in prompt following and visual quality, as shown in Figure 5. Additionally, ReNO is much faster than other noise optimization methods, as mentioned in the global rebuttal.

Noise optimization is a bit like an "adversarial attack" that achieves its goal by adding noise to the original input

While we agree that there are connections to the literature on adversarial attacks, we would like to emphasize that the changes in the generated images are specifically not adversarial. We agree that leveraging different optimization techniques, e.g. from the adversarial attack literature, to reduce the number of iterations/time required for convergence could be an interesting future direction.

We thank the reviewer for their detailed comments. We are especially glad that they appreciated the simplicity and effectiveness of ReNO. Below, we address the concerns raised in the review.

ReNO is essentially a runtime optimization approach that leverages advanced reward models to achieve state-of-the-art text-to-image generation capabilities. Indeed, the authors opt to optimize initial noise as a learnable parameter. Could I choose to make the parameters of a UNet learnable to achieve a similar objective? Have the authors attempted to apply ReNO to text-to-image diffusion models, such as SD, using techniques like gradient checkpoint and LoRA?

Yes, we also briefly tried optimizing the parameters of the U-Net with LoRA. This leads to the model generating images with visual artifacts which is caused due to "reward-hacking". In contrast, only optimizing the noise keeps the model untouched and, thus, does not lead to it generating adversarial images as long as the noise stays in distribution. Moreover, fine-tuning with reward models poses a number of other challenges. [lines 40-51] For example, the SDXL U-Net has 2.6B parameters, which makes it computationally infeasible to fine-tune at inference for every prompt. Even with LoRA and gradient checkpointing (as done by AlignProp[57]), this would still have 10M+ parameters to train (vs ~16k parameters for the noise optimization), which would take several minutes, if not hours (AlignProp fine-tunes SD1.5 on 4 GPUs for 24 hours).

Has the proposed ReNO been tested on video generation diffusion models for text-to-video tasks?

Our noise optimization framework is directly applicable to video generation. However, at the moment, while there are human preference reward models for video generation, there are no one-step video generation models publicly available, which are otherwise prohibitively expensive to train in a resource-constrained setting. Once open-source one-step video models are available, ReNO would be ideally suited to further enhance the performance of video generation.

The authors are encouraged to include the following references

We thank the reviewer for pointing out these related works, especially the related concurrent InitNO, and we will add these to the paper. While InitNO also looks to optimize the initial noise, this is done by computing a loss function using attention maps as opposed to optimization with human preference reward objectives.

We thank the reviewer for their detailed comments. We are especially glad that they enjoyed the paper. Below, we address the concerns raised in the review.

Limited comparison to related optimization approaches

We address the comparison to DOODL in the global response. For D-Flow, the proposed method takes even longer (30-40 minutes for a 350M parameter model).

Insufficient analysis of impact on image diversity / Diversity of outputs

We address this in the global response. Would you like us to analyze any other metrics to measure the distribution of generated images with ReNO compared to without ReNO?

Limited analysis of potential negative impacts or failure modes

ReNO can successfully enhance the general capabilities of existing T2I models, including better prompt following and visual quality. As with all T2I models, this can be used for positive and negative applications. Additionally, since the objective is to optimize for the human preference reward models, the model is biased towards aspects that these models especially focus on. Reward models could also be prone to biases because of, e.g., their training data and, thus, this could be a potential risk in amplifying biases in existing models and a source of undesirable outputs. However, ReNO also opens up the possibility of including a bias-mitigating reward model.

Reward model robustness

We would like to highlight Tables 1 & 7, where we benchmark the choice of different reward models. We did observe undesirable outputs without the noise regularization, as sometimes images with severe artifacts are generated, because of the noise going out of distribution, that still achieve a high reward score. Additionally, the reward models focus on some aspects more (e.g., colors, counting) than others (e.g., spatial understanding in the prompt). Specifically, we found that for a prompt including "under/above" or "left/right" the reward model scores can be very similar independent of which one is chosen for the same image.

Computational Efficiency

We would like to highlight Figure 5, where we show the improvement in attribute binding on T2I-Compbench with increasing iterations. We see rapid improvements for the first 10-15 iterations (4-6 seconds for SD-Turbo on one A100), where it already surpasses popular open-source multi-step models like multi-step SDXL and PixArt-$\alpha$. We support this with some qualitative examples in Figure 6. After 50 iterations (20 seconds), we see diminishing returns and see no major gains beyond 75 iterations (30 seconds).

Integration with other Techniques:

Our experiments (Tables 2 & 3) indicate that we get better results with stronger base models. Therefore, fine-tuning base models to improve their performance or orthogonal techniques like prompt engineering should enhance the results further and be straightforward to incorporate together with ReNO.