Devil is in the Details: Density Guidance for Detail-Aware Generation with Flow Models

We would like to thank the Reviewer for their time and efforts to scrutinize our submission. We address the raised concern below.

File with new figures: https://anonymous.4open.science/r/DensityGuidance-20E6/Density_guided_sampling___Rebuttal.pdf

The practical utility of the proposed method is a bit unclear due to lack of evaluation. It would be great to understand whether the theoretical claims could be properly validated? E.g. why not follow methodology from [Karczewski'25] for quantiative/qualitative analysis?

Thank you for this suggestion, we have now included an extensive evaluation of the proposed methods building on the methodology of [Karczewski’25]. Specifically:

Explicit Quantile Matching (EQM)

We estimated the quantile function for the CIFAR model as described in line (311 left). We tested $K = [16, 32, 64, 128, 256, 512, 1024]$ and found that using $K=128$ is enough to ensure a correlation between the desired value of log-density and the obtained one is above 99%. Based on the estimated quantile function $\phi_t$, we estimate $b_t = \frac{d}{dt}\phi_t$ with a moving average of the finite difference estimates.

Furthermore, we found that the difference between the desired values of log-density and the obtained ones goes to zero as we decrease the discretization error (increase the number of sampling steps). Interestingly, for lower number of sampling steps, even though we do not obtain exact desired values of likelihood, the correlation between the desired values and the obtained ones remains above 99%, even for as few as 32 Euler sampling steps. This means that for all values of the number of sampling steps, we saw a monotonic relationship between the target $\log p_0$ and the amount of detail (PNG size). Please see Figure 17.

Finally, we also show that we can obtain exact values of likelihoods even when sampling stochastically by using results from Appendix F, and the Euler–Maruyama algorithm. We tested different amounts of added noise: $\varphi(t)= r g(t)$ for $r=[0.1, 0.5, 0.9]$. As expected, as the amount of noise increases, the required number of steps to take to achieve exact likelihoods also increases. Please see Figure 18.

Prior Guidance vs Density Guidance vs Stochastic Density Guidance

We quantitatively compared Density Guidance (DG), and Prior Guidance (PG) using the EDM2 model. We measures the correlation between the hyperparameter ($q$ for Density guidance and $||x_T||$ for prior guidance) and the obtained $\log p_0$. We found 66% for DG and 68% for PG.

Furthermore, we compared DG and PG with stochastic sampling. I.e.using Eq 25 for Stochastic Density Guidance (SDG), and Eq 7 for “Stochastic Prior Guidance” SPG (i.e. regular stochastic sampling after rescaling the latent code). We tested two scenarios: Adding noise early: $\varphi(t)=0.2g(t)$ for $\log SNR(t) < -4.03$, and $\varphi(t)=0$ otherwise; Adding noise late $\varphi(t)=0.3g(t)$ for $\log SNR(t) > -3$, and $\varphi(t)=0$ otherwise.

We found the correlation between the hyperparameter and the obtained $\log p_0$ to be 50% for SDG and 25% for SPG. We summarize all correlations in the table below.

For DG the drop in correlation from deterministic to stochastic sampling can be explained by the same reasoning as for the EQM, i.e. stochastic sampling requires significantly more sampling steps to achieve the desired levels of $\log p_0$ (Figure 18).

For PG, stochastic sampling is not principled, i.e. the more noise we add during sampling, the less information is contained in the starting point $x_T$. For example, if $\varphi(t)=g(t)$ for all $t$, then the process is the Reverse SDE, and $p(x_0|x_T)$ does not depend on $x_T$, and thus scaling the latent code has no effect on the final sample. Hence the need for Density Guidance for stochastic sampling.

Please see Fig 20 for details on the evaluation of log-densities and corresponding PNG files sizes, and Fig 21 for the visualization of the stochastic samples.

Additional Experiments

We have also added the following:

• Analysis of the impact of Guidance on perceptual metric NIQE (Fig 14, more details in the response to Reviewer 8ij3)
• More samples and quantitative results with Stable Diffusion (Fig 19)
• Results with models using Classifier-Free Guidance (Fig 22, more details in the response to Reviewer reGE, Q9)
• A new State-of-the-art model FLUX (Fig 23-25, more details in the response to Reviewer reGE, under SOTA models)
• A rigorous proof of the hypothesis we posed in Appendix D about the asymptotic behaviour of $h(x)$ for the Gaussian Mixture (Theorem 1 in the uploaded file)

We thank the Reviewer again for their constructive feedback, which strengthened our claims and improved the quality of our submission. We hope that we have adequately addressed the concerns, and you will consider raising your score.

We thank the Reviewer for their thorough evaluation of our work and very insightful questions! We address the raised points below.

File with new figures: https://anonymous.4open.science/r/DensityGuidance-20E6/Density_guided_sampling___Rebuttal.pdf

Glossary:

• Density Guidance - DG
• Prior Guidance - PG
• Score Alignment - SA
• Classifier-free guidance - CFG

Q1: Only two models analyzed for SA, and no theoretical guarantee.

This is true that there is no theoretical guarantee for SA. There cannot be such a guarantee as we show:

• In Fig 3, right - for the CIFAR model, SA does not hold for 3% of the latent codes (line 209)
• In Appendix C.3, we provide an example of a Gaussian mixture (exact scores known), where SA does not hold.

This emphasizes the point from the paper: SA does not always hold. This in part motivates the DG approach, because PG is not always guaranteed to work.

We discuss FLUX at the end of the response.

Q2: no quantitative evidence of Explicit Quantile Matching.

Please see our response to Reviewer ppbq.

Q3: Can you prove that Eq 21 implies Eq 18?

We actually do not make that claim. Eq 21 is based on results in Appendix D, which we have now extended by a rigorous proof in the Gaussian Mixture case (Theorem 1). The motivation is to keep the samples in the typical regions of $p_t$, but we do not guarantee exact quantiles.

Q4: Few StableDiffusion samples. Also, semantic changes visible.

We have included more levels for Stable Diffusion, with PNG sizes (Fig 19). Regarding semantic changes - this is true, and can be even more drastic as in the train example with PG in Fig 19. We do not guarantee that only the low-level features change in all cases. In the extremes semantic changes can happen as well. However, it is consistent with the amount of detail as measured by PNG.

Q5: Quantitative analysis of DG

Please see our response to Reviewer ppbq.

Q6: Only two levels in Fig 10. Does stochastic guidance differ from PG?

We added more samples and levels in Fig 21 for both DG and PG. Perceptually, both seem to be monotonically controlling the detail, but we argue in the response to Q5: DG is more accurate.

Q7: What are practical application scenarios?

Due to character limit, please refer to our response to Reviewer J9Xk, where we discuss potential applications.

Q8: When might the method fail?

A potential issue is applying DG in low dimensions. We use the fact that $h(x)$ is approximately Gaussian (Appendix D). This only holds when the dimensionality is large.

Another approximation we make is discussed in lines 1147-1161. We justify it on two datasets. If one wants to increase the accuracy further, Eq 119 can be used instead, which makes no approximations. It comes at a cost of one additional Jacobian-Vector-Product.

Q9: Would the method work with CFG?

As explained in the CFG paper, CFG can be interpreted as classifier guidance with an implicit classifier. This means that CFG is just a regular diffusion process, but with a different base distribution, favouring a certain class. Thus, all our results apply without changes, just with a redefined target distribution.

We sampled with an EDM2 model with CFG and found consistent behaviour with other models. Please see Fig 22.

L175, "$v_t$" is not discussed

We denote by $v_t$ the score pushed forward from $T$ to $t$. We refer to it later in the text, e.g. in Eq 12. We will make it more explicit in the final revision.

No reference for Eq. 3.

The reference is Chen et al. 2018. We will make it more explicit.

I didn't find such claims in Song et al. 2021b

It can be seen in the Appendix in Figure 6 - it is not referenced in the main text. Authors call it "temperature rescaling" (reducing norm of embedding).

Will the methods work for SOTA models like FLUX?

We have included new results with FLUX.1[dev]. Fig 23 shows samples with PG and DG, and Fig 24 shows a DG image with PNG and TIFF filesize comparison. Fig 25 shows the coupling of PNG and TIFF filesizes over guidance.

FLUX shows different behavior to other models. Images are richer, but detail variations from guidance are milder. The weaker DG effect can be attributed to the FLUX model coupling a latent diffusion on 16x64x64 space with a strong decoder to 3x768x768, whose effect on logp is unknown. We only control the latent portion of the model. FLUX is undocumented and unpublished.

The effect of DG on PNG filesizes also becomes inconsistent: adding more semantic detail doesn’t necessarily increase filesize, possibly due to the images already being highly realistic and rich in patterns. Furthermore, PNG is only an approximation of the true information content of the image. We include comparison to TIFFs, which shows more consistent coupling between filesize and detail.

We thank the Reviewer again for their high-quality review that significantly contributed to improving our submission.

We would like to thank the Reviewer for their support of our work. Below we address the raised concern.

File with new figures: https://anonymous.4open.science/r/DensityGuidance-20E6/Density_guided_sampling___Rebuttal.pdf

I would appreciate it if the authors could provide practical advice on how the proposed density guidance might improve current sampling methods.

The Density Guidance works on any diffusion model without retraining, or finetuning and requires no extra cost during sampling. We now ran a new experiment and measured the predictions of the generated samples using NIQE [1], which is a metric for image quality assessment reported to correlate strongly with human judgement.

Potential applications of detail control

We believe that potential applications of the presented methods are image editing, where the user might want to control the amont of detail in the image. From [2] we also know that highest densities contain cartoon-like images, and thus user can have fine grained control over the spectrum between realistic images and cartoons describing the same scene. People also acknowledge that image generation can be used for “aiding designers in producing striking scenes for video games” article and we believe that detail control can become an addition to that toolkit.

We also note that there has been interest among the practitioners in explicitly controlling the amount of detail in image generation:

• Modifying Stable Diffusion to generate less detail Thread
• Modifying Stable Diffusion to generate more detail Thread

Finally, we would also like to point out that the density-guidance is derived to control the log-density of the generated samples. We know from prior literature [2], that for image data, this correlates with image detail. Perhaps in domains different than images, controlling log-density may be desirable for other purposes. Density Guidance can be used for that as well. Investigation of domains other than image data is out of scope for this work but certainly an interesting direction that we hope this work could pave way for.

We hope our response has addressed the Reviewer's concerns, and that the additional experiments provided in the uploaded file further strengthen your support for our submission, which we hope will be reflected in an updated score.

---

[1] Mittal et al. "Making a “completely blind” image quality analyzer" (IEEE Signal processing 2012)

[2] Karczewski et al. "Diffusion Models as Cartoonists: The Curious Case of High Density Regions" (ICLR 2025)

We thank the Reviewer for their positive comments and constructive feedback. We address the raised points below.

File with new figures: https://anonymous.4open.science/r/DensityGuidance-20E6/Density_guided_sampling___Rebuttal.pdf

Discuss some papers about perceptual quality metrics and detail-preservation.

Certainly. In the camera-ready revision we will include a discussion on perceptual quality including the reference-based metrics such as the ones suggested (LPIPS, FID, SSIM), as well as no-reference-based, such as NIQE [3].

The user study or perceptual evaluation is missing. LPIPS, FID, SSIM or user study could be added to strengthen the claims.

Thank you for this suggestion. The metrics proposed are "Reference-based" metrics, which compare the generated images to reference ones. LPIPS and SSIM score a single image against a single reference image, while FID compares the set of generated images to the set of "real" images. The issue with LPIPS and SSIM is that, for a given generated image, we do not have a corresponding "ground truth" image to compare to. FID would have been more suitable for our use-case, however it is computationally expensive, requiring generating tens of thousands images [1], which was beyond our coomputational budget. It has also been reported to not always agree with human judgement [2].

We thus propose to use NIQE [3], which is a "no-reference" image quality evaluation metric, which is reported to correlate strongly with human judgement. It provides a single number per image, which indicates whether an image has been distorted (a lower number - higher quality). It was used e.g. by [4] to evaluate super-resolution diffusion models.

We evaluated Density and Prior Guidance for EDM2 model, and the now included State-of-the-art model FLUX. In Figure 14 you can see that:

• For EDM2 model: guided samples can obtained better (lower) NIQE scores than regular samples (gray area);
• For FLUX model: regular samples already score optimally.

After a visual inspection of optimally scoring guided samples (as measured by NIQE) in Figure 15, we noticed that NIQE actually prefers images with significantly less detail than regular samples. For the FLUX model (Figure 16), there were no perceptual differences between regular samples and best NIQE scoring ones.

The paper introduces [...] score alignment, density guidance, and stochastic density guidance, but it does not conduct a ablation study to evaluate the contributions of each component.

Thank you for this question. We take this opportunity to clarify:

• Score alignment (SA) is a novel framework to verify whether a known procedure (Prior Guidance) will be effective in practice
• (Stochastic) Density Guidance is a novel algorithm proposed by us, which is principled and can be used with any diffusion model (regardless of whether SA holds)

That said, we now performed an extensive evaluation of Prior and Density Guidance, both quantitative and qualitative, including novel models. Please see our response to Reviewer ppbq for more details.

How does density guidance compare to methods that use explicit perceptual loss functions, such as LIPIS for controlling detail?

The difference between Density Guidance (DG) and models trained with perceptual loss functions is two-fold. First, DG can be used, without any finetuning or retraining, and for no extra cost, on models which were trained without any perceptual losses. Second, it provides a way to control the generations. One can generate images with either high or low level of detail, depending on the use-case. Models trained with perceptual losses do not have that capability.

What is the relationship between preceptual metrics and score alignment?

Score Alignment guarantees that Prior Guidance effectively changes log-density in deterministic sampling. We show in Figure 14 how that can impact the perceptual metrics.

How different does the model perform in ODE sampling the stochastic sampling with Density Guidance?

Please see our response to Reviewer ppbq for the details on the comparison on different modes of sampling.

We thank the Reviewer again for their useful suggestions that helped improve our work. We hope that our clarifications and additional experiments addressed all concerns and ask for a reconsideration of the score.

---

[1] Heusel et al. "GANs trained by a two time-scale update rule converge to a local Nash equilibrium." (NeurIPS 2017)

[2] Liu et al. "An improved evaluation framework for generative adversarial networks." (arXiv 2018)

[3] Mittal et al. "Making a “completely blind” image quality analyzer" (IEEE Signal processing 2012)

[4] Sami et al. "HF-Diff: High-Frequency Perceptual Loss and Distribution Matching for One-Step Diffusion-Based Image Super-Resolution." (arXiv 2024)