Feedback Guidance of Diffusion Models

First and foremost we would like to thank the reviewer for their detailed yet concise feedback. We appreciate that you recognize that our method is supported by strong mathematical foundations as well as the fact that it brings interesting insights in the context of prompt/trajectory-specific behavior.

Response to Weakness #1: Computational overhead

One of the key advantages of the iterative posterior estimation approach is that it comes at negligible additional cost as it solely relies on precomputed quantities, namely the conditional and unconditional score functions already required for the denoising itself. To further emphasize this point we have added a sentence in the main document in section 3.3 mentioning:

“A crucial advantage of computing the posterior using the scheme describe above is that it causes negligible computational overhead as all required quantities, in particular $\mathbf{\mu}_c(\mathbf{x}_t|c)$ and $\mathbf{\mu}(\mathbf{x}_t)$, are already computed.”

Response to Weakness #2: Posterior estimation parametrisation

The specific choice of parametrizing the posterior through the additional bias parameter $\delta$ is described in detail in appendix B2. While it is true that we could have made other choices, we believe that the choice of a linear transformation is well defended in the paper. It should also be pointed out, that even though this is the simplest possible choice, it still manages to lead to a method outperforming standard baselines, which we believe to be a strong demonstration of the potential of our approach. In the future, we are looking forward to seeing more work analyzing the potential of different parametrizations of the posterior.

Response to Weakness #3: Limited evaluation

Finally, we wish to address the reviewer's concern that the small-scale experiments may not be sufficient for assessing the method's significance. We agree that strengthening the paper in this regard would significantly improve its quality and impact.

Extension from small EDM2-XS to larger EDM2-L

To address this, during the rebuttal period we conducted the same analysis previously performed with the EDM2-XS model using the much larger EDM2-L model, providing more comprehensive and qualitative insights. The initial results, computed on 10k generated images, are presented in Table 1 below. The extension towards 50k generated images, as well as the addition of the Precision and Recall metrics, is under way but will not be ready in time for the initial rebuttal.
We can observe that the results on EDM2-L are more similar for the different methods, compared to the small model. This is because the EDM2-L model is far more qualitative and requires much less guidance, a result expected from our interpretation of guidance as an error correcting scheme. As a consequence, all guidance approaches are more similar to each other.

Table 1: Performance comparison on EDM2-L (10k images)

Additional T2I results with Stable Diffusion 2 on MS-COCO 30k

Similarly, to further support our findings in the T2I setting, we evaluated the FID, FD$_{\text{DinoV2}}$ alongside the average CLIP and Aesthetic Scores for different guidance schemes (CFG, LIG, and our proposed FBG) using Stable Diffusion 2 on the standard MS-COCO 30k dataset. These results, currently limited to a subset of 3k prompts, are reported in Table 2. Results on all 30k prompts will become available next week.
The MS-COCO30k dataset contains prompts that originate from captioned images, which all remain fairly standard. We expect our approach to work even better in context where the prompts are much more varied. To remain consistent with the literature we however still chose to evaluate on this standard benchmark, on which FBG still slightly outperforms alternative approaches.

Table 2: Quantitative evaluation of T2I on MS-COCO benchmark (3k images)

$^*$In this setting, the best LIG interval corresponded to applying guidance on the whole trajectory.

We hope that these results help convince the reviewers of the more general validity of the method, as well as of our commitment to incorporate their valid critique to improve the current submission.

Summary

We appreciate the reviewer's concerns and questions, and are convinced our responses as well as the additional experimental results have the potential to improve the manuscript considerably. We hope the reviewer reassesses the manuscript in light of this rebuttal, to see whether it warrants increasing their original score.

We would like to start by thanking the reviewer for their acknowledgement of the novelty of our approach. Furthermore, we are glad that they found the paper both interesting and easy to follow.

Response to Weakness #1: Comparison to other baselines

The reason that our main focus was on comparing our method to CFG and LIG is because these two guidance methods are respectively the most popular and the most performant in the literature. We have, internally, played with other alternative fixed schedulers such as those described in the “Analyzing weight-schedulers” paper [1], but our results have always demonstrated that choosing a finite guidance interval as proposed in LIG always surpassed these. Our understanding is that guidance at the beginning of the diffusion process always has a negative impact on FID, which means that only the sine, linear or $\Lambda$ guidance shapes are prominent options. All three are however active too early, having detrimental effect on the performance. Furthermore, we wanted the main focus to be on the spatio-temporal dependence of our guidance scheme, which is why we did not compare to methods such as CFG++, a reference which we have now aded to the related work section.

[1] WANG, at, al. Weight-Scheduler: Analysis of classifier-free guidance weight schedulers.

Response to Weakness #2: Unclear derivations/metrics

For the description of the metrics, such as FD$_{\text{DinoV2}}$ or Precision/Recall, see our answer to Question #2.

To make sure that the derivations are easily tractable, we have added two additional lines in the derivation of the main body as well as a detailed derivation in the appendix. Equation 7 now reads:
Seeing that this was pointed out by two of the reviewers, we have added two additional lines in the derivation of the main body as well as a detailed derivation in the appendix. Equation 7 now reads:

\\begin{split} \\nabla_x\\log p(x_t \\mid c)&=\\nabla_x\\log\\big(p_{\\theta,t}(x_t|c)-(1-\\pi)p_{\theta,t}(x_t)\big)\\\\&=\\frac{1}{p_{\\theta,t}(x_t|c)-(1-\\pi)p_{\\theta,t}(x_t)}\\big(\\nabla_x p_{\\theta,t}(x_t|c)-(1-\\pi)\nabla_x p_{\\theta,t}(x_t)\\big)~\\\\&= \\nabla_x\\log p_{\\theta,t}(x_t) +\\lambda(x_t, t)\\Big(\\nabla_x\\log p_{\\theta,t}(x_t \\mid c)-\\nabla_x\\log p_{\\theta,t} (x_t)\\Big) ~\\ \\end{split}

We have also added an additional inline equation:

"To obtain the last equation, we make use of the identity $\nabla_x p(x) = p(x) \nabla_x \log p(x)$ and then reorder the terms to obtain a familiar looking guidance equation."

We thank the reviewer for pointing this out, seeing that the theoretical underpinning of our approach is fundamental to our work it is of uttermost important that the derivation of our guidance scheme is clear.

Response to Weakness #3: Insufficient Experiments

We wish to address the reviewer's concern that the small-scale experiments may not be sufficient for assessing the method's significance. We agree that strengthening the paper in this regard would significantly improve its quality and impact.

Extension from small EDM2-XS to larger EDM2-L

To address this, during the rebuttal period we conducted the same analysis previously performed with the EDM2-XS model using the much larger EDM2-L model, providing more comprehensive and qualitative insights. The initial results, computed on 10k generated images, are presented in Table 1 below. The extension towards 50k generated images, as well as the addition of the Precision and Recall metrics, is under way but will not be ready in time for the initial rebuttal.
We can observe that the results on EDM2-L are more similar for the different methods, compared to the small model. This is because the EDM2-L model is far more qualitative and requires much less guidance, a result expected from our interpretation of guidance as an error correcting scheme. As a consequence, all guidance approaches are more similar to each other.

Table 1: Performance comparison on EDM2-L (10k images)

Additional T2I results with Stable Diffusion 2 on MS-COCO 30k

Similarly, to further support our findings in the T2I setting, we evaluated the FID, FD$_{\text{DinoV2}}$ alongside the average CLIP and Aesthetic Scores for different guidance schemes (CFG, LIG, and our proposed FBG) using Stable Diffusion 2 on the standard MS-COCO 30k dataset. These results, currently limited to a subset of 3k prompts, are reported in Table 2. Results on all 30k prompts will become available next week.
The MS-COCO30k dataset contains prompts that originate from captioned images, which all remain fairly standard. We expect our approach to work even better in context where the prompts are much more varied. To remain consistent with the literature we however still chose to evaluate on this standard benchmark, on which FBG still slightly outperforms alternative approaches.

Table 2: Quantitative evaluation of T2I on MS-COCO benchmark (3k images)

$^*$In this setting, the best LIG interval corresponded to applying guidance on the whole trajectory.

We hope that these results help convince the reviewers of the more general validity of the method, as well as of our commitment to incorporate their valid critique to improve the current submission.

Response to Question #1: Prompt alignment metric

CFG serves not only to improve prompt alignment but also to maintain overall image quality, as DMs underperform on all metrics without guidance. Its key advantage is letting users trade image diversity for stronger alignment by adjusting the guidance scale. This makes FBG’s compatibility with other methods like CFG and LIG particularly valuable: FBG can improve baseline quality, while the guidance scale remains available for users seeking stronger alignment. Given the relevance of metrics like CLIP‑score and Aesthetic scores in T2I, we report these alongside FID and FD in our Stable Diffusion experiments.

Response to Question #2: Metric explanations

We thank the reviewer for pointing out the absence of explanations regarding the nature of the used metrics. After all, these are not as common as the FID/CLIP-Score and should therefore be explained. To remedy this we have added a sentence describing the FD$_{\text{DinoV2}}$ metric:

“The FD-DinoV2, similarly to the FID, measures the Fréchet Distance between generated images and reference images inside the DinoV2 embedding space. The DinoV2 embedding space is trained on a large diverse unlabeled dataset, in stark contrast with the InceptionV3 network used for FID which is solely trained on ImageNet classes and is therefore insensitive to certain features such as human faces. ”

as well as a sentence describing the Precision-Recall metrics:

“The precision and recall metrics, very useful to analyze the quality-diversity tradeoff of a method, measure respectively how many of the generated images lie close to the reference image manifold (quality) and inversely how many of the reference image lie close to the generated image manifold (diversity). ”

Response to Question #3: Computational overhead of FBG

One of the key advantages of the iterative posterior estimation approach is that it comes at negligible additional cost as it solely relies on precomputed quantities, namely the conditional and unconditional score functions already required for the denoising itself. To further emphasize this point we have added a sentence in the main document in section 3.3 mentioning:

“A crucial advantage of computing the posterior using the scheme describe above is that it causes negligible computational overhead as all required quantities, in particular $\mathbf{\mu}_c(\mathbf{x}_t|c)$ and $\mathbf{\mu}(\mathbf{x}_t)$, are already computed.”

Response to Question #4: Comparison to other deterministic weight schedulers

The obtained state- and time-dependent guidance scale shares some aspects with simple predefined guidance schemes, such as LIG or some schemes described in the “Analyzing CFG weight schedulers” work [1]. In particular, we find that our guidance scale is small during the early stages of diffusion, increases during intermediate stages and finally decreases towards later stages. These trends are perfectly in line with the best performing weight-schedulers described in the aforementioned work [1] as well as with the scheduler proposed by LIG.

Response to Question #5: Implementation of FBG_CFG

The Hybrid FBG-CFG scheme is implemented in the provided code under the name of ‘Hybrid_CFG_FBG’. We have further added an algorithm, as well as a paragraph in the appendix section describing how precisely the two guidance schemes can be combined. Essentially, the easiest way to understand this hybrid scheme is to see the total guidance scale as the sum of the two terms, i.e. $\lambda_{\text{tot}}(x,t) = \lambda_{\text{FBG}}(x,t) + \lambda_{\text{CFG}}$.

Summary

We appreciate the reviewer's concerns and questions, and are convinced our responses as well as the additional experimental results have the potential to improve the manuscript considerably. We hope the reviewer reassesses the manuscript in light of this rebuttal, to see whether it warrants increasing their original score.

First and foremost, we would like to thank the reviewer for their in depth review of our manuscript. We appreciate that the reviewer recognizes the novelty of the dynamic posterior dependent guidance scale. We are happy that they found the work interesting, and also thank them for having taken the time to come up with interesting and challenging questions that, in our eyes, will improve the paper.

Response to Weakness #1: Requirement for more exhaustive experiments

We first address the reviewer's concern that the small-scale experiments may not be sufficient for assessing the method's significance. We agree that strengthening the paper in this regard would significantly improve its quality and impact.

Extension from small EDM2-XS to larger EDM2-L

To address this, during the rebuttal period we conducted the same analysis previously performed with the EDM2-XS model using the much larger EDM2-L model, providing more comprehensive and qualitative insights. The initial results, computed on 10k generated images, are presented in Table 1 below. The extension towards 50k generated images, as well as the addition of the Precision and Recall metrics, is under way but will not be ready in time for the initial rebuttal.
We can observe that the results on EDM2-L are more similar for the different methods, compared to the small model. This is because the EDM2-L model is far more qualitative and requires much less guidance, a result expected from our interpretation of guidance as an error correcting scheme. As a consequence, all guidance approaches are more similar to each other.

Table 1: Performance comparison on EDM2-L (10k images)

Additional T2I results with Stable Diffusion 2 on MS-COCO 30k

Similarly, to further support our findings in the T2I setting, we evaluated the FID, FD$_{\text{DinoV2}}$ alongside the average CLIP and Aesthetic Scores for different guidance schemes (CFG, LIG, and our proposed FBG) using Stable Diffusion 2 on the standard MS-COCO 30k dataset. These results, currently limited to a subset of 3k prompts, are reported in Table 2. Results on all 30k prompts will become available next week.
The MS-COCO30k dataset contains prompts that originate from captioned images, which all remain fairly standard. We expect our approach to work even better in context where the prompts are much more varied. To remain consistent with the literature we however still chose to evaluate on this standard benchmark, on which FBG still slightly outperforms alternative approaches.

Table 2: Quantitative evaluation of T2I on MS-COCO benchmark (3k images)

$^*$In this setting, the best LIG interval corresponded to applying guidance on the whole trajectory.

We hope that these results help convince the reviewers of the more general validity of the method, as well as of our commitment to incorporate their valid critique to improve the current submission.

Response to Weakness #2: Introduction of $t_0$ and $t_1$

Our Feedback Guidance (FBG) approach, like Limited Interval Guidance (LIG), has three hyperparameters, making initial tuning harder than CFG. However, unlike LIG, FBG is derived from first principles, offering insights into why guidance is needed. Its main advantage is that it automatically detects when and where strong guidance is required, removing the need for a changeable guidance scale—unlike CFG or LIG, which use fixed schedulers independent of prompt or state. Once valid hyperparameters are set at the model/application level, FBG self‑regulates guidance across prompts and timesteps, focusing on parts needing the strongest correction, as shown in our T2I results. We acknowledge the concern about introducing these hyperparameters, which is why we added:

“The hyperparameters introduced for FBG are conceptually interpretable and need to be set only once. They allow the model to self-regulate guidance strength across prompts and timesteps, reducing the need for extensive prompt specific tuning compared to fixed-scheduler approaches.”

Response to Weakness #3: Unclear statements

We have gone through the manuscript in close detail again to remove any possibly confusing statements, such as the typos mentioned (such as the “DCFG” present in Table 1, which was the previous acronym we had internally used for our FBG scheme and which was unfortunately left unchanged before the submission). In this regard, we also spend close attention to Q4 and Q5, for which additional sentences/derivations have been added to the main document.

Response to Question #1: Selection of $t_0$ and $t_1$

While $t_0$ and $t_1$ require manual tuning, we designed them to be highly interpretable, simplifying this process. As outlined, they correspond to the average moments guidance starts and stops (similar to $t_{\text{start}}$ and $t_{\text{end}}$ in LIG), and intuitive reasoning—e.g., earlier or wider intervals producing greater changes—helps calibrate them. Figures 3a and 8a further show that across a reasonable range of $t_0$/$t_1$ values, FBG outperforms CFG on EDM2‑XS using FD$_{\text{DinoV2}}$ and FID. While making $t_0$ and $t_1$ dynamic (e.g., automatically adjust to earlier timesteps for low‑frequency features like landscapes) is a promising direction we are highly interested in, we believe our fixed‑parameter approach already represents a significant advance over static guidance schemes.

Response to Question #2: Larger scale models + Flow Matching models

For the analysis of larger scale models we refer to our answer of Weakness #1.
Regarding the adaptability to Flow Matching Models, although left as future work, it is our belief that a very similar Feedback Guidance scheme can easily be derived and described in this context. The novel ideas introduced, such as the dynamic posterior dependent guidance scale, are in no way connected to the specific choice of diffusion models described in this paper. We are very excited about future works looking into extending the framework of Feedback Guidance to Flow Matching models, Stochastic Interpolators or Schrödinger Bridges.

Response to Question #3: Evaluation of T2I

In the context of T2I, we have focused on obtaining FID/FD$_{\text{DinoV2}}$ values as well as CLIP and Aesthetic Scores on MS-COCO 30k, on which, similarly to what was obtained using EDM2-XS, FBG outperforms CFG and performs on par with LIG. We also agree with the reviewer that careful visualizations are key to understanding a paper. To improve on the current, limited, illustrative examples, we have have prepared multiple additional images similar to Fig. 4a and Fig. 5 that will be added to the appendices. Unfortunately, to remain in line with the NeurIPS rebuttal guidelines, we can not share them at this stage.

Response to Question #4: Unclear derivation of the dynamic guidance scale

Seeing that this was pointed out by two of the reviewers, we have added two additional lines in the derivation of the main body as well as a detailed derivation in the appendix. Equation 7 now reads:

\\begin{split} \\nabla_x\\log p(x_t \\mid c)&=\\nabla_x\\log\\big(p_{\\theta,t}(x_t|c)-(1-\\pi)p_{\theta,t}(x_t)\big)\\\\&=\\frac{1}{p_{\\theta,t}(x_t|c)-(1-\\pi)p_{\\theta,t}(x_t)}\\big(\\nabla_x p_{\\theta,t}(x_t|c)-(1-\\pi)\nabla_x p_{\\theta,t}(x_t)\\big)~\\\\&= \\nabla_x\\log p_{\\theta,t}(x_t) +\\lambda(x_t, t)\\Big(\\nabla_x\\log p_{\\theta,t}(x_t \\mid c)-\\nabla_x\\log p_{\\theta,t} (x_t)\\Big) ~\\ \\end{split}

We have also added an additional inline equation:

"To obtain the last equation, we make use of the identity $\nabla_x p(x) = p(x) \nabla_x \log p(x)$ and then reorder the terms to obtain a familiar looking guidance equation."

We thank the reviewer for highlighting this, as the theoretical underpinning is fundamental to our work, making a clear derivation of our guidance scheme is essential.

Response to Question #5: Rationale behind additive vs. multiplicative assumption

The interpretation of guidance schemes through error assumption models is in our eyes one of the most relevant contributions of our work. We agree with the reviewer that explaining the rationale behind the choice of an additive error model is important, which is why we have added two sentences in the main body.

“The additive assumption can be seen as less restrictive than the multiplicative one as it allows the learned conditional distribution $p_{\theta,t}(x_t|c)$ to be non-zero in regions where the true conditional distribution $p_t(x_t|c)$ is zero, a feat the multiplicative assumption is incapable of. Due to the joint training pipeline, and the fact that training pairs often contain more than a single element, such an overlap of the learned distributions is in practice highly likely.“

We would however like to emphasize that our point is not that the additive error assumption is fully representative of the true error of the learned model, but merely that this new assumption might lead to different properties for the guidance scheme. In the future, analyzing different, more intricate, error models might lead to further findings, which we look forward to analyzing.

Summary

We appreciate the reviewer's concerns and questions, and are convinced our responses as well as the additional experimental results have the potential to improve the manuscript considerably. We hope the reviewer reassesses the manuscript in light of this rebuttal, to see whether it warrants increasing their low original score.

We would first like to thank the reviewer for their detailed analysis, which recognizes the quality, originality and significance of our work. In particular, we appreciate the reviewer’s acknowledgments of our efforts to provide clear and usable code, as well as realizing that our proposed scheme comes with negligible computational overhead when compared to CFG.

Response to Weakness #1: limited experiments

We first address the reviewer's concern that the small-scale experiments may not be sufficient for assessing the method's significance. We agree that strengthening the paper in this regard would significantly improve its quality and impact.

Extension from small EDM2-XS to larger EDM2-L

To address this, during the rebuttal period we conducted the same analysis previously performed with the EDM2-XS model using the much larger EDM2-L model, providing more comprehensive and qualitative insights. The initial results, computed on 10k generated images, are presented in Table 1 below. The extension towards 50k generated images, as well as the addition of the Precision and Recall metrics, is under way but will not be ready in time for the initial rebuttal.
We can observe that the results on EDM2-L are more similar for the different methods, compared to the small model. This is because the EDM2-L model is far more qualitative and requires much less guidance, a result expected from our interpretation of guidance as an error correcting scheme. As a consequence, all guidance approaches are more similar to each other.

Table 1: Performance comparison on EDM2-L (10k images)

Additional T2I results with Stable Diffusion 2 on MS-COCO 30k

Similarly, to further support our findings in the T2I setting, we evaluated the FID, FD$_{\text{DinoV2}}$ alongside the average CLIP and Aesthetic Scores for different guidance schemes (CFG, LIG, and our proposed FBG) using Stable Diffusion 2 on the standard MS-COCO 30k dataset. These results, currently limited to a subset of 3k prompts, are reported in Table 2. Results on all 30k prompts will become available next week.
The MS-COCO30k dataset contains prompts that originate from captioned images, which all remain fairly standard. We expect our approach to work even better in context where the prompts are much more varied. To remain consistent with the literature we however still chose to evaluate on this standard benchmark, on which FBG still slightly outperforms alternative approaches.

Table 2: Quantitative evaluation of T2I on MS-COCO benchmark (3k images)

$^*$In this setting, the best LIG interval corresponded to applying guidance on the whole trajectory.

We hope that these results help convince the reviewers of the more general validity of the method, as well as of our commitment to incorporate their valid critique to improve the current submission.

Response to Weakness #2: no human evaluation

We fully agree with the reviewer that in terms of quality metrics, the end goal remains human evaluation. Unfortunately, in the short amount of time available during the rebuttal, performing such an analysis remained unfeasible. Instead, we rely on common practice in the field, which has demonstrated the strong correlation between human preferences and FD metrics, more particularly the FD$_{\text{DinoV2}}$ [1]. In the context of T2I we believe that the a much wider ranger of metrics is preferable, which is why we chose to also provide both CLIP-Score and Aesthetic Scores. We hope that these additional metrics help convince the reviewer of the validity of the proposed evaluation approach.

[1] Stein et al., “Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models”

Response to Question 1: experiments on text-to-video?

Regarding the analysis of larger models, we refer the reviewer to Response to Weakness #1. Unfortunately, due to the short time available during the rebuttal phase, testing in the context of T2V, or on DiT, was not feasible with the computational resources available. We hope that our evaluation setting of choice (in line with the tradition of image benchmarks and metrics, to validate research on diffusion models) remains acceptable to the reviewer.

Response to Question 2: on underspecified prompts

As no experiments have been performed in the context of T2V, we can not comment on this specific setting. However, the questions remains interesting and insightful in the context of T2I. Our dynamic guidance method adjusts according to the guidance needed for each prompt. This does not mean that less detailed (“easier”) prompts are less well followed; on the contrary, their lack of specificity allows for more diverse secondary features. The key point is that not all prompts require the same level of guidance, and low guidance does not necessarily imply low quality. In this regard FBG provides a key edge over established methods such as CFG which always applies the same amount of guidance regardless of the amount of detail and/or complexity of the prompt. This is well highlighted by the results shown in Fig. 4 of the paper.

Response to Questions 3-4: on iterations of: mix scores + take step + update guidance

An important note is that the graph displayed in Fig. 2 is unrolled in time, implying that applying more iterations corresponds to a change in the total number of steps used in the diffusion process. Theoretically, the more steps are used the more accurate the posterior approximation becomes. To avoid any future confusion for readers we have decided to add the following sentence after line 204:

“This control diagram is progressively unrolled during the denoising process, implying a succession of computing the score functions, mixing them, applying a denoising step, updating the value of the guidance scale and repeating a fixed amount of times until a fully denoised image is obtained.”

As our main focus lied on a theoretically grounded method, we preferred testing the use of different solvers (2nd order Heun and stochastic) which operate in fundamentally distinct ways. The analysis of varying the number of sampling steps, such as analyzing the case of few-step generation which we considered as non-central to our work, was left as future work.

Response to Question 5: add prompt alignment metrics

To beter evaluate the quality of the proposed approach we evaluated the T2I results using a wider range of metrics, including the FID,FD$\_{\text{DinoV2}}$ , CLIP-Score and the Aesthetic Score. We strongly agree with the reviewer that in the case of T2I with the absence of human evaluation, using at least a variety of metrics is key. However, we would like to point out that in the context of Imagenet, which contains almost exclusively well-centered, well-lit images, these metrics can be seen as slightly less relevant than the provided FD$_{\text{DinoV2}}$ and the Precision and Recall curves, which also measure the quality-diversity trade-off of the method [1]. We hope that the reviewer is satisfied by the additional metrics, on which FBG also outperforms concurrent approaches.

Summary

We appreciate the reviewer's concerns and questions, and are convinced our responses as well as the additional experimental results have the potential to improve the manuscript considerably. We hope the reviewer reassesses the manuscript in light of this rebuttal, to see whether it warrants increasing their original score.

We would first like the reviewer for the in depth reading of our manuscript. In particular, we are glad that they appreciate the novelty of our novel state- and time-dependent guidance approach as well as its solid theoretical underpinnings.

Response to Weakness #1: Computational overhead

One of the key advantages of the iterative posterior estimation approach is that it comes at negligible additional cost as it solely relies on precomputed quantities, namely the conditional and unconditional score functions already required for the denoising itself. To further emphasize this point we have added a sentence in the main document in section 3.3 mentioning:

“A crucial advantage of computing the posterior using the scheme describe above is that it causes negligible computational overhead as all required quantities, in particular $\mathbf{\mu}_c(\mathbf{x}_t|c)$ and $\mathbf{\mu}(\mathbf{x}_t)$, are already computed.”

Response to Weakness #2: Description of Fig. 1/Table 1

We thank the reviewer for this valuable advice. We have included descriptive labels for the three lines present in Fig 1. which describe the three trajectories using the adjectives: “good”, “okay” and “poor”. Their meaning in the context of the generative trajectories is then further explained in the caption:

"Figure1: Illustrative diffusion trajectories and their hypothetical guidance scales in a 1D setting. Trajectories located further from the mode close to the decision window, such as the trajectories denoted by "okay" and "poor", receive higher guidance. On the contrary trajectories that are clearly going to the right mode, such as that denoted by "good", receive negligible guidance"

Regarding Table 1, the caption has been modified to explain the use of bold and underlined:

"We underline the best performing method per sampler, and further emphasize the best overall method using bold values."

Response to Weakness #3: FBG compared to LIG

We acknowledge that when measured using the FID, FBG does not outperform LIG, as mentioned in the abstract and recognized by the reviewer. Indeed, we solely claim performance comparable to LIG with as main advantage a robust theoretical underpinning.

Response to Weakness #4: Testing FBG on larger architectures

Finally we wish to address the reviewer's concern that the small-scale experiments may not be sufficient for assessing the method's significance. We agree that strengthening the paper in this regard would significantly improve its quality and impact. It should also be noted that the described theoretical framework is certainly not limited to a specific architecture and that FBG is fully compatible with models such as DiT. We believe this versatility, gained by rigorous mathematical foundations, to be one of the main strengths of our paper.

Extension from small EDM2-XS to larger EDM2-L

To address this, during the rebuttal period we conducted the same analysis previously performed with the EDM2-XS model using the much larger EDM2-L model, providing more comprehensive and qualitative insights. The initial results, computed on 10k generated images, are presented in Table 1 below. The extension towards 50k generated images, as well as the addition of the Precision and Recall metrics, is under way but will not be ready in time for the initial rebuttal.
We can observe that the results on EDM2-L are more similar for the different methods, compared to the small model. This is because the EDM2-L model is far more qualitative and requires much less guidance, a result expected from our interpretation of guidance as an error correcting scheme. As a consequence, all guidance approaches are more similar to each other.

Table 1: Performance comparison on EDM2-L (10k images)

Additional T2I results with Stable Diffusion 2 on MS-COCO 30k

Similarly, to further support our findings in the T2I setting, we evaluated the FID, FD$_{\text{DinoV2}}$ alongside the average CLIP and Aesthetic Scores for different guidance schemes (CFG, LIG, and our proposed FBG) using Stable Diffusion 2 on the standard MS-COCO 30k dataset. These results, currently limited to a subset of 3k prompts, are reported in Table 2. Results on all 30k prompts will become available next week.
The MS-COCO30k dataset contains prompts that originate from captioned images, which all remain fairly standard. We expect our approach to work even better in context where the prompts are much more varied. To remain consistent with the literature we however still chose to evaluate on this standard benchmark, on which FBG still slightly outperforms alternative approaches.

Table 2: Quantitative evaluation of T2I on MS-COCO benchmark (3k images)

$^*$In this setting, the best LIG interval corresponded to applying guidance on the whole trajectory.

We hope that these results help convince the reviewers of the more general validity of the method, as well as of our commitment to incorporate their valid critique to improve the current submission.

Summary

We appreciate the reviewer's concerns and questions, and are convinced our responses as well as the additional experimental results have the potential to improve the manuscript considerably. We thank the reviewer for reassessing the manuscript in light of this rebuttal.