We thank the reviewer for their valuable feedback. We are especially encouraged that they recognized the novelty of ERG as an inference-time attention modulation and appreciated its ability to improve both quality and diversity. We are glad they found the concept interesting and the paper well written. Their acknowledgment of ERG’s strong performance in the unconditional generation setting is very much appreciated.

Below we address all the questions raised by the reviewer.

W1. “Not enough transparency in terms of evaluation. Datasets should be stated more clearly.”
We thank the reviewer for raising this point. We discuss the datasets and the evaluation setup in Section 4.1.
We will make sure to update the experimental section to separate dataset setup and architecture details in order to improve clarity.
If the reviewer points to specific information that they think is missing from the text or not clearly explained, we would be happy to address it during the discussion period and update it in the paper.

W2. "The authors should include ImageNet 256 x 256 using a pretrained DiT-XL/2 and cite the original metrics from the respective papers." As suggested, we evaluated with DiT-XL/2 using the open-source implementation, and show the results below. The positive influence of ERG is clearly demonstrated in these results.

W3. "A comparison using sFID, InceptionScore, Recall and Precision would be good as this gives even more ground for fair comparisons."

We thank the reviewer for this suggestion. The extended list of metrics for our main results in Table 1 of the paper is presented below.

For precision & recall, we observe similar trends to density and coverage with ERG+CADS achieving the best precision (72.72). ERG on its own achieves a precision of 70.92 compared to 65.71 for CFG. Recall is slightly lower when using ERG alone (41.43 vs 43.95 for CFG), but ERG+APG achieves the best recall of 50.25. For Inception Score, ERG improves over the CFG baseline (43.16 vs 37.25 for CFG) and achieves the best score when combined with APG (53.47). Similarly, ERG+APG results in significant sFID and Inception score improvement, with ERG alone on par with APG.

Q1. “Why is ERG so much better in Coverage in the unconditional case while being only marginally better in FID and density?”

ERG shows a stronger improvement in coverage in the unconditional case, while gains in FID and Density are more modest. This is largely due to our multi-metric optimization strategy: we tune our models to perform well across all metrics—coverage, FID, and density—rather than optimizing exclusively for one.

Coverage captures the diversity of generated samples, whereas density reflects sample quality, and FID balances both. The significant boost in coverage specifically stems from applying a scheduled attention temperature, which allows ERG to gradually increase the influence of guidance over time. In contrast, SEG applies strong guidance from the very beginning, which tends to over-optimize for quality early in generation at the expense of diversity and consistency. This tradeoff is discussed in Section C.5 and illustrated in Figure 15 of the appendix.

Q2. “Why is there such a sharp change in measured performance in Fig 8?”

As discussed in Section C.4 of the appendix, we observe that attention layers exhibit distinct functional roles across the network, which can be broadly grouped into three stages: (1) encoding (layers 0–10, low entropy), (2) modeling (layers 10–25, high entropy), and (3) decoding (layers 25–38, low entropy). We find that modifying the rectification mechanism in either the encoding or decoding stages leads to sharp drops in performance due to the emergence of low-level artifacts in the generated outputs. This explains the abrupt change observed in Figure 8, and the same effect can be seen in Figure 16.

Thank you for raising this point.

After careful investigation we realized that for the DiT experiment, we reported FID using the autoguidance evaluation code while all other metrics (sFID, Precision, Recall, IS) were reported using the code from guided-diffusion. While both libraries use the same Inception network, they both provide a set of reference statistics for ImageNet which are different.

While evaluating the DiT generations (at 256 resolution) with the autoguidance codebase, we were using the ImageNet512 reference statistics under the hood, resulting in an incorrect setup, which explains the higher FID numbers.

We re-ran the experiment using the guided-diffusion library to match exactly the evaluation setup from the DiT paper for 10 different random seeds. The results of the rectified experiment are reported below.

For the baseline, we obtain an average FID of 2.38 which is still higher than the one reported in the paper but is plausible given the variance of the results.

Compared to the baseline, I-ERG still achieves a better FID (2.15 vs. 2.38), and given the statistics, also yields smaller variance on all metrics other than recall, showing more stable performance while varying random seeds.

Thank you once again for pointing out this inconsistency.

We thank the reviewer for their thoughtful and encouraging feedback. They highlighted the novelty of adapting attention layers at test time using a Hopfield energy perspective, as well as the simplicity and effectiveness of the proposed method. We appreciate their recognition of the thoroughness of our experimental evaluation across a wide range of metrics, models, and datasets. Their acknowledgment of the method's strong performance in both conditional and unconditional sampling scenarios is also greatly valued.

Below we address all the questions raised by the reviewer.

W1. Motivation for $\tau$ We thank the reviewer for highlighting the importance of the temperature hyperparameter $\tau$. While not part of the original Hopfield energy formulation, τ is a theoretically grounded extension that modulates the sharpness of the softmax in the LogSumExp term of the energy function (eq. (1) in the paper). Specifically, it enables control over the entropy of the attention distribution, allowing us to interpolate between diffuse and sharply peaked attention.

Mathematically, introducing $\tau$ rescales the similarity scores as $\log \sum_i \exp(\tau \beta x_i^T \psi)$, corresponding to a softmax temperature adjustment.

From a statistical mechanics perspective, $\tau$ acts as an inverse temperature in a Boltzmann distribution, balancing exploration (low $\tau$, high entropy) and exploitation (high $\tau$, low entropy) when matching attention keys and queries.

Crucially, introducing $\tau$ does not break the concave-convex structure of the energy function, and the CCCP optimization remains valid and convergent. This allows us to embed $\tau$ directly into the energy landscape and perform test-time entropy manipulation in a principled manner.

Unlike prior methods such as SEG, which apply local Gaussian smoothing post hoc, our approach integrates $\tau$ into the optimization dynamics via CCCP descent, enabling non-local and theoretically interpretable control over attention.

We will clarify this motivation and its theoretical basis in the revised paper.

W2. Under performance in terms of FID We thank the reviewer for pointing out the slightly higher FID values in some configurations. Our evaluation strategy is designed to prioritize balanced performance across fidelity, diversity, and consistency, rather than optimizing for FID alone.

As shown by [1], FID-optimal checkpoints often lie far from the Pareto front when jointly considering realism, consistency, and diversity, and in models such as SDXL, optimizing for FID can significantly reduce realism (Section 4.1, Fig. 3). Dhariwal et al. [2] further show that lower guidance scales minimize FID and recall, while higher guidance scales improve perceptual sharpness and structure, measured by IS and precision even though they increase FID (Figure 3).

In line with these findings, we adopted a multi-objective evaluation strategy. To directly address this question, we conducted an additional experiment optimizing hyperparameters for FID. We will include results (presented below) for completeness.

Minor weaknesses Thanks for pointing out this typo, it will be corrected in future versions of the paper.

[1] Consistency-diversity-realism Pareto fronts of conditional image generative models, Astolfi et al., NeurIPS'24.

[2] Diffusion Models Beat {GAN}s on Image Synthesis, Dhariwal et al., NeurIPS'21.

Thank you for the clarification. All my concerns have been resolved after reading the rebuttal.

We thank the reviewer for their thoughtful and encouraging feedback. We are particularily grateful that they appreciate the clarity and organization of the paper, the versatility and simplicity of ERG, and its ability to enhance diverse generative tasks without the need for retraining or auxiliary models.

Below we address the questions raised by the review.

W1. “Performance improvement is relatively minimal, especially in terms of FID”

We acknowledge the reviewer’s observation. Our evaluation setup is designed to optimize overall generation quality, balancing fidelity, diversity, and consistency—not just a single metric like FID. This makes direct comparisons on FID alone potentially misleading.

As shown by [3], FID-optimal checkpoints often lie far from the Pareto front when jointly considering realism, consistency, and diversity, and in models such as SDXL, optimizing for FID can significantly reduce realism (Section 4.1, Fig. 3). Dhariwal et al. [4] further show that lower guidance scales minimize FID and recall, while higher guidance scales improve perceptual sharpness and structure, measured by IS and precision even though they increase FID (Figure 3).

However, to directly address this concern, we conducted an additional experiment where we tuned each method specifically to optimize FID, selecting the best hyperparameters (e.g., guidance scale, ERG scale) for each.

CFG achieves its best FID at a guidance scale of 1.25, while ERG achieves its optimal FID of 4.93 using a guidance scale of 1.25 and an ERG scale of 2.0. This demonstrates that ERG is capable of outperforming prior guidance methods (e.g., SEG, APG) when optimized for FID alone, in addition to providing better trade-offs under broader evaluation criteria as can be seen in the Pareto fronts in Figure 2 of the main paper.

W2. “The evaluation metrics used in this paper are not very standard. Commonly used metrics, such as Inception Score (IS), sFID, Precision, and Recall, are not reported, which somewhat diminishes the persuasiveness of the experimental results.”

We thank the reviewer for their insight on this point. We originally chose to report a subset of metrics that robustly capture the distinct facets of generative performance without overwhelming the reader. Specifically, we believe that density and coverage (D&C) offer more reliable and interpretable fidelity and diversity evaluations than precision and recall (P&R), as detailed in $1$ . While P&R have advanced the separation of fidelity and diversity, they exhibit key limitations:

• They fail to detect matches between identical distributions, often reporting suboptimal values even when real and generated data are sampled from the same distribution ( $1$ - Fig. 3).
• They are vulnerable to outliers, as a single real or fake outlier can inflate the manifold estimation and artificially boost scores ( $1$ - Fig. 3, left).
• They depend on arbitrarily chosen hyperparameters, such as the number of nearest neighbors k, without principled guidance for selection ([1] - sec. 3.1).
• In contrast, D&C address these issues by using a superposition-based manifold estimation that is more robust to data sparsity and outliers, yield analytically tractable behavior under ideal conditions ([1] - sec. 3.3), and support systematic hyperparameter tuning ([1] - sec. 3.4).

Therefore, we opted for D&C as they faithfully reflect the fidelity-diversity trade-offs of generative models in both synthetic and real-world scenarios.

That said, we fully agree on the value of standard benchmarks for comparability. We now include Inception Score (IS), sFID, and Precision & Recall for completeness.

For precision & recall, we observe that similar trends to density and coverage with ERG+CADS achieving the best precision (72.72). ERG on its own achieves a precision of 70.92 compared to 65.71 for CFG. Recall is slightly lower when using ERG alone (41.43 vs 43.95 for CFG), but ERG+APG achieves the best recall of 50.25. For Inception Score, ERG improves over the CFG baseline (43.16 vs 37.25 for CFG) and achieves the best score when combined with APG (53.47). Similarly, ERG+APG results in significant sFID and Inception score improvement, with ERG alone on par with APG.

W3. “Other guidance techniques, such as CFG, can be applied to autoregressive image generation models. Could ERG similarly improve the performance of such models as well?”

We appreciate the reviewer's suggestion. While guidance methods like CFG have been successfully applied to autoregressive image generation, extending ERG to this setting is non-trivial and would require further research. In diffusion and flow-based models, such guidance can be interpreted theoretically due to the iterative nature of the sampling process which operates on the global image representation. In contrast, models like LlaMAgen $2$ sequentially predict the series of $h \times w$ tokens without access to the full image context, no refinement process similar to diffusion/flow sampling occurs. We conducted a preliminary investigation using the LLaMAgen model but did not observe stable improvements with an out-of-the-box implementation of ERG compared to CFG.

This suggests that directly transferring ERG to autoregressive architectures may not be straightforward, potentially due to differences in the way attention and generation unfold across time steps in such models, which renders some parameters in ERG incompatible, such as the I-ERG kickoff threshold which does not have a direct parallel in the autoregressive setting.

Nonetheless, we believe this is a promising direction for future work and are interested in exploring how ERG-like mechanisms could be adapted to autoregressive settings more effectively.

$1$ Reliable fidelity and diversity metrics for generative models. Naeem et al., ICML, 2020.

$2$ Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation, Sun et al., 2024.

$3$ Consistency-diversity-realism Pareto fronts of conditional image generative models, Astolfi et al., NeurIPS, 2024.

$4$ Diffusion Models Beat GANs on Image Synthesis, Dhariwal et al., NeurIPS, 2021.

We thank the reviewer for their thoughtful feedback. We are also grateful for their acknowledgement of the motivation and technical soundness of ERG, the broad experimental work validating the method’s effectiveness across diverse settings and their positive comments on the clarity and organization of the paper.

Below we address the questions raised by the review.

W1. Yes, our method is based on hopfield energy interpretation of the attention so it is only applies to models with attention layers. We believe that this limitation is not major as the vast majority of large-scale state-of-the-art models currently adopt DiT-based architectures (see SD >= 3.0, FLUX, SORA, Bagel, WAN, MovieGen etc.).

Compared to some previous works, our method is less restrictive. For instance, Autoguidance requires access to a weaker model, and SEG/PAG were only developed for UNet models with attention, their practical usefulness for large scale SOTA models (DiT) remained unclear.

W1.1. The reason for focusing on DiT architectures lies in their architecture inducing a more complex design space for methods such as SEG, and because they have become the standard for large-scale generation. However, we agree that providing evaluations on different architectures would strengthen our work. Following the setup in Section F of the appendix, we provide additional (I-)ERG results for SDXL, PixArt-alpha, DiT and EDM2 (autoguidance). SDXL and EDM2 use UNet architectures while PixArt-alpha uses local window attention. For SDXL and Pixart-alpha, we use the opensource version from Hugging Face alongside the EvalGIM evaluation framework. For EDM2 and DiT, we use the evaluation setups from their respective open-source implementations.

W2. We thank the reviewer for this interesting question. We provide two interpretations of ERG based on entropy regularized RL and variational inference.

Let $p$ be the density function for the strong model and $p^\tau$ the one induced by the attention rectification mechanism we introduce in the paper.

I-ERG

We begin with the I-ERG guidance update, which modifies the model’s score estimate as:

$\nabla_x \log p^\tau(x) = \nabla_x \log p(x) + w \cdot \left( \nabla_x \log p(x) - \nabla_x \log p^\tau(x) \right)$

Rewriting the right-hand side, we obtain:

$\nabla_x \log p^\tau(x) = \nabla_x \log p(x) + w \cdot \nabla_x \log \left( \frac{p(x)}{p^\tau(x)} \right)$

Integrating both sides with respect to $x$, we find:

$p^\tau(x) \propto p(x) \cdot \exp \left( w \cdot R(x) \right), \quad \text{where } R(x) = \log \left( \frac{p(x)}{p^\tau(x)} \right)$

This form reveals that I-ERG guidance parallels a posterior distribution that reweights the base model’s density $p(x)$ by an exponential function of the reward signal $R(x)$.

This exponential reweighting aligns with objectives used in entropy-regularized RL or KL control, where the optimal distribution maximizes an expected reward under a KL penalty with respect to a prior:

$\pi^* = \arg\max_{\pi} \mathbb{E}_{x \sim \pi}[R(x)] - \frac{1}{\lambda} \mathrm{KL}(\pi \Vert p)$

Here, $\lambda$ serves as an inverse temperature controlling the sharpness of reward influence. The solution to this optimization problem has the form:

$\pi(x) \propto p(x) \cdot \exp(\lambda R(x))$

This shows that the I-ERG update implicitly implements the maximum entropy policy framework, steering the sampling distribution toward high-reward regions while maintaining proximity to the base model $p(x)$. Equivalently, using Bayes rule, it can be viewed as defining a joint distribution over $x$ under $p(x)$ and the energy induced by reward $R(x)$, (assuming conditional independence).

C-ERG

In the conditional setting, assume we start from a model $p(x|s)$ conditioned on some semantic signal $s$. Let $s^\tau = s + \Delta s$ represent a degraded or weakened version of the conditioning (e.g., via temperature scaling or perturbation).

The C-ERG guidance update is given by:

$\nabla_x \log p^{ERG}(x|s, s^\tau) = \nabla_x \log p(x|s) + w \cdot \left( \nabla_x \log p(x|s) - \nabla_x \log p(x|s^\tau) \right)$

Or, equivalently:

$\nabla_x \log p^{ERG}(x|s, s^\tau) = (1 + w) \cdot \nabla_x \log p(x|s) - w \cdot \nabla_x \log p(x|s^\tau)$

To analyze this further, we perform a first-order Taylor expansion of the degraded score $\nabla_x \log p(x|s^\tau)$ around $s$:

$\nabla_x \log p(x|s^\tau) \approx \nabla_x \log p(x|s) + \frac{\partial^2 \log p(x|s)}{\partial x \, \partial s^\top} \cdot \Delta s$

Substituting this into the update:

$\nabla_x \log p^{ERG}(x|s, s^\tau) \approx \nabla_x \log p(x|s) - w \cdot \left( \frac{\partial^2 \log p(x|s)}{\partial x \, \partial s^\top} \cdot \Delta s \right)$

Thus, C-ERG introduces a Jacobian-vector product correction, nudging the sample along directions where the score is most sensitive to perturbations in the conditioning. This helps reinforce fine-grained semantic detail in generation, even when the conditioning signal is noisy or coarsened.

This Jacobian-vector product has a natural interpretation: it reflects how the image-level score $\nabla_x \log p(x|s)$ changes when the conditioning $s$ is slightly perturbed. This resembles classifier guidance, where one steers the generation based on $\nabla_x \log p(y|x)$ — the gradient of a classifier with respect to the image.

Here, the conditioning is a continuous vector $s$, and C-ERG guides generation using:

$\frac{\partial^2 \log p(x|s)}{\partial x \, \partial s^\top} \cdot \Delta s$

which is the direction in image space most sensitive to changes in the semantics of $s$. This amounts to amplifying features that are especially responsive to semantic precision.

W3. The distinction between our method and Smoothed Energy Guidance (SEG) can be better clarified. While SEG also modifies attention energies in diffusion models for guidance—both in conditional and unconditional settings—the key differences lie in the mechanism of rectification and its broader applicability. Below, we summarize the main novelties of our method over SEG:

• ERG modifies attention energies not only in the denoiser but also in the text encoder, allowing finer control over consistency and diversity.
• Our ablations demonstrate that temperature rescaling (ERG) consistently outperforms the Gaussian blurring approach used in SEG.
• SEG shows an unfavorable tradeoff between quality and consistency when applied to DiT architectures. We address this by introducing a kickoff threshold, which controls steerability while preserving high quality.
• ERG is compatible with DiT-based models (and UNet models with attention) we provide clear implementation guidelines for models of varying sizes. SEG paper provides a limited scope in comparison.
• Unlike prior works that report only limited metrics such as FID and IS, we conduct extensive evaluations across multiple generation settings. We thoroughly examine trade-offs between quality, diversity, and consistency, and offer practical insights for applying ERG across architectures.

Q1&2. Retraining models is not required for our method. We chose to use our internal model to adhere to internal guidelines and, more importantly, to ensure tighter control over key variables that often differ across open-source models—such as training data, architecture, image resolution, task type, and the specifics of the diffusion/flow scheduler and sampler. By using an internal model, we minimize confounding factors and enable a fairer comparison of guidance methods. To address the reviewer’s concern, we have also added a new set of experiments using publicly available implementations of popular open-source models (see the tables above). Thus, we now cover SD 3.0, SD 3.5 (Table 8 of the appendix), SDXL, PixArt-alpha, DiT, and EDM2 (tables above).

Q3. We believe the requested ablation is presented in Table 6 & Figure 17, and discussed in Section C.3 of the appendix. We compare I-ERG with a kickoff threshold $\kappa = 0$ with SEG, hence the only difference under this setting is the attention rectification mechanism (Gaussian blurring vs entropy rectification). Results reported showcase better detail modeling qualitatively and better scores for FID, density, coverage and CLIPScore with entropy rectification than with Gaussian blurring. If the reviewer has more specific questions about these experiments, we would be happy to address them.

Q4. We indirectly address this in section C.2 of the appendix. By analyzing the orthogonal and parallel components of the guidance term, we find ERG to provide a stronger orthogonal signal wrt CFG, enabling better quality at lower guidance strengths, which reduces oversaturation issues. We will better clarify this link in the paper.

S1. Thank you for pointing out this typo. It will be corrected in future versions of the paper.

[1] Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control, Domingo-Enrich et al., ICLR 2025.