Thank you for your valuable feedback and insightful comments on our work. Please see below for our point-by-point responses to your comments.

Weakness 1

Reference [29] is a highly related method which seeks multiple perception-distortion tradeoff points with test-time iterations. Similarly to the authors work, it is a (progressive) mode-seeking strategy. The authors do not discuss similarities and differences of their work with respect to [29]. I think this point should be significantly expanded. Moreover, since [29] essentially addresses the same problem, the authors should include experimental comparisons with it. At the current stage, it is unclear whether the authors' paper brings significant novelty or experimental improvements with respect to [29].

We thank the reviewer for highlighting this related line of work. We now provide a detailed clarification of the conceptual and methodological distinctions between KDS and InDI [29], and will incorporate this discussion more prominently in the revised manuscript.

1. Different Motivations : We would like to clarify the key difference in motivation: InDI is not a posterior mode-seeking algorithm. It performs a greedy, step-wise optimization on the next latent state, which does not guarantee convergence to overall posterior. As a demonstration, Appendix B in the InDI paper [29] shows that the InDI estimate deviates from the MAP estimate for a simple Gaussian signal. In contrast, KDS directly seeks a consensus mode among the ensemble's predictions of the final clean output, a more direct objective for improving image fidelity.
2. Different Methods (Sampler vs. Training/Testing pipeline ): KDS is a training-free, plug-and-play module designed for broad applicability, enhancing any pre-trained conditional diffusion model without retraining. This is distinct from InDI’s integrated training and sampling pipeline.
3. Direct Performance Comparison is Misleading: The metrics reported in [29] are from a different dataset setting. InDI is expected to yield metrics similar to its baseline Conditional Diffusion Model, but with faster execution (as discussed in the InDI paper, Figure 12). Therefore, its metrics should closely align with our LDM-SR + DDIM baseline.

Weakness 2 And Question 2

An expanded discussion on computational complexity (and runtime experiments) would be useful to better understand the practical cost of the method How complex is the proposed method in terms of runtime compared to the baseline without KDS?

Thank you for your question. We included a runtime comparison in Table 4 of our initial manuscript. As detailed in Table 4, our KDS method with N=10 particles takes 28.4 seconds. This runtime is nearly identical to a standard 10-particle ensemble without KDS (which takes 27.3 seconds for a Best-of-N selection). This marginal (~4%) increase in runtime is a highly favorable tradeoff for the significant improvements KDS provides in both fidelity and perceptual quality. We will make sure to clearly reference this study in the revised manuscript for better visibility.

Question 1

The experimental results seem to suggest that both distortion and perception metrics are improved by KDS. This is somewhat counterintuitive as it goes against the perception-distortion tradeoff principle, and does not happen in other works like [29] where improving one degrades the other. I wonder if the authors have an explanation for this? Could it be that the chosen baselines are rather weak? (e.g. [29] reports significantly better PSNR and FID numbers on DIV2K)

Thanks for the question. Please find our responses below.

Adherence to the P-D Tradeoff: KDS does not violate the perception-distortion tradeoff principle but rather operates on a superior tradeoff curve. As Figure 6 in our submission demonstrates, adjusting the KDS bandwidth hyperparameter (h) allows for a trade-off between distortion and perception along the improved frontier. A larger 'h' value enhances distortion performance at the cost of perception, and vice versa.

Explanation for Simultaneous Improvement: Standard diffusion samplers often introduce artifacts (e.g., unnatural textures, misplaced details). These artifacts degrade both perceptual quality (increasing LPIPS) and fidelity to the ground truth (lowering PSNR). KDS leverages the "collective wisdom" of an ensemble to identify and suppress these spurious features. By producing a cleaner, more robust restoration that is closer to the true data manifold, it simultaneously reduces distortion and improves perceptual quality.

Strength of Baselines and Comparison to [29]: The baselines we use (e.g., DiffBIR (ECCV 2024), SeeSR (CVPR 2024)) are strong and SOTA for these tasks. The superior metrics reported in [29] are not directly comparable due to different training and testing protocols. As stated in our previous response, InDI [29]'s goal is to match its own baseline's quality with faster sampling. A fair comparison would therefore be between InDI's results and our baseline LDM-SR results, over which KDS provides a significant and consistent improvement.

Question 3

It seems that 1x1 is the best patch size. Is this due to the curse of dimensionality for even moderately larger sizes or is it due to the low number of chosen particles? Would larger patches with a lot more particles work better (without caring for complexity)?

We appreciate this insightful question regarding the tradeoffs between patch size and ensemble size. Our current default choice of 1×1 patches with N=10 aims for a practical balance for most use cases, but we agree that richer patch interactions (with increased compute budget) are worth pursuing.

According to your suggestion, we conducted a new ablation study, increasing the number of particles (10 -> 40) for larger patches (1x1 -> 4x4), and observed an improvement in performance. Therefore, increasing the number of particles and using larger patch size can enhance performance by mitigating the curse of dimensionality.

We will include this observation and study in the revised appendix.

Table: New Ablation Study on Patch-size

We sincerely thank the reviewer for the thoughtful and constructive feedback, which has helped us better articulate the motivation, implementation, and novelty of our approach. Please see below for our point-by-point responses to your comments.

Weakness 1:

My major complaint is twofold. First, as primary motivation, the authors aim to "steer particles towards the modes" (line 42) and "steer diffusion sampling towards high-density" (line 58). To me, an obvious solution to this problem is to utilize the already estimated score function. I do not understand why the authors consider the gradient of KDE instead of the learned score. It is not mentioned in the paper why this would be a better choice. Second, these two problems, i.e., "mode-seeking" and "log-density control", are already solved for diffusion models. Specifically: [1] derives a principled ODE that tracks the modes in diffusion model sampling. Even Figure 6 in [1] looks very similar to Figure 2 in this submission. [2] provides a principled ODE for sampling in diffusion models and controlling the log-density of the generated samples.

We sincerely thank the reviewer for their thoughtful and constructive feedback. In particular, we appreciate the opportunity to clarify the motivation behind Kernel Density Steering (KDS) and to address the simplifications in our introductory equations.

Motivation Behind KDS

The central motivation for KDS, rather than directly relying on the learned score function, is that score-based guidance explores the learned posterior in the noisy latent space, whereas **KDS explicitly refines this posterior via consistency across particles in the clean latent space. This refinement enables the method to be more robust to spurious modes and better aligned with the final restoration objective—namely, accurate, high-fidelity image reconstruction.

Clarifying Equations (1) and (2)

We acknowledge that the simplified Equations (1) and (2) in our introduction did not fully convey the algorithm’s core mechanics. For clarity, we will revise them in the manuscript to match the detailed and correct version provided in Algorithm 2.

Corrected Equation 1:

Corrected Equation 2:

$\text{Old:} \quad d\mathbf{z}_t^{(i)} = \left[ f(t)\mathbf{z}_t^{(i)} - \frac{1}{2} g(t)^2 \left((1-\delta_t)\mathbf{s}_t^{(i)} + \delta_t \tilde{\mathbf{z}}_t^{(i)} \right) \right] dt$, where $\mathbf{s}_t = -\frac{1}{\sqrt{1-\bar{\alpha} _t}}{\epsilon} _{\theta}(z_t, t)$, $\tilde{\mathbf{z}}_t$ is get from old equation (1).

$\text{New:} \quad d\mathbf{z}_t^{(i)} = \left[ f(t)\mathbf{z}_t^{(i)} - \frac{1}{2} g(t)^2 \left((1-\delta_t)\mathbf{s}_t^{(i)} + \delta_t \tilde{\mathbf{s}}_t^{(i)} \right) \right] dt$, where $\mathbf{s}_t = -\frac{1}{\sqrt{1-\bar{\alpha} _t}}{\epsilon} _{\theta}(z_t, t), \tilde{\mathbf{s}}_t^{(i)} = \frac{z_t^{(i)} - \sqrt{\bar{\alpha}_t} \tilde{z}_0^{(i)}}{\sigma_t^2}$.

Key Conceptual Distinctions from [1, 2]

We believe the core conceptual gap lies in what mode the method attempts to reach. The methods in [1, 2] steer toward high-density regions of the posterior over noisy latents. This is not equivalent to steering toward modes of the clean image distribution—especially in restoration tasks, where fidelity to the ground truth is paramount. KDS addresses this mismatch by aligning particles in the clean space and propagating that alignment backward.

We respectfully highlight several conceptual distinctions between our method and [1, 2], which we now cite and discuss more thoroughly in the revised manuscript:

1. Clean-space consensus vs. noisy-space score guidance

   • Score-based methods ([1, 2]) operate in the noisy latent space, guiding each particle using a learned gradient $\nabla \log p(\mathbf{z}_t)$.
   • KDS, in contrast, aligns samples by computing a nonparametric consensus in the clean latent space ( $\hat{\mathbf{z}}_0$ ) and steers particles toward that. This yields guidance that is directly tied to high-fidelity reconstruction rather than abstract density estimation.

2. Multi-particle interaction vs. independent sampling

   • [1, 2] use advanced solvers but apply them independently per particle.
   • KDS introduces explicit inter-particle coupling: each particle’s update depends on the collective configuration of the entire ensemble. This interaction is essential for achieving robust consensus.

3. Restoration-focused vs. generic generation objectives

   • Prior works are designed for general generation, where seeking high-density regions often favors over-smoothed or generic-looking samples.
   • KDS is purpose-built for restoration, aiming to suppress non-consensus artifacts and hallucinations by aligning the ensemble around shared, plausible details.

4. Empirical evidence on restoration benchmarks

   • As requested, we include a direct empirical comparison with HDS[1] and DGS [2] on RealSR. Our results confirm that KDS yields consistently sharper and more faithful reconstructions. Please refer to our answer to your Question 2 for more details.

Summary

In short, KDS seeks the mode—not just a dense region in latent space, but the one consistent with high-quality restoration in the clean image domain. This distinction clarifies why KDS diverges from the score-based trajectory of [1, 2].

We will revise the manuscript accordingly to:

• Incorporate the corrected equations
• Cite and discuss [1] and [2] more thoroughly
• Emphasize the clean-space mode-seeking philosophy behind KDS

We are grateful again to the reviewer for helping us sharpen the presentation of our core ideas.

Weakness 2:

Statistical significance. In the submission checklist, the authors claim that they "reported error bars suitably and correctly defined". I do not think this is true. I would even claim the opposite. None of the quantitative results seem to have any error bars reported...

We thank the reviewer for highlighting the absence of error bars and the need for more robust statistical analysis. The reviewer is correct that our initial presentation of worst-case performance was insufficient for demonstrating statistical significance. We addressed this by adding standard errors for all quantitative metrics. We also conducted significance tests (e.g., paired t-tests) against baselines. Due to space constraints, we'll present full statistical analysis in the paper, but for the rebuttal, we've included it for one table that also incorporates a reviewer-suggested baseline.

Question 1:

Why do authors use the KDE instead of the already estimated score function?...

Thank you for this excellent question. We clarify now that KDS is not an alternative to the learned score—it is complementary. Each particle is still guided by the model score (∇ log p(z_t)), but this is augmented with a secondary signal derived from the particle ensemble via the mean-shift update in clean latent space. This hybrid guidance improves robustness: the learned score offers prior knowledge, while KDS enforces internal consistency. Together, they balance generative expressiveness with stability. As shown in Figure 4 and Table 2, KDS notably improves worst-case particle quality—something model-only scores often fail to address.

Question 2:

How does the method compare to using the known algorithms for mode-seeking [1] or log-density control [2]?

Thanks for bringing these two interesting work into our attention.

We have conducted new experiments on the RealSR dataset to compare KDS with HDS [1] and DGS [2]. While HDS achieves strong PSNR, it tends to oversmooth and reduce perceptual quality, as reflected in higher LPIPS and lower CLIPIQA scores. DGS partially addresses this tradeoff but still underperforms compared to KDS.

In contrast, KDS achieves the best overall balance, with the best LPIPS, SSIM, CLIPIQA. This confirms that KDS avoids artifacts and preserves detail by steering particles toward clean-space consensus.

Super-Resolution on Real-World Datasets

Statistically significant differences ($P < 0.05$) compared with DDIM are marked with a ★.

Question 3:

Are the quantitative improvements over the baseline statistically significant?

Thanks for the question. We have conducted paired t-tests on our main results to verify that the improvements are statistically significant. We have also added standard errors to the reported metrics. For the key comparisons in our paper, the improvements are indeed significant. For instance, when applying KDS to the DDIM sampler on the DIV2K dataset (Table 1), the gains in PSNR, SSIM, LPIPS and CLIPIQA are all statistically significant with p-values < 0.05.

Thank you for your valuable feedback and insightful comments on our work. Please see below for our point-by-point responses to your comments.

Weakness 1

Some aspects of the proposed methodology are not clearly explained and lack theoretical justification. (a) Why do the authors choose kernel density estimation over other distance metrics, such as a vanilla $\ell_2$ loss? (b) The abstract states that "the patches are steered towards shared, higher-density regions." A clearer explanation is needed as to why a higher-density region is beneficial for sample quality.

Thank you for your comments. Please find our responses below.

1. Why KDE over L2? Our choice of Kernel Density Estimation is a direct response to the well-known limitations of L2-based objectives.When minimizing the L2 distance directly across all particles, the optimal solution is their arithmetic mean. However, this approach often leads to an out-of-domain reconstruction. As confirmed in our own new ablation study, guidance based on L2 minimizes distortion at the expense of producing highly blurry, and perceptually low quality reconstructions. In contrast, KDS uses the mean-shift vector—the gradient of the log-KDE—to perform mode-seeking. This steers the solution towards high-density regions representing strong ensemble consensus, where predictions are sharp, detailed, and perceptually realistic because they correspond to a probable, mode-aligned outcome, not a simple average.

2. Why are high-density regions better? High-density regions are beneficial precisely because they represent "collective wisdom". Diffusion models can produce spurious artifacts, but these artifacts are typically random and idiosyncratic to a single sample. They therefore reside in low-density regions of the ensemble's distribution. By steering all particles towards shared high-density modes, KDS acts as a collaborative filter, effectively suppressing these random artifacts and enforcing a consistent, robust restoration. The empirical evidence in Figure 4, which shows KDS dramatically improves the worst-case particle performance, strongly supports this conclusion.

Table: KDS with Different Guidance on RealSR Dataset

Weakness 2

The quantitative improvement appears marginal, e.g., the DISTS metric in Table 1.

We respectfully disagree that the improvements are marginal and would like to provide additional context to clarify their significance.

While the improvement on any single metric like DISTS may appear modest, KDS achieves consistent gains across a diverse set of metrics. For instance, in Table 1 with the DiffBIR backbone, KDS improves PSNR by 1.1 dB and LPIPS by 0.031. These are substantial gains for a training-free sampling method and showcase a clear improvement in the perception-distortion tradeoff.

KDS is highly effective at reducing certain types of artifacts, but the DISTS metric is often less sensitive to these improvements, as demonstrated in other literature [1]. More importantly, our extensive qualitative results in the appendix (e.g., Figures 3, 5, 7) visually demonstrate non-trivial improvements in sharpness and fidelity that are not fully captured by any single metric.

To formalize this, we will incorporate statistical significance tests into our revised tables, as suggested by Reviewer nw3F. This will clearly demonstrate the statistical validity of these improvements. Please refer to the table provided in our response to Reviewer nw3F.

[1]: Park SH, Moon YS, Cho NI. Perception-oriented single image super-resolution using optimal objective estimation. (CVPR 2023)

Question 1

There are several other kernel functions available, such as the Laplacian kernel. Is the Gaussian kernel superior to other kernels in this context? How do different kernels affect the final performance?

We appreciate this excellent question. While our paper used a Gaussian kernel, the KDS framework is flexible and not dependent on this specific choice. As per your suggestion, we have conducted a new ablation study comparing the Gaussian kernel with the Laplacian kernel.

Our results show that while both kernels provide a significant improvement over the baseline, the Gaussian kernel achieved slightly better performance in our experiments. For example, on the DIV2K dataset, the Gaussian kernel resulted in [e.g., 22.37 PSNR], while the Laplacian kernel achieved [e.g., 22.25 PSNR].

This confirms that KDS is robust to different kernel choices, though the smooth gradients of the Gaussian kernel appear to be marginally more effective in this context. We will add this new ablation study and discussion to the appendix.

Table: KDS with Different Kernels on RealSR Dataset

Question 2

What is the motivation behind using a patch-wise mechanism? Why don't the authors directly use the whole image for Kernel Density Steering (KDS)? I am concerned that a patch-wise mechanism might introduce seaming artifacts into the image.

Motivation (Overcoming the "Curse of Dimensionality"): As we detail in Section 3.1, performing Kernel Density Estimation on the entire high-dimensional image latent is statistically infeasible with a practical number of particles (e.g., N=10). A prohibitively large ensemble would be required to get a meaningful density estimate. By operating on low-dimensional, non-overlapping patches, we can accurately estimate local density modes even with a small $N$, effectively overcoming this curse of dimensionality.

How Seaming Artifacts are Avoided: To address concerns about seaming artifacts, we emphasize that our patch-wise refinements are not applied independently. Instead, we integrate them into a global prediction-update loop at every sampling step:

1. Global Prediction: The diffusion network first predicts the entire clean image, establishing global coherence.
2. Local Refinement: KDS steers patches of this prediction toward a higher-fidelity ensemble consensus.
3. Unified Global Update: Crucially, the guidance from all patches is combined to compute a single update for the entire image latent. This refined global latent is then passed to the next sampling step. Because the network always operates on the whole image, which has been informed by the local KDS refinements, it naturally enforces consistency across all patch boundaries, eliminating seams.

Our extensive visual results confirm that this approach does not introduce such artifacts. Thanks again for this insightful question, we will include this discussion in the revised manuscript.

Thanks again for your feedback! We've incorporated your suggestions and have new results from our experiments. We hope these changes successfully address your concerns. Please let us know if you have any further questions or concerns. Thank you.

Thank you for your valuable feedback and insightful comments on our work. Please see below for our point-by-point responses to your comments.

Weakness 1

The computational cost scales linearly with the ensemble size and there is no adaptive reduction mechanism.

The computational cost of KDS, like other ensemble-based methods [46, 47] discussed in our submission, scales linearly with the number of particles, $N$. We consider the ensemble size N an intuitive and tunable hyperparameter that offers users fine-grained control over the balance between computational cost and quality. As we demonstrate in Figure 6, this trade-off is predictable and graceful: practitioners can achieve significant gains with even a moderate ensemble size (e.g., $N=10$) or invest more compute for consistent further improvements, tuning the method precisely to their budget.

We are grateful for the reviewer's idea of designing an adaptive reduction mechanism. In light of the extensive experiments on choosing the hyper-parameters in the submission, this idea is an excellent suggestion for future work. We have incorporated a discussion of it into the limitations and future work section of our revised manuscript. For instance, future work could explore strategies such as pruning particles that are far from the consensus or reducing N in later, less stochastic sampling stages.

Weakness 2

The kernel bandwidth selection h is manually tuned; the author lacks an automated strategy.

We appreciate this valuable suggestion. We agree that automatic bandwidth selection is a critical and well-studied problem in kernel density estimation (KDE), with a rich body of existing literature dedicated to it.

Following your suggestion, we have run a new ablation using a standard automated strategy (the median heuristic). The results confirm our method is effective without tedious manual tuning.

Our new results show this automated approach is highly effective. For example, on the DIV2K dataset, the automated strategy achieved [e.g., 24.24 PSNR / 0.338 LPIPS], which is directly comparable to the performance of our manually tuned bandwidth (24.39 PSNR / 0.339 LPIPS). Both automated and manual approaches significantly outperform the baseline.

This confirms that KDS is robust and flexible enough to incorporate standard hyperparameter automation techniques, enhancing its practical applicability. We will add the full ablation study to the appendix.

Table: New Experiment – Automatic Bandwidth Selection

Weakness 3 and Question 2

Have you considered extending KDS to other inverse problems (e.g., denoising, deblurring, MRI/CT) or temporal tasks such as video restoration?

This is an excellent insight—thank you for raising it. We believe KDS is a general framework that can be readily extended to a wide range of inverse problems, which is one of its key strengths.

The reason for its broad applicability is that KDS is a training-free, plug-and-play inference framework. It can be applied to enhance the output of any pre-trained conditional diffusion model, regardless of the specific task for which it was trained.

• For other inverse problems (e.g., deblurring, denoising, medical imaging), the extension is direct. As long as a conditional diffusion model exists for the task, KDS can be seamlessly integrated at inference time to improve fidelity and robustness.
• For temporal tasks like video restoration, KDS offers a particularly exciting path forward. Our patch-wise mechanism could be extended to 3D spatio-temporal patches. This would enable KDS to seek consensus not only among an ensemble of samples for a single frame but also across adjacent frames, providing a powerful mechanism for improving temporal consistency.

We appreciate this forward-looking suggestion and have incorporated a discussion in our revised manuscript.

Question 1

Is it possible to develop an adaptive strategy for choosing ensemble size N or patch dimensions based on noise level or sample distribution? Is it possible to integrate automatic bandwidth selection methods (e.g., plug-in estimators) to alleviate manual tuning of h?

Thanks for the insightful question. Please find the detailed answer to your question below.

Adaptive Ensemble (N) and Patch Dimensions: We agree that an adaptive strategy would be a valuable extension. We see this as a compelling direction for future work. Developing methods to dynamically adjust the ensemble size $N$ or patch dimensions based on consensus metrics during the sampling process could significantly enhance efficiency. We will add this to our future work discussion in the revised paper.

Automatic Bandwidth Selection (h): We agree that automating this is important and have already conducted a new experiment to address this point. We implemented an automated bandwidth selection strategy using a standard median heuristic. The results have been shown in the response to the second weakness point.

Question 3

Could you compare or combine KDS with Best-of-N selection using non-reference metrics (e.g., NIQE, CLIP scores) to highlight relative advantages? Thanks for asking this question.

We provided a direct comparison in Appendix A.1.6 (Table 8 and Figure 8) of the submitted paper, which highlights the clear advantages of KDS over a Best-of-N (BoN) selection strategy. The revised manuscript will ensure that this study is clearly referenced in the main manuscript for visibility.

1. Quantitative Comparison: As shown in Table 9, KDS significantly outperforms BoN (CLIPIQA) across both fidelity (PSNR, SSIM) and perceptual metrics (LPIPS): While BoN achieves a higher CLIPIQA score (0.774), it degrades fidelity and perceptual quality—highlighting the risk of optimizing for a single, potentially misaligned metric. In contrast, KDS steers generation via internal consensus, producing reconstructions that are sharper, more consistent, and less prone to artifacts. Please find the detailed answer to your question below.

2. New Ablation: KDS + BoN: Following the reviewer’s suggestion, we conducted a new ablation combining KDS with BoN. The hybrid approach (+KDS+BoN) achieves a more balanced result:

   • Improves PSNR by +0.68, SSIM by +0.044, and reduces LPIPS and DISTS compared to BoN alone.
   • Retains a strong CLIPIQA score (0.770), close to BoN’s best.

   This shows that KDS enhances the quality of samples, making BoN selection more reliable. Without KDS, BoN often selects from a noisy ensemble, favoring samples that "fool" the metric. With KDS, the candidates are cleaner and more faithful, leading to a better perceptual–distortion trade-off.

3. Summary: BoN can boost specific metrics, but lacks generalization. KDS provides a principled, task-aligned alternative by enforcing ensemble consistency in the clean space. It improves performance across the board—and even strengthens BoN when combined. Thanks again for review's insightful question, we will highlight this more clearly in the revised manuscript.

Table: Comparison of BoN and BoN + KDS Methods

Thank you for your detailed review and valuable feedback. We hope our rebuttal has addressed your concerns and clarified the points that led to your initial borderline assessment. We believe our work has been strengthened through this discussion, and we trust that the new information will lead you to reconsider your final score. Please let us know if there are any remaining questions we can answer. Thank you!