SD-KDE: Score-Debiased Kernel Density Estimation

We would like to thank reviewer 4YAs for their thoughtful review, which has made our paper better.

W1 (presentation of theorem 1): Thank you for these suggestions, we will update as you suggested to make the presentation more clear and concise.

W2 (corollary 1): The main intention of corollary 1 is to give some intuition on how the bias decompose when we only have an estimate for the score. In the two terms one is proportional to the gradient of p, which will be small near modes, and the other is proportional to p, and will be small in the tails of the distribution. Taken together, we can get practical intuition that the bias may concentrate more on moderate density regions rather than extreme trail or modes. We thank the reviewer for pointing out that this is not central to the paper, and we will hence move it to the appendix for the camera ready.

W3 (MNIST figure): In our revision, we will quantitative evaluations to better support the MNIST results. Specifically, we use Classifier confidence: The average top-1 confidence, \frac{1}{N} \sum_{i=1}^{N} \max_{k} , P(y = k \mid \text{image}_i), from a pretrained MNIST classifier on generated samples, which serves as a proxy for sample realism. This is more clearly showing the advantage of our method.

W4 (convergent taylor series condition theorem 1): Thank you for pointing this out, we have updated the script with this condition.

Q1 (same vs different points SD-KDE): In practical settings, often only one dataset of points is provided, hence in our experiments with using the KDE to estimate the Score for SD-KDE, we estimated the score from the same set of datapoints as was used for SD-KDE. In the updated manuscript we will include a detailed analysis of the tradeoff between using independent vs overlapping samples for the estimates.

Minor comments: Thank you for pointing out the formatting errors, indeed, N(0,σ) should be N(0,σ^2), we will use Theorem throughout, and will update the legend in figure 2 to use σ instead of std for the camera ready.

We thank the reviewer cUSv for their review.

We would like to point out that SD-KDE can be applied effectively without invoking diffusion models, in Figure 5 in the paper, SD-KDE is applied with a score estimate from the Silverman KDE.

W1 (lack of novelty/motivation): As described in our paper, kernel density estimation is an important modeling technique in a variety of fields, such as anomaly detection, clustering, data visualization, and dynamical systems. We provide

1. A theoretical results for better asymptotic convergence
2. A simple algorithm to compute the SD-KDE
3. Practical empirical analysis showing the improved performance of the method

Based on this, we hope that the reviewer will reconsider the low scores given to the paper.

We would like to thank reviewer f5yx for their thoughtful review, which has made our paper better.

W1(score definition): We thank the reviewer for suggesting to allocate more space early on in the paper to formally define the score, and some key properties and applications. We will update this for the camera ready paper.

W2(informal steps in proof): The error is h^4 and in a few line after becomes h^8 because h^4 is the bias, and h^8 is the mean square error contributed from the bias (so it is (h^4)^2 = h^8).

I understand that the vanishing of the h^3 term is vague but to expand all the term is actually quite laborious, though possible. We can also do that, but it does not provide much intellectual benefit for the reader. We will include the expansion in the appendix of the camera ready paper.

Q1(stronger bound without assuming score estimate): The analysis of KDE (as well as SD-KDE) relies on the separability of MSE into bias term and variance term. The bias term (before squared) is linear, making it tractable. If we want to include the error from score, the error will go in to the bias term in a non-linear manner (before expectation), so it will become intractable.

We would like to thank reviewer for their thoughtful review, which has made our paper better.

W1(Theoretical results require exact score): We believe that it may be possible to extend the analysis in the current work to situations where we don’t know the exact score, this will be the topic of follow up research. W2(Visual results on MNIST): In our revision, we have added quantitative evaluations to better support the MNIST results. Specifically, we use Classifier confidence: The average top-1 confidence, \frac{1}{N} \sum_{i=1}^{N} \max_{k} , P(y = k \mid \text{image}_i), from a pretrained MNIST classifier on generated samples, which serves as a proxy for sample realism. This is more clearly showing the advantage of our method. W3(Tuning of step size and band width): Once the band-width is set, there is a fixed formula for the step size based on the band width. We observed that using a similar constant on the band width term as in the Sliverman method works well without being very sensitive to the exact constant value. We will perform an in-depth analysis in the camera ready version of the paper. W4(comparisons with other KDE methods): We want to thank the reviewer for this valuable suggestion, we will compare the SD-KDE method with the more recent methods for the camera ready paper.

Q1(high dimensional setting): Thank you for the suggestion, we have included 10 dimensional gaussian mixture results for the camera ready paper. Q2(efficiency comparison with normalizing flow and autoregressive modeling): Compared with these neural approaches, SD-KDE offers a much simpler fitting scheme, where each point is moved in the direction of the score and then applying standard KDE.

We would like to thank reviewer ftbi for their thoughtful review, which has made our paper better.

“Standard KDE is super-fast; what is the empirical scaling of the method?” SD-KDE inherits the runtime profile of standard KDE. The only extra work is evaluating a score nnn times (once per query/sample). In many common settings this overhead is negligible—either because the score has already been computed or cost is low compared to standard KDE, or because the score is estimated directly from the KDE—so the effective empirical scaling is the same as standard KDE. Importantly, SD-KDE is independent of any particular score–estimation technique; the wall-clock time is that of standard KDE plus the cost of the practitioner’s chosen score oracle. To make this concrete, we will include in the revision: (i) a scaling plot comparing wall-clock time vs. number of queries for standard KDE and SD-KDE across our datasets, and (ii) a minimal timing script so reviewers can reproduce the results. These additions transparently show that SD-KDE’s runtime matches standard KDE up to the n-fold score evaluations described above.

“Also, learning a score requires a fairly large number of points, can you experiment to see what is the behaviour of the learnt density as you vary this parameter?” On “learning a score requires many points” and sensitivity to data size: thank you for raising this. SD-KDE does not necessarily require a learned score: in some of our experiments the score was taken directly from the KDE (via its gradient), so the method could work with exactly the same data requirements as standard KDE. The score oracle is modular; if one chooses to learn a score, that is optional and orthogonal to SD-KDE itself. That said, we agree that it is useful to see sensitivity when a learned score is used. In the revision, we will add an ablation that varies the number of training points for the score estimator while keeping the KDE fixed. Concretely, we subsample the score-training set and fit a standard score estimator on each subset, and report SD-KDE quality versus (i) pure KDE and (ii) SD-KDE with the KDE-derived score. We typically observe that performance improves smoothly with more score-training data and plateaus once score variance is below the KDE’s own smoothing error; importantly, even at small training sizes SD-KDE remains competitive because the KDE already provides a usable score. We will include the sensitivity plot and a minimal script for reproducibility in the revision.

“Finally, I am left somewhat uncertain as to what is the advantage of this approach to directly using a generative model such as diffusion, particularly since the score needs to be learnt training a diffusion model.” We appreciate the question. SD-KDE does not require training a diffusion model: the method works out-of-the-box with a score obtained directly from the KDE (via its gradient), so its data and compute requirements match standard KDE. When a learned score is available, SD-KDE offers a different value proposition from training a full generative model: it converts any score oracle–analytic, pretrained, or lightweight–into a normalized, likelihood-evaluable density with the calibration and interpretability of KDE. In contrast, diffusion models are optimized for sample quality and typically do not provide tractable likelihoods without additional, costly machinery. Crucially, our theory shows that density error is controlled by score error under the stated metric and that this error is still controlled under a score estimate. Hence an approximate or mismatched score can still improve over plain KDE when it is “good enough”–and SD-KDE reduces to standard KDE if not. This modularity is practically useful: (i) many workflows already produce a score as a byproduct (e.g., from a previously trained model), which SD-KDE can reuse with no retraining; (ii) in domains like tabular or scientific data, training a high-capacity diffusion model is often unnecessary, while SD-KDE yields gains at the cost and sample size of KDE. A practical advantage of SD-KDE is that it can reuse off-the-shelf scores. For instance, a score pre-trained on generic natural images can serve as a useful prior when our target dataset consists only of dog images. While the dog-specific score will differ, natural images share strong regularities—edges, textures, and sparse multi-scale structure (e.g., curvelet-like representations in the sense of Donoho) and broader manifold structure—so the generic score is a reasonable approximation. Under our theory, density error tracks score error; thus an approximate but “good enough” score can yield clear gains over plain KDE, and if it isn’t good enough SD-KDE defaults to standard KDE. We hope this clarifies that SD-KDE is complementary to diffusion models: it leverages a score if one is available and delivers a calibrated, normalized density with competitive accuracy.

“Theoretical results require exact knowledge of the score, the sensitivity to errors in score estimation is unknown.” Thank you for the opportunity to clarify this point. While Theorem 1 is stated for an oracle score, Corollary 1 directly addresses the estimated-score case and quantifies sensitivity. Under the same bandwidth and step size as Theorem 1, the corollary gives an explicit bias expression whose leading term is proportional to the size of the score error (and its first spatial derivative), with higher-order terms suppressed. In plain language: the bias grows linearly with the score error, and the added MISE term grows with the square of that error. Thus SD-KDE is stable to misspecification, and approaches the oracle behavior as the score improves.