ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model

We thank the reviewer for their thorough feedback and valuable suggestions. We have addressed the reviewer’s concerns below:

Weaknesses:

1." Wrong proof of Equation (12): This equation plays a key role for exact likelihood........"

We sincerely thank the reviewer for pointing this out and bringing it to our attention. We now have an updated proof-sketch stated below, which will also be included in more detail in the revised version:

Setup. Fix $x\ge0,\ \gamma>0,\ \lambda=\gamma x$. With $Z_\gamma\mid X=x\sim\mathrm{Pois}(\lambda)$ define

$$R_x(\gamma)=\sum_{z\ge0} p_\gamma(z|x)\ \log\frac{p_\gamma(z\mid x)}{p_\gamma(z)} .$$

We can split this using the product-rule for derivatives:

$$\frac{\mathrm d}{\mathrm d\gamma} R_x = T_1 + T_2,$$

where

$$T_1 = \sum_{z}\partial_\gamma p_\gamma(z|x) \log \frac{p_\gamma(z|x)}{p_\gamma(z)},\text{ } T_2 = \sum_{z}p_\gamma(z|x)\ \bigl(\partial_\gamma\log p_\gamma(z|x)-\partial_\gamma\log p_\gamma(z)\bigr).$$

$\partial_\gamma\log p_\gamma(z|x)=z/\gamma-x$.

Taking the required conditional expectation drives this term to zero.

For the marginal, we have:

$$\partial_\gamma\log p_\gamma(z)= \mathbb{E}\left[-X+\frac{Z_\gamma}{\gamma}|\Big| Z_\gamma=z\right].$$

Hence:

$T_2=\mathbb{E}\bigl[\mathbb{E}[X|Z_\gamma]-Z_\gamma/\gamma\bigr]$

Now, let $r(z)=\log\frac{p_\gamma(z\mid x)}{p_\gamma(z)}$. We use the Poisson-Stein identity:

$$\mathbb{E}\bigl[(Z_\gamma-\lambda)h(Z_\gamma)\bigr] =\lambda\mathbb{E}\bigl[h(Z_\gamma+1)-h(Z_\gamma)\bigr]$$

with $h=r$, this gives

$$T_1=\frac{1}{\gamma}\mathbb{E}[(Z_\gamma-\lambda)r(Z_\gamma)] =x\\mathbb{E}[r(Z_\gamma+1)-r(Z_\gamma)].$$

We now have these ratios: $\frac{p_\gamma(z+1\mid x)}{p_\gamma(z\mid x)}=\frac{\gamma x}{z+1}$ and $\frac{p_\gamma(z+1)}{p_\gamma(z)}=\frac{\gamma\\langle X\rangle_{z}}{z+1}$ with $\langle X\rangle_{z}:=\mathbb{E}[X|Z_\gamma=z]$ (from Lemma 4) yield

r(z+1)-r(z)=\log\frac{x}{\langle X\rangle_{z}}, \text{ and } T_1=x\\mathbb{E}\\left[\log\frac{x}{\langle X\rangle_{Z}}\right].

Combine these two terms now, to get:

\frac{d}{d\gamma}R_x = x\\mathbb{E}\\left[\log\frac{x}{\langle X\rangle_{Z}}\right] +\mathbb{E}\\left[\langle X\rangle_{Z}-\frac{Z_\gamma}{\gamma}\right].

Since $\mathbb{E}[Z_\gamma\mid X=x]=\gamma x$, the second expectation equals $\mathbb{E}[\langle X\rangle_{Z}]-x$. Therefore:

$$\frac{\mathrm d}{\mathrm d\gamma} R_x =\mathbb{E}\\bigl[x\log\frac{x}{\langle X\rangle_{Z}}-x+\langle X\rangle_{Z}\bigr] =\mathbb{E}\\bigl[\ell(x,\widehat x^{*})\bigr],$$

where $\widehat x^{*}=\langle X\rangle_{Z}$ and $\ell(u,v)=u\log(u/v)-u+v$. This is Eq. (12).

2. "Missing proof: In the proof of Lemma 2, I only see the "only if"......."

On Lemma 2 ("if" part): We are once again grateful to the reviewer for catching this. The appendix currently proves the "only-if'" direction (linear conditional mean $\Rightarrow$ Gamma prior). We will add the converse, which follows directly from Poisson--Gamma conjugacy: if $X \sim \mathrm{Gamma}(\alpha, \beta)$ and $Z_\gamma \mid X \sim \mathrm{Pois}(\gamma X)$, then $X \mid Z_\gamma = z \sim \mathrm{Gamma}(\alpha + z, \beta + \gamma)$, hence $\mathbb{E}[X \mid Z_\gamma = z] = (\alpha + z)/(\beta + \gamma) = az + b$ with $a = 1/(\beta + \gamma)$, $b = \alpha/(\beta + \gamma)$. This completes both the "if" and "only if" directions of the proof.

3. "Weak numerics: In Table 1, among the six distributions chosen......"

Note: We sincerely request the reviewer to have another glance at Table 1.

There is a column for “true NLL” and boldfaces here reflect whichever baseline yields the closest estimate to the ground-truth NLL, thereby becoming the “best values”. We acknowledge that this may not have been immediately clear, and we will revise the caption of this table to clarify our intent.

On this metric, ItDPDM outperforms LTJ on 4 out of 6 distributions. The remaining two cases (BNB, NBinomMix) have an implicit bias towards LTJ due to the binomial thinning process that underlies LTJ’s noising scheme. Overall this emphasizes the numerical advantage of ItDPDM: it consistently yields likelihood estimates that more closely align with the true NLL across a diverse set of discrete distributions.

Regarding generation quality: This is apparent currently due to CIFAR10's 32x32 resolution: we also refer the reviewer to Appendix Fig. 19 for comparing a denoised CIFAR-10 sample from DDPM and ItDPDM. In the context of symbolic music, Appendix Fig. 10 shows that the pitch histogram from ItDPDM-generated symbolic music closely matches the ground truth distribution. Additionally, Fig. 9 (App. B.5.1) includes piano roll visualizations for unconditionally generated samples. We will include: 1) more image generation examples for reference, including some high-resolution images and 2) generative fidelity metrics for a high-resolution image dataset (like CelebA) in the final version for further strengthening our work.

4. "Simplified univariate setting: Both the sampler and the loss........"

Experiments for multivariate setting: We acknowledge that the results presented in the paper are restricted to a univariate setting for simplicity. To further demonstrate ItDPDM’s ability to model multivariate discrete data, we extended our synthetic experiments to a bivariate (2D) joint distribution. First example: we define the joint distribution as a product of two independent univariate marginals using the same parameters from the 1D synthetic experiments. We report the WD metrics in this case as follows: (with the same training settings as in Appendix C.2.)

Secondly, in addition to the above simple multivariate setting, we also consider two dependent 2D count distributions:

• Bivariate Poisson: Let $U\sim\mathrm{Pois}(\lambda_0)$, $V_1\sim\mathrm{Pois}(\lambda_1)$, $V_2\sim\mathrm{Pois}(\lambda_2)$ be independent; define $X_1=U+V_1$, $X_2=U+V_2$. Marginals: $X_1\sim\mathrm{Pois}(\lambda_0+\lambda_1)$, $X_2\sim\mathrm{Pois}(\lambda_0+\lambda_2)$. Setting $\lambda_0 = 3, \lambda_1 = 2, \lambda_2= 2$, we draw $U,V_1,V_2$, and thereby $(X_1,X_2)$.

• Gamma–Poisson: Let $\Theta\sim\mathrm{Gamma}(k = 2,\beta = 1)$; given $\Theta$, we draw $X_i\mid\Theta\sim\mathrm{Pois}(\Theta)$ independently for $i=1,2$.

In these two settings, the evaluated WD scores are:

Theoretical extension to multi-variate settings: Additionally, we would like to call attention to [2] from ISIT’13, which generalizes the I-MMLE identities for vector poisson channels; this would imply that all the results of this paper naturally extend to a multi-variate setting. (along with the updated proofs stated above)

We will incorporate these extensions in the appendix to emphasize the natural extension and benefits of ItDPDM to multivariate settings. Model/loss are evaluated pixel-by-pixel for our proposed method and all the baselines: we will also add this note in the final version. We also refer the reviewer to the pseudocode in Algorithm 1, step 6 for clarification regarding this.

5. "Unclarities and typos: I list the following for consideration,..."

Thank you for pointing this out, we understand that this could have caused confusion. Line 132 states that in Lemma 4, we express the optimal estimator in terms of the marginal, which P_{Z_\gamma} was supposed to capture. In Eq. (12), we acknowledge that this is a typo and we will replace \calP with P.

Regarding Algorithm 1, line 4: we thank you for pointing out this typo, we will replace it with the correct form from Figure 4:

$z_\gamma \sim \frac{\text{Poisson}(\gamma x)}{1+\gamma}$

We hope this clarifies that it is no longer inconsistent with figure 4. For clarification regarding Algorithm 1, line 6: ($q$… is importance sampling) we refer the reviewer to Appendix Sec. A. 2. We will also mention this in the main paper for final version. Details regarding the data transform function can be found in Appendix Sec. B. 2. We also acknowledge the typo in Algo 2, line 5, and we will change this to $\hat{x}_0$ for the final version. And, finally, we understand that there is no substantial reason to cite [12] here, and will omit it from the pseudocode in the final version.

We sincerely thank the reviewer again for their thoughtful and critical feedback. We believe the points mentioned above: both theoretical and empirical, meaningfully strengthen the paper and address your concerns.

References:

[1] Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information‑Theoretic Diffusion. ICLR 2023.

[2] Liming Wang, Miguel Rodrigues, and Lawrence Carin. Generalized Bregman Divergence and Gradient of Mutual Information for Vector Poisson Channels. IEEE International Symposium on Information Theory ISIT 2013.

We sincerely thank the reviewer Ptu5 for raising their score. Additionally, we appreciate the reviewer’s additional questions and hope they found the earlier clarifications helpful.

“I couldn't follow why $T_1=\frac{1}{\gamma}\mathbb{E}[(Z_\gamma-\lambda)r(Z_\gamma)]$.”

As $p_{\gamma}(z|x) = e^{-\gamma x}\dfrac{(\gamma x)^{z_\gamma}}{z_\gamma!}$, then the derivative w.r.t. $\gamma$ can be written compactly via a log-derivative:

$$\partial_{\gamma} p_{\gamma}(z|x) = p_{\gamma}(z|x)(\frac{z_\gamma}{\gamma}-x)$$

Therefore, $T_1$ can be expressed as:

$T_1 = \sum_z p_{\gamma}(z|x)(\frac{z_\gamma}{\gamma}-x) \log \frac{p_\gamma(z \mid x)}{p_\gamma(z)}$

which implies: (with $\lambda = \gamma x$)

$T_1 = \frac{1}{\gamma}\mathbb{E}[(Z_\gamma - \lambda)r(Z_\gamma)]$

where we defined above $r(z) = \log \frac{p_\gamma(z|x)}{p_\gamma(z)}$

Eq. (12) can interestingly also be derived using a limit-based approach as in the Incremental Channel proofs in App. D.3 or a similar approach as above starting from Theorem 4.2 in [1].

We will incorporate the detailed version of the proof sketch for Eq. (12) in the final version.

“While the 2D experiments (esp. the dependent one) are good to see, I still feel that the scenarios are too simple.”

For image‐like data, most baselines adopt a factorised likelihood: each pixel’s channel count is treated as an independent Poisson (or Gaussian) while spatial correlation is captured by the convolutional backbone. In that sense, moving from 1-D to a 2-D grid does not require a new analytical noise model; it only changes the dimensionality of the tensor the UNet processes. Our implementation follows this standard practice: PRL is summed pixel-wise, and the network learns cross-pixel structure exactly as in existing diffusion models.

Additionally, to illustrate that the Poisson framework itself scales, we have incorporated the toy bivariate (both independent and correlated) experiments and cited [2] which extends the I-MMLE identity (and all other results that follow) to vector Poisson channels. Together, the theory (vector extension) and the toy numerics demonstrate that ItDPDM is fundamentally scalable with the pixel-wise formulation being simply the common and empirically effective choice for images. We will emphasise this rationale in the main paper and point readers to ItDPDM's (natural) multivariate extensions in the appendix.

“Still, in the paper, it is not clear to me how the model and loss are evaluated for image data such as CIFAR-10. I would presume that [2] (as cited above) is used here.”

Forward process: independent Poisson noise is added to every pixel channel $x_{i,j,c}\in\{0,\ldots,255\}$.

Reverse model: a 2-D UNet predicts the full image $\hat{x}$, so spatial dependencies are learned by the network rather than through factorised likelihood terms.

Loss: the Poisson-reconstruction loss is summed pixel-wise (Algo 1, Line 6): $\ell = \sum_{i \in B} \mathrm{PRL}\bigl(x_i,\hat x_i\bigr)$ mirroring the scalar definition. Gradients remain valid due to the vector-Poisson extension from [2] preserving the I–MMLE identity component-wise.

We hope that these clarifications resolve the remaining points of confusion and thank the reviewer once again for their thoughtful comments and for raising their score.

References:

[1] Rami  Atar and Tsachy  Weissman. Mutual Information, Relative Entropy, and Estimation in the Poisson Channel. IEEE Transactions on Information Theory (2012).

[2] Liming Wang, Miguel Rodrigues, and Lawrence Carin. Generalized Bregman Divergence and Gradient of Mutual Information for Vector Poisson Channels. IEEE International Symposium on Information Theory ISIT 2013.

We sincerely thank the reviewer for their constructive feedback on our work regarding the novelty of our method, choice of experiments and for their intriguing follow-up questions. We have addressed the reviewer’s comments below:

1. "I get the point of Fig. 3, but it looks strange at a glance. I didn't feel I fully understood it. I guess Fig. 6 is supposed to show that you fixed this problem?"

Fig. 3 (the diagnosis) contrasts a continuous Gaussian DDPM (green) fit with the true density (blue) on the NYC-Taxi pick-up data. The underlying distribution is sharply bimodal. The top panel overlays the density estimate produced by DDPM (since it maps discrete counts to a continuous latent), which smooths the second peak into a long shoulder on the discrete histogram; therefore totally failing to capture the second peak.

Fix shown: Yes, Fig. 6 plots the learned PMF of each model for all six synthetic datasets, demonstrating that ItDPDM (orange) now places correct mass on the modes while DDPM (green) still flattens the tail. In the revision: we will also (i) label the two modes explicitly in the caption, and (ii) add a sentence pointing the reader forward to Fig. 6 for the quantitative confirmation, avoiding any ambiguity.

2. "Eq. 11 is an intriguing expression that reminds me of your reference [56]. I believe you could write a first order ODE for P_gamma(z), then use this to define sampling dynamics (based on adding counts in a random order autoregressive way). This would avoid the need for gamma noise scheduling."

Exactly (great observation)! Eq. 11 is actually a closed form solution to the following first order ODE:

$$\partial_\gamma P_\gamma(z) = \frac{(z+1)P_\gamma(z+1) - (\gamma + z)P_\gamma(z)}{\gamma}$$

Solving this Kolmogorov-forward ODE forward (or its time-reversal) yields a continuous-time counting process akin to the discrete-state SDE in [56]. We choose the log-SNR schedule plus importance sampling for two practical reasons:

(i) it let's us reuse standard CT diffusion solvers (Euler, Heun, DPM-Solver), and

(ii) it helps us build our implementation on top of standard Gaussian baselines.

The autoregressive “add counts in random order” interpretation is a promising idea; we will note this connection in Sec. 3 and include a brief discussion in Appendix Sec. G for future work.

3. "Did you need to define a gamma noise schedule? I missed that detail. It would be nice in general if there were space to discuss the sampling more."

An explicit/nuanced “noise schedule” is not necessary: $\gamma$ is treated as a continuous random variable. During training, we sample:

$$\alpha = \log \gamma \sim \mathrm{Logistic}(\mu=0,s=1);\text{ } \gamma = e^{\alpha},$$

clipped to the tail‐bound interval [$\gamma_{\min} \approx 10^{-5},\,\gamma_{\max}\approx 10^{5}$].

This importance‐sampling procedure places more draws where the PRL integrand has high variance and provides an unbiased Monte Carlo estimate of the integral (Eq. 19).

For generation (Alg. 2) we integrate the reverse process from $\gamma_{\max}$ down to $\gamma_{\min}$. By default we discretize that range into 100 log-spaced points, but because the formulation is continuous-time, one can freely (i) use fewer points for a speed-up or (ii) plug in any adaptive ODE solver without retraining.

We additionally refer the reviewer to R1 response point (2.) under the Weaknesses section for an ablation on the gamma schedule.

4. “Another nice paper that I was reminded of when you gave Lemma 2: Dytso, A., & Poor, H. V. (2020). Estimation in Poisson noise: Properties of the conditional mean estimator. IEEE Transactions on Information Theory, 66(7), 4304-4323.”

Thank you for recalling our attention to this reference, we will cite this in our final version. We are once again grateful and sincerely appreciate that reviewer 3Fg2 found our work interesting.

We thank the reviewer for their thorough, constructive feedback, and appreciate that they found our work novel. We also believe that the concerns raised mostly stem from misunderstandings, which we clarify below:

“Weaknesses: The method can only estimate distributions for non-negative data and tends to produce high-biased estimates………”

On non-negativity & domain suitability: ItDPDM is explicitly designed for non-negative discrete data (e.g., images, counts, symbolic sequences), so the non-negativity constraint is a feature, not a limitation. Many real-world data domains (pixel intensities, photon counts, MIDI tokens, word frequencies, genomics, etc.) are naturally non-negative and discrete with no pre/post-processing needed. Below are some examples of application areas supporting this argument: (DS = discrete-state)

• Images & audio are ultimately stored as integers: 8-bit RGB or 16-bit PCM; discrete-state diffusions already show gains here ([1], [2], [3])
• Symbolic music (note numbers, durations), MIDI event streams: state-of-the-art models use discrete diffusions ([1], [4])
• Photon-limited imaging, low-light vision record integer photon counts ([5], [6]),
• Single-cell RNA-seq, metagenomics, and other omics assays produce gene–count matrices ([7]),
• Neuromorphic event cameras output per-pixel spike counts [8]
• Word-frequency bags in NLP (Zipf distribution), Poisson processes in queueing are classic count data [9].

We also refer the reviewer to the last App. Sec. K on related work that highlights the growing need for handling non-negative discrete data carefully.

ItDPDM also works for any continuous data: App. Sec. C.4 demonstrates how ItDPDM generalises even to continuous distributions with ItDPDM outperforming DDPM in all cases except one (App. Table 7). We note that preprocessing would be necessary for general continuous distributions to bring them to a non-negative range after which ItDPDM can be directly used.

For most continuous data in practice, a simple shift/rescale (e.g., mapping $[0,1]$ to $[0,\infty)$) or more generally, an invertible transform would suffice and does not affect results with regards to likelihood computation, denoising or generation.

Regarding the “over-bright” appearance in Fig.~4: This snapshot shows a result of using a few denoising steps ($T=10$) starting at very low SNR ($\log \gamma \approx –5$), where the Poisson prior contributes near-zero photons. Since intensity reconstructions start from black and accumulate photons, some intermediate frames appear brighter. However, final samples (after all reverse steps) are well-balanced, as evidenced by comparable or better FID/SSIM on CIFAR-10 (see Table~3) and App. Fig. 19. For this particular image from the CIFAR10 dataset, we empirically verify the denoised image by the proposed ItDPDM to be closer to the ground truth (in terms of mean squared error) than the one denoised by the Gaussian DDPM baseline. This eliminates any concerns regarding over brightening of image.

Figure 10 in App. C.1 compares generated vs. ground truth pitch histograms for symbolic music, showing no systematic amplification bias (or a shift). To further clarify, the final Gaussian denoised image is not the ground truth, with the ground truth in fact being closer to the Poisson output. We totally understand that this could have caused confusion, and will add the ground truth image for reference in the final version.

On the “high-biased” NLL estimate: Our NLL estimator is an analytic upper bound (Eq.~18), reflecting the gap between the learned denoiser $\hat{x}$ and the Bayes-optimal estimator $\hat{x}^*$. This is a principled design choice: we report a conservative likelihood rather than a variational lower bound like ELBO. In practice, this gap shrinks with training and is $<0.02$ bits/dim on CIFAR-10.

Questions:

1. “All the experiments in this paper predict $\hat x$ rather than $\hat \epsilon$....."

All models in the paper output the clean signal estimate $\hat x(z_\gamma,\gamma)$ (without any explicit reparametrization); firstly, we do not train an $\hat\varepsilon$-network for the Poisson channel, because $\varepsilon$ has no natural analogue in this case. For the Gaussian channel, the two parameterisations are interchangeable, as shown below: (starting from the Gaussian channel introduced in the main paper)

$

\varepsilon - \hat{\varepsilon} &= z_{\gamma} - \sqrt{\gamma} x - (z_{\gamma} - \sqrt{\gamma}\hat{x}) = \sqrt{\gamma} (x - \hat{x}) \Longrightarrow
|\varepsilon - \hat{\varepsilon}|_2^{2} = \gamma\|x - \hat{x}|_2^{2}.

$

This implies: $MMSE_{x}(\gamma) = \mathbb{E}\bigl[\lVert x - \hat{x}(z_\gamma,\gamma) \rVert_2^2\bigr],$ and therefore

$MMSE_{\epsilon}(\gamma) = \mathbb{E} \bigl[\lVert \varepsilon - \hat{\varepsilon}(z_\gamma,\gamma) \rVert_2^2\bigr] = \gamma MMSE_{x}(\gamma)$

Therefore, under the squared-error loss, the two MMSEs differ only by a scale factor. We also verify this empirically, with both the variants of DDPM yielding similar performance. For the Poisson channel, we optimise the Poisson Reconstruction Loss (PRL), which is typically expressed in $\hat x$-space. Therefore, reporting MSE in $\hat x$-space keeps metrics consistent across Gaussian and Poisson baselines.

2. “In the synthetic data experiments, DDPM does not….”

• Table 1 numbers are mean ± s.d. (standard deviations) over five random-seed runs (50k samples each); error bars are printed in the table and these details are outlined in Appendix Sec. C.2.
• DDPM is trained with both MSE and PRL objectives (results shown below), with the PRL training exhibiting better performance. This emphasizes two takeaways:

(i) proposed PRL objective improves performance regardless of channel and

(ii) using poisson channel + PRL (i.e. ItDPDM) is the best overall.

• Regarding the apparent similarity to LTJ: a zoom-in of the PMFs (refer Fig.11–12, App.C.3 “Zoomed-in look at PMF plots”) and the full metric table shows that ItDPDM outperforms LTJ on:

a) 9/12 distributions in Wasserstein distance (WD) (including continuous distributions) and

b) 4/6 distributions in negative log-likelihood (NLL).

The synthetic suite therefore highlights (a) the limitations of Gaussian DDPM on skewed or zero-inflated counts and (b) the edge of our exact-likelihood ItDPDM over the variational LTJ.

3. “For the real-world data experiments, ……….”

• We would like to clarify that Table 2 compares the test-set NLLs of various schemes (not the FID). All models are trained under a fixed compute budget (600 epochs, 100 denoise steps). While the IDDPM backbone has ~2x more parameters and uses DDIM-style self-conditioning; its convergence under this shallow schedule is slightly slower than DDPM, resulting in marginally higher NLLs. With extended training upto 2,000 epochs or sampling with 1000-steps, IDDPM can regain the expected (small) advantage (upto 0.02 bits/dim).
• The key takeaway remains that: PRL improved NLL significantly (by 35-40%) irrespective of architecture, and the Poisson+PRL combination (ItDPDM) yields the lowest NLL overall. Thus, the results here underscore our central claim: that a Poisson-consistent, PRL-based objective is the best for modelling discrete, non-negative data, while backbone choice only fine-tunes performance once sufficient training budget is available.

Further real-world experiments:

We also ran experiments on the CelebA (64x64) (image) dataset for 500 epochs (30 hours wall-clock time) with 100 logSNR (denoising) steps per image, for proposed ItDPDM and Gaussian DDPM baseline. ItDDPM has an absolute 7 point lower FID score (or 0.21 dB better) compared to the Gaussian DDPM baseline.

We once again thank reviewer 7eDB for their thoughtful feedback and we truly hope the above mentioned points provide further clarification regarding the experimental parts.

References:

[1] Matthias Plasser, Silvan Peter, and Gerhard Widmer. Absorbing‑State Discrete Denoising Diffusion Probabilistic Models for Symbolic Music Generation. (2023).

[2] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured Denoising Diffusion Models in Discrete State‑Spaces. NeurIPS 2021.

[3] Mingyuan Zhou, Tianqi Chen, Zhendong Wang, and Huangjie Zheng. Beta Diffusion. NeurIPS 2023.

[4] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic Music Generation with Diffusion Models. IJCAI 2021.

[5] Cindy M. Nguyen, Eric R. Chan, Alexander W. Bergman, and Gordon Wetzstein. Diffusion in the Dark: A Diffusion Model for Low‑Light Text Recognition. 2024.

[6] Shlomo Shamai and Aaron D. Wyner. A Binary Analog to the Entropy‑Power Inequality. IEEE Transactions on Information Theory, 36(6): 1428–1430, 1990.

[7] Valentine Svensson and Lior Pachter. Droplet scRNA‑seq is not zero‑inflated. Genome Biology, 21(1): 35. 2020

[8] Xu Zheng, Yexin Liu, Yunfan Lu, Tongyan Hua, Tianbo Pan, Weiming Zhang, Dacheng Tao, and Lin Wang. Deep Learning for Event‑based Vision: A Comprehensive Survey and Benchmarks.

[9] I. Bar‑David. Communication under the Poisson regime. IEEE Transactions on Information Theory, 1969.

Thank you for the response.

• Regarding the issue of image brightness, I don't find Fig. 19 particularly convincing. Visually, Fig. 18 appears to be of better quality than Fig. 16.

• As for the results in Table 1, I’m mainly concerned about whether the reported NLL values are averaged over multiple runs, since the table doesn’t indicate standard deviations.

That said, I find the other clarifications convincing. I’ve already given a positive score and am inclined to keep my current rating unchanged.

We sincerely thank the reviewer for their thorough feedback and valuable suggestions. We believe that many of the concerns raised might stem from misunderstandings, which we clarify below:

Weaknesses:

1. “I am not sure if I understand correctly but this paper…..”

Eq. 17 rewrites the KL-divergence between the forward Poisson $q_\gamma$ and the learnt reverse $p_\theta$ as an expectation over $\gamma$. So, the integrand is analytic: after marginalising out $z_\gamma$, no latent variables remain. Hence, the NLL is exactly defined and only numerical step is integral evaluation. In practice, we use importance-sampling quadrature for this, yielding an unbiased estimate. Conventional diffusion incurs two levels of approximation: a) the ELBO/surrogate loss replaces the true log-likelihood, b) Monte-Carlo quadrature (or the integral). Our Poisson formulation eliminates the first level to work with the true likelihood as in [1], which also refers to their method as “exact likelihood”. To avoid any ambiguity we will replace “exact likelihood” with “likelihood-consistent objective with tractable 1-D quadrature” and provide an upper bound on the estimator's variance in the final version.

2. “There is no comparison to recent discrete-state…..”

In the case of images and symbolic music (line 255), we already compare against a state-of-the-art variant of the vanilla D3PM ([4]) from [2]. With same synthetic setup, we performed an ablation study on the top-2 gamma schedules for each baseline and the results (evaluated using WD) are as follows:

We will incorporate this ablation study in our final version.

"3. CIFAR 10 and LakhMODO results are only trained over 600 epochs….."

Figure 7 in the main paper shows the loss curves flattening early; thus the 600-epoch setting reported is already in a converged regime (drop in loss of $10^{-4}$ order thereafter for every 100 epochs). We limited all baselines to 100 steps, 600 epochs to keep compute budgets comparable (as in [1]). Extended training of ItDPDM to 1500 epochs earlier with larger log-SNR grid gave very little NLL gain (<0.002 bpd).

4. “Also, no robustness experiment is provided…….”

Our experiments already target sparse/zero-inflated and noisy regimes:

Synthetic experiments: In Sec. 5.1, for sparsity, we include ZIP and PoissMix: datasets which have mass mostly at 0 and ItDPDM achieves the best performance on both.

Noisy data: We assert that ItDPDM demonstrates strong reconstructive capabilities for noisy data. For instance, 1000 CIFAR-10 images corrupted at very low SNR (log SNR = –5) are recovered to log SNR = 4 (ref. image in App. Fig.19). This is also illustrated in Fig. 4, where reverse process recovers the image despite extensive zeroing out of pixels via Poisson noising. This highlights the robustness of ItDPDM, and we will add more reference images, including high-resolution CelebA images for the final version (in the Appendix).

Theoretical justification: In addition, [3] provides guarantees for a mismatched estimator (for bounded divergence between source and “mismatched” distribution), so even for sparse/noisy data, the near-optimal denoiser in the Poisson channel has theoretical guarantees (upper bounds) on the estimation error.

5. “There are redundancies and inconsistencies……”

Notations: We clarify the key notations as follows: $\hat X*$ to denote the optimal denoiser (essentially conditional expectation of $X$ given $Z_\gamma$, which is further denoted by $\langle X \rangle_z$) and $\hat X$ to denote learnt suboptimal denoiser. We will rephrase notation to avoid any ambiguity in the final version.

Implementation: The identity itself is exact: its integrand depends on the Bayes‐optimal estimator $\hat x* (z_\gamma,\gamma)$. At training/inference time, we substitute the learned network output $\hat x(z_\gamma,\gamma)$ in place of $\hat x*$ and evaluate the resulting integral using the unbiased Monte Carlo estimator (see Sec.~4). In practice, this is computed as outlined in (1.)

6. Despite the theoretical guarantees, it is hard to reproduce……..”

We plan to release code, configs, and pre-trained checkpoints. In terms of the wall-clock time, the training time is 3.5 minutes per epoch. The wall-clock time is 35 hours (600 epochs) for both ItDPDM and other baselines. All experiments were run on an AWS g5.12xlarge instance (us-west-2):

7. "The idea of replacing squared error or generalizing better than ELBO is oversold as it only applies to discrete non-negative data"

We thank the reviewer for pointing this out, we will revise the wording as needed to accurately reflect that the improvement primarily applies to discrete non-negative data. At the same time, we would like to point out that a Poisson-consistent objective is valuable for the large growing class of discrete, count-valued, zero-inflated data. For detailed evidence, examples and exposition, we refer the reviewer XDNg to our response to "Weaknesses" section to reviewer 7eDB [a]. (due to lack of space here)

ItDPDM works for any continuous data: App. Sec. C.4 demonstrates how ItDPDM generalises even to continuous distributions with ItDPDM outperforming DDPM in all cases except one (App. Table 7). (some preprocessing would be necessary for general distributions, this is again discussed in [a])

Questions:

1. “The theorems are proved using the Poisson channel….”

The Poisson noising is analogous to Gaussian noising in DDPM: it defines how we corrupt samples, not what underlying data distribution is. All the theorems make no assumption on $p(x)$, so the theory naturally extends to any data modality. Our experiments include strongly multimodal data (PoissMix, NB-Mix, BNB); along with real-world datasets that are highly multimodal.

2. “What is the gap……”

Example: pick a binary $\mathbf{X} \in \{\pm 1\}$ with equal probability (BPSK) through a Gaussian channel $Y = \sqrt{\gamma} X + N$, where $N \sim \mathcal{N}(0,1)$. The classic I-MMSE identity gives: $\frac{d}{d\gamma} I(X;Y) = \frac{1}{2} \mathrm{mmse}(\gamma) \Rightarrow \frac{1}{2} \int_0^\infty \mathrm{mmse}(\gamma) \ d\gamma = I(X;Y)\big|_0^\infty.$

At $\gamma = 0$, $I = 0$; as $\gamma \to \infty$, $I \to H(X) = \ln 2$ (nats). Therefore, the exact value is: $I = \frac{1}{2} \int_0^\infty \mathrm{mmse}(\gamma) d\gamma = \ln 2$.

For this BPSK case, a convenient form of the MMSE is: $\mathrm{mmse}(\gamma) = 1 - \mathbb{E} \left[ \tanh^2\left(\sqrt{\gamma} Y \right) \right],$ which follows from the posterior mean: $\mathbb{E}[X |Y] = \tanh(\sqrt{\gamma} Y)$. Using an $n$-sample Monte-Carlo (MC) estimator here shows a $\frac{1}{\sqrt n}$-error decay (this can also be proven using CLT) and is of the order of $10^{-3}$ for $n > 1000$. Similarly, for MMLE, a Gamma prior on $X$ gives an exact closed-form linear estimator (Lemma 2 in the paper) and the MC estimator again shows a $\frac{1}{\sqrt n}$-decay with increase in $n$. For example, if $X \sim \text{Gamma}(2,3)$, the MC error is of the order of $10^{-2}$ for $n > 1000$.

3. “Is there a reason that PRL works better on.”

The paper emphasizes PRL as an objective primarily for non-negative data and Zipf/PoissMix datasets exhibit sparse, heavy-tailed behaviour suited PRL. Half-Cauchy is an over-dispersed continuous case where LTJ’s thinning process has an extra quadratic-tail KL term to better fit those outliers. In this case, PRL is only second to LTJ and better than DDPM, while remaining superior in other continuous cases.

4. “How would this work……”

We refer the reviewer to: point (4.) in Weaknesses for low SNR scenarios. and point (2.) for log SNR-based ablation. We use SoTA architectures (UNet for images, ConvTransformer for music) while tuning the 1) training objective and 2) diffusion modeling.

We once again thank Reviewer XDNg for their feedback and for giving us the opportunity to address their concerns. We truly hope the above response helps address and clarify the mentioned weaknesses/questions.

References:

[1] Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information‑Theoretic Diffusion. ICLR 2023.

[2] Matthias Plasser, Silvan Peter, and Gerhard Widmer. Absorbing‑State Discrete Denoising Diffusion Probabilistic Models for Symbolic Music Generation. (2023).

[3] Rami  Atar and Tsachy  Weissman. Mutual Information, Relative Entropy, and Estimation in the Poisson Channel. IEEE Transactions on Information Theory (2012).

[4] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured Denoising Diffusion Models in Discrete State‑Spaces. NeurIPS 2021.

We sincerely thank you for your valuable time, thoughtful feedback, and your kind engagement with our rebuttal. If there are any remaining concerns or points that you feel we have not addressed fully, we would be truly grateful for the opportunity to clarify further or make improvements in line with your suggestions.

If our responses have addressed your concerns, we would be deeply appreciative if you would consider reflecting that in your final score. Regardless, we are very thankful for your constructive input throughout the review process.

Best regards,

ItDPDM Authors