We thank the reviewer for the thoughtful review and helpful comments! Below is our discussion and answers to key questions.

Selection of $\beta$ hyperparameter

Tuning can be performed easily with a validation set.

Lack of theoretical contribution

We are not claiming strong theoretical contributions. We do provide some theoretical insight into the behavior of the DDPN loss, which is that mean and dispersion can become entangled during training. Specifically, in Equation 3 we show that the partial derivative of the likelihood function strongly depends on the predicted dispersion.

DDPN does not consistently outperform baselines

As claimed in the introduction, DDPN significantly outperforms models designed for discrete count problems in nearly every case. DDPN also meets or exceeds the performance of Gaussian-based regressors in nearly every case. Our claim is that DDPN is competitive with Gaussian regressors, but also properly represents probabilities over discrete count data.

Missing measures (i.e., calibration and sharpness)

We report Median Precision, a measurement of the sharpness of the predictive distribution below. We have added a revised version of the paper and provided a table of MP for all datasets in Appendix E.1.

As discussed in the second paragraph in Section 4, we omit ECE because of a known bias when comparing continuous and discrete regressors (Young and Jenkins 2024). Instead, we use NLL since it is a proper scoring rule and accounts for both mean accuracy and calibration. We use the continuity correction to ensure that the comparison between discrete and continuous predictive distributions is fair (See paragraph 2 of section 4).

Spencer Young and Porter Jenkins. On measuring calibration of discrete probabilistic neural networks, 2024. arXiv preprint arXiv:2405.12412

OOD detection is overclaimed

In figure 4 a larger $\Delta$ value corresponds to a larger difference between means of the two distributions. If the difference of means is larger, it follows that the central tendency of the distributions is more different. The p-value (from the permutation test) describes how likely this difference is to be statistically meaningful. Therefore, we can see that DDPN has by far the largest delta value, $\Delta=0.460$, demonstrating that it is the best at separating ID and OOD entropy values. This delta value is nearly twice as large as the next best method (Gaussian DNN at $\Delta$=0.238), which is not statistically significant, and more than twice as large as Immer et al, which is statistically significant. For these reasons, we claim that DDPN “exhibits superior out-of-distribution detection.”

We thank the reviewer for the thoughtful review and helpful comments! Below is our discussion and answers to key questions.

Lack of significance of experimental results

As claimed in the introduction, the base DDPN significantly outperforms models designed for discrete count problems in nearly every case. Additionally, the base DDPN meets or exceeds the performance of Gaussian-based regressors in nearly every case. Our claim is that DDPN is competitive with Gaussian regressors, but also properly represents probabilities over discrete count data. This claim is supported by our experiments.

The introduction of $\beta-DDPN$ is to give tunable preference to mean fit or calibration. $\beta-DDPN$ meets or exceeds the performance of its Gaussian counterpart proposed by Seitzer, but also properly represents probabilities over discrete count data.

OOD experiment: comparison to Immer et al.

In figure 4 a larger $\Delta$ value corresponds to a larger difference between means of the two distributions. If the difference of means is larger, it follows that the central tendency of the distributions is more different. The p-value (from the permutation test) describes how likely this difference is to be statistically meaningful. Therefore, we can see that DDPN has by far the largest delta value, $\Delta=0.460$, demonstrating that it is the best at separating ID and OOD entropy values. This delta value is nearly twice as large as the next best method (Gaussian DNN at $\Delta$=0.238), which is not statistically significant, and more than twice as large as Immer et al, which is statistically significant. For these reasons, we claim that DDPN “exhibits superior out-of-distribution detection.”

Why is Double Poisson more appropriate compared to other distributions?

The core problem is that we wish to model probabilities over discrete count data.

The Gaussian distribution is flexible, but is a continuous distribution, and therefore is inappropriate for count data. We identify three pathologies with simply applying the Gaussian to count data in Section 1:

First, the continuous predictive distribution will assign non-zero probability mass to infeasible real values that fall in between valid integers. Second, the predictive intervals are unbounded and can assign non-zero probability to negative values when the predicted mean is small. Third, the boundaries of the predictive intervals (i.e., high density interval or 95% credible interval) are likely to fall between two valid integers, diminishing their interpretability and utility.

DDPN resolves all three of these pathologies

Existing discrete likelihood models produce proper distributions over count data, however they are rigid and cannot model under-, equi-, and over-dispersion. DDPN is more flexible (resulting in better performance) and can model all of three of these cases.

Missing baselines from 'boosting' experiments

While we do not perform any experiments with boosting, we believe the reviewer is referring to our experiments with Deep Ensembles. To the best of our knowledge, Deep Ensembles with very recent Gaussian regressors (Immer et al., Stirn et al., and Setizer et al.) have never appeared in the literature, and are therefore completely new baselines and were not included in the original manuscript. To address the reviewer's concern have added a revision and now include these results in Appendix E.2. In all cases, the DDPN ensemble outperforms these additional baselines in terms of MAE.

We thank the reviewer for the thoughtful review and helpful comments! Especially for the opportunity to discuss and clarify connections of DPPN to previous work. Below is our discussion and answers to key questions.

Connections to Joint GLM

The original Double Poisson GLM presented by Efron 1986 is in fact a joint GLM as it jointly models mean and dispersion of a response variable. McCullagh and Nelder, 1989 define the joint GLM in the following form:

$\eta_i = g(\mu_i) = \boldsymbol{x}_i^T\boldsymbol{\beta}$

$\zeta_i = h(\phi_i) = \boldsymbol{u}_i^T \boldsymbol{\delta}$

Where $E(d(Y_i, \mu_i)) = \phi_i$, and $d(Y_i, \mu_i)$ is a dispersion statistic, which depends on the mean. Here, the covariates, $\boldsymbol{x}_i$ and $\boldsymbol{u}_i$, can either be shared or be different.

This model has two apparent defining properties: 1) that the first two moments depend on covariates; and 2) the dispersion depends on some way on the mean, $\mu_i$

In Section 2.2.1 of our manuscript, we define the DP GLM using the same specification as Efron 1986:

Given parameter vectors $\boldsymbol{\alpha}$ and $\boldsymbol{\beta} = [\beta_0, \beta_1, \beta_2]^T$, let $\text{log}(\hat{\mu_i}) = \hat{\eta_i} = \boldsymbol{\alpha}^T\boldsymbol{x}_i$ and $\hat{\sigma}_i = \frac{M}{1 + e^{-(\beta_0 + \beta_1 \hat{\mu}_i + \beta_2 \hat{\mu}_i^2)}}$

Thus, it is apparent that the DP GLM baseline in our experiments is in fact a joint GLM because 1) DP GLM parameterizes the first two moments as a function of covariates, and 2) the dispersion depends on the mean. In the same section, we discuss limitations of this approach

This approach has two key limitations: 1) the predicted dispersion, $\hat{\sigma}_i$, does not directly depend on the input $\boldsymbol{x}_i$, and is instead a function of the predicted mean, $\hat{\mu}_i$); 2) the hyperparameter $M$ introduces an upper bound on the dispersion, which in turn curtails the feasible range of confidence values. In practice, the authors set $M=1.25$, which hardly allows for under-dispersion ($\sigma > 1$). Both of these measures significantly limit the family of distribution functions the model can learn. Follow-up studies all assume a constant dispersion term, applying even stronger limits on flexibility. In contrast to these, our proposed approach drastically expands the family of functions that can be modeled. DDPN learns a non-linear mapping that can be trained on complex data and can fully disentangle the mean and dispersion, allowing for pure heteroscedastic regression. We also introduce a tunable hyperparameter that allows for custom prioritization between mean fit and overall likelihood calibration.

We also argue that DDPN offers the following advantages:

in contrast to these, our proposed approach drastically expands the family of functions that can be modeled. DDPN learns a non-linear mapping that can be trained on complex data and can fully disentangle the mean and dispersion, allowing for pure heteroscedastic regression. We also introduce a tunable hyperparameter that allows for custom prioritization between mean fit and overall likelihood calibration.

Additionally, we show empirically that DDPN improves over the Joint GLM in Table 2 for two tabular datasets.

Comparing likelihoods and computing calibration

As discussed in the second paragraph in Section 4, we omit ECE because of a known bias when comparing continuous and discrete regressors [1]. Instead, we use NLL since it is a proper scoring rule and accounts for both mean accuracy and calibration. We use the continuity correction [2] to ensure that the comparison between discrete and continuous predictive distributions is fair (See paragraph 2 of section 4).

[1] Spencer Young and Porter Jenkins. On measuring calibration of discrete probabilistic neural networks, 2024. arXiv preprint arXiv:2405.12412

[2] "Continuity Correction." Wikipedia, Wikimedia Foundation, 23 Nov. 2024, https://en.wikipedia.org/wiki/Continuity_correction.

Significance of MAE

As claimed in the introduction, DDPN significantly outperforms models designed for discrete count problems in nearly every case in terms of MAE. Additionally, the DDPN meets or exceeds the MAE performance of Gaussian-based regressors in nearly every case. In some cases, the MAE differences may be within the standard error of the MAE reported, resulting in a statistical tie. This is still consistent with our claim that DDPN is competitive with Gaussian regressors, but also properly represents probabilities over discrete count data.

How does the initialization of $\phi$ mediate the effect of $\beta$?

We have added a revision and include two experiments studying this question in Appendix E.3.

Experiment 1: What if we initialize phi to some high value (via the initial bias in the output head)?

Experiment 2: What is the effect of phi initialization when training with beta? Does it affect the ability of beta to steer the model toward the true mean?

I thank the authors for their efforts in providing detailed responses to our comments. However, several critical points remain unresolved. Thus, I keep my original rating for this paper.

Regarding the relationship with Joint GLMs

• While both Joint GLMs and DP GLMs model the frist two moments, this alone does not imply that the two models are identical. For instance, the authors noted that $\text{Var}(Y_i)$ is bounded by the hyper-parameter $M$ in DP GLMs, whereas such a bound does not generally apply in Joint GLMs.
• Joint GLMs model the orthogonal dispersion parameter $\phi_i$ rather than directly modeling $\text{Var}(Y_i)$. In the standard exponential dispersion model framework, the variance typically takes the form $\text{Var}(Y) = \phi \cdot V(\mu)$. In the case of Gaussian distribution, $\text{Var}(Y_i)$ can be directly modeled since $V(\mu_i)=1$. However, for Poisson distributions, $\mu_i$ and $\text{Var}(Y_i)$ would not be orthogonal, since $V(\mu_i)=\mu_i$. As far as I understand, this is why joint GLMs typically model $\phi_i$ rather than $\text{Var}(Y_i)$. Therefore, without further justification, the authors' claim regarding the first limitation could be perceived as unconvincing and potentially overstated from certain perspective.

Comparing NLLs from different distributions

• Applying a continuity correction may not justify directly comparing the NLL values from different distributional assumptions, as it goes against fundamental principles of likelihood theory.

We thank the reviewer for the thoughtful review and helpful comments! Below is our discussion and answers to key questions.

Modest Improvements over baselines on Amazon dataset

While some of the more modern Gaussian regressors (i.e., Stirn, Seitzer, Immer), are indeed competitive with DDPN, they are still subject to the pathologies and mis-specification issues discussed in Section 1. Our claimed contributions are not that DDPN outperforms all baselines all of the time. Rather, we claim that DDPN can meet or exceed the performance of modern Gaussian regressors, and also produce a properly specified predictive distribution over discrete count data, which the Gaussian does not. If we truly believe that the data are discrete, we should incorporate that belief into our modeling approach.

Comparison to simpler distributions

In all of our experiments we compare to simpler distributions (Table 2, Table 3). Specifically, we compare DDPN to the Poisson DNN and the Negative Binomial (DNN), which are both simpler distributions. Poisson has a single parameter for mean and variance, while the NB DNN has two parameters, but is restricted to variance >= mean, and is therefore less flexible.

We see on tabular data with naturally high dispersion conditioned on the observations (i.e., Bikes in Table 2), Poisson is comparable to DDPN. However, on lower dispersion Tabular data (i.e., Collision in Table 2) and in the more high dimensional data in Table 3, DDPN outperforms existing discrete approaches. In general, DDPN can fit data of all dispersions, even when technically “misspecified” as we demonstrate in Appendix A.2.

Sensitivity to choice of $\beta$

In appendix A.4 we perform a hyperparameter study of the effect of Beta on the COCO-people dataset.

Why does $\beta=0.5$ seem to work particularly well for ensembles?

It is possible that $\beta=0.5$ yields more diverse independently trained models. Thus we capture a greater variety of plausible minima in weight space. In this scenario each mode's mistakes cancel the others out, resulting in a stronger ensemble. If $\beta=1$, it’s likely that the models all tend towards similar solutions where they fit the mean with high (sometimes too-high) confidence. Conversely, if $\beta=0$ the models all tend towards similar solutions with high uncertainty explaining away poor mean fit. A follow-up work should study this more in-depth.