Fully Heteroscedastic Count Regression with Deep Double Poisson Networks

We appreciate the reviewer’s thoughtful comments.

Introducing additional uncertainty approaches, such as ensembling methods (Deep Ensembles, Batch Ensembles, etc.), could be beneficial.

An important aspect of our work is the interplay between DDPN and Deep Ensembles. We demonstrate this connection throughout the paper. In Section 3.4 we show how individual DDPNs can be combined as ensembles. Then, in the bottom half of Table 2 we present results with ensemble DDPNs. Table 2 demonstrates that learning ensembles of DDPNs improves accuracy and the quality of predictive uncertainty.

We suspect that similar results will hold for BatchEnsembles, as this type of ensemble just changes how the weights of each member are derived: combining slow shared weights and fast, independent rank one weights. The principles we propose in this paper could easily be applied to other types of ensembles. We leave the validation of this hypothesis to future work.

Finally, in the discussion with Reviewer 68NH, we discuss how DDPN can be combined with other UQ methods such as Bayesian Neural Nets, Evidential Methods, Laplace Approximation and Monte Carlo Dropout.

We appreciate the reviewer’s thoughtful comments and helpful feedback.

Comparison to the Natural Posterior Network

We have followed the official repository to download the bike-sharing dataset file and pre-process exactly as used in the paper reviewer mentioned. For training, the paper mentioned they perform training after the grid search in the space [1e−2, 5e−4], since the search step is not specified, we took the log-scale space step as below: [0.01, 0.005623, 0.003162, 0.001778, 0.001, 0.000708, 0.0005], and found the lr as: 0.003162 in our model’s configuration. Following the exact same settings, we conducted evaluation on 5 rounds of training and report the mean and standard deviation of our model’s performance on RMSE, the results are:

How does DDPN compare to other uncertainty methods?

In short, the objective function proposed in Equation 1 enables the network to capture aleatoric uncertainty over count data. We show how this can be combined with Deep Ensembles to better capture epistemic uncertainty. Table 2 shows this combination is effective. DDPN is presented in our paper in terms of maximum likelihood + ensembles for 1) simplicity, 2) effectiveness, and 3) likelihood of community adoption. DDPN can easily be combined with other UQ methods.

Bayesian Neural Networks

One could put a prior over the weights of the network (ideally an isotropic Gaussian prior). Let $\theta$ denote the neural network parameters, $f_\Theta(x_i)$, and $\mathcal{D}$ denote the training dataset. The log posterior is:

$\log p(\theta | \mathcal{D})) = \log \frac{1}{Z} + \log p(\mathcal{D} | \theta) + \log p(\theta)$

where $\frac{1}{Z}$ is the normalizing partition function and is usually dropped during inference.

Fortunately, the negative log likelihood, $-\log p(\mathcal{D} | \theta)$ is already defined in Equation 1 of our paper, and the log gaussian prior is easy to compute, $\log p(\theta) = -\frac{1}{2} \log (2\pi \sigma_0^2) - \frac{1}{2\sigma_0^2} (\theta - \mu_0)^2$, where the prior hyperparameters are $\mu_0$ and $\sigma_0^2$.

Then one could choose the preferred inference algorithm: MAP, HMC, SGLD etc… and estimate the posterior.

Empirically, Bayesian neural networks can outperform Deep Ensembles when using high fidelity inference algorithms such as Hamiltonian Monte Carlo. However, in practice MCMC-based inference is impractical and DEs often outperform less exact inference methods (i.e., SGLD, Variational Inference etc…) [Izmailov et al. What Are Bayesian Neural Network Posteriors Really Like? ICML’21]. We suspect the same results hold for DDPNs.

Moreover, many recent works have directly connected Deep Ensembles to Bayesian Inference by showing that DEs are a coarse approximation of the posterior, sampled and multiple modes with no local uncertainty [Fort et al. Deep Ensembles: A Loss Landscape Perspective. 2019][Wilson and Izmailov. Bayesian Deep Learning and a Probabilistic Perspective of Generalization. NeurIPS’20].

The effectiveness, simplicity and attractive theoretical properties of DEs motivated our decisions to use them in our experiments.

Evidential Approaches

DDPN could also be easily applied with evidential regression techniques [Amini et al. Deep Evidential Regression. NeurIPS’20]. Because DDPN uses a likelihood function during training, one would simply have to specify evidential priors over the parameters of the DDPN, $p(\mu)$ for the mean and $p(\gamma)$ for the inverse dispersion. Then, the network would be trained to predict the parameters of the higher-order evidential distribution.

Laplace Approximation (LA)

This is perhaps the easiest since LA is typically a post-hoc technique. One would train the DDPN in the standard way described in our paper. Then, one could apply any of the post-hoc second-order, covariance approximation methods described in [Daxberger et al. Laplace Redux – Effortless Bayesian Deep Learning. NeurIPS’21]

Dropout-based uncertainty

Monte Carlo dropout estimates epistemic uncertainty by randomly dropping out weights at test time and approximates the Bayesian posterior.

DDPN can easily be combined with MC dropout by 1) training the single-member DDPN to convergence, and 2) applying the MC dropout procedure with $T$ different forward passes through the dropped out model [Gal and Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. ICML’16]

However, [Lakshminarayanan et al. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NeurIPS’17] show that MC dropout is clearly inferior to DEs.

We thank the reviewer for the thoughtful feedback.

Monotonicity vs tend to infinity

We agree with this remark and propose to change Def. 3.2 to:

where $\lim_{\hat{\phi} \to \infty} d(\hat{\phi}) = \infty$ and $\lim_{\hat{\phi} \to \infty} a(\hat{\phi}) = 0$

Fortunately, the proof in Appendix C.3 holds under this new definition, as these limits are in fact used to show that the residual error tends towards 0 (lines 1108-1109). With this change, our proof of Prop. 3.4 also remains valid (since logx tends to infinity and 1/x tends to zero).

Normalizing constant

To simplify our objective, we followed previous work (see below) and assumed $c(\mu, \gamma) = 1$. This facilitates easier differentiation. We will make this more clear in App. A.1.

• See Fact 1 (Eqn. 2.10) of [Efron, B. "Double exponential families and their use in generalized linear regression." Journal of the American Statistical Association. 1986]
• Follow-up work has also set $c(\mu, \gamma)=1$ [Chow, N., and David Steenhard. 2009. "A flexible count data regression model using SAS Proc nlmixed”]

Max/Min in Appendix 1

We propose two changes to the derivation of our objective to increase clarity and align with convention:

1. Replace max/min with argmax/argmin
2. Clarify that we maximize over $N$ training examples (see below) and derive the per-instance loss defined in Equation 1 (see discussion below)

The NLL becomes: $\arg\min_{\mu_i, \gamma_i} [ -\sum_{i=1}^N \log p(y_i | \mu_i, \gamma_i)]$.

Lack of Log link in Appendix A.1

In App. A.1 we derive the training objective. In contrast, Section 3 describes how it can be used to train DDPN. The connection between the two is trivial (exponentiate the log link to evaluate Eq. 1) and is stated in Footnote 1 (pg. 4).

Lack of Summation in Equation 1

Eq. 1 expresses the loss for a single training example, $x_i$. We state in lines 183-184 that the loss is:

averaged across all prediction / target tuples…in the dataset

To be more explicit, we propose to change $\mathcal{L}$ to $\mathcal{L}_i$.

Prop. 3.1: DDPN regressors and full heteroscedasticity

In line with prior work, we use Efron's approximations for the first two moments in the proof of Prop. 3.1. We propose to revise this proposition: With mild assumptions, DDPN regressors are fully heteroscedastic (where we assume that Efron's approximations hold).

How good are Efron's approximations? We introduce the concept of moment-deviation functions (MDFs) to assess this theoretically:

Let $Q$ be a family of distributions parametrized by $\psi\in\mathbb{R}^d$. Suppose we are given $\hat{\psi_n}:\mathbb{R}^n\to\mathbb{R}^d$, which outputs $\hat{\psi_n}(\boldsymbol{\mu})$ s.t. the first $n$ moments of $Z\sim Q_{\hat{\psi_n}(\boldsymbol{\mu})}$ are nearly equal to target moments $\boldsymbol{\mu}=(\mu_1,...,\mu_n)$. Then for any pair $(Q,\hat{\psi_n})$, {$\varepsilon_i:\mathbb{R}^n\rightarrow\mathbb{R}$} for $i=1..n$ are moment-deviation functions if, for any valid $\boldsymbol{\mu}$, if $Z\sim Q_{\hat{\psi_n}(\boldsymbol{\mu})}$, we have $|\langle Z^i\rangle -\mu_i|\leq\varepsilon_i(\boldsymbol{\mu})$ for all $1\leq i\leq n$.

We focus on n=2, (error for the mean/variance). If we can pick $\hat{\psi}$ s.t. $\varepsilon_1,\varepsilon_2$ are small, $Q$ is flexible, as there are parameters that can roughly achieve any mean and variance. In the Gaussian case ($\mathcal{N},\mathbb{I}$), we have $\varepsilon_1 =\varepsilon_2=0$.

Proposition

Let $DP$ denote the Double Poisson family. Set $\hat{\psi_2}(\mu_0, \sigma_0^2) = (\mu_0,\frac{\mu_0}{\sigma_0^2})$. Letting $\gamma_0=\frac{\mu_0}{\sigma_0^2}$, the MDFs for $(DP,\hat{\psi_2})$ are:

$

    \varepsilon_1(\mu_0,\sigma_0^2) &= \left|\frac{\sum_{y=0}^{\infty}s(\mu_0, \gamma_0, y)(y - \mu_0)}{\sum_{y=0}^{\infty}s(\mu_0,\gamma_0,y)}\right|  \\\\
    \varepsilon_2(\mu_0, \sigma_0^2)&=\left|\frac{d(\mu_0,\gamma_0)\gamma_0^{\frac{1}{2}}\sum_{y=0}^{\infty}s(\mu_0,\gamma_0,y)-\gamma_0(\sum_{y=0}^{\infty}s(\mu_0, \gamma_0,y)(y-\mu_0))^2}{\gamma_0(\sum_{y=0}^{\infty}s(\mu_0,\gamma_0,y))^2}\right|

$

where: $h(z)=\frac{e^{-z} z^z}{z!}, r(\mu,\gamma,z)=\gamma(z-\mu +z\log\mu -z\log z), s(\mu,\gamma,z)=h(z)\exp(r(\mu,\gamma,z)),$ and $d(\mu,\gamma)=\gamma^{-1/2}\left[\sum_{y=0}^{\infty}s(\mu, \gamma, y)(\gamma(y-\mu)^2-y)+\sum_{y=0}^{\infty}s(\mu,\gamma,y)(y-\mu)\right]$.

If desired, we can provide the proof to this proposition in the follow-up response. We plot the error incurred via Efron’s estimates on a grid of target means and variances, using 100th partial sums (https://anonymous.4open.science/r/ddpn-651F/deep_uncertainty/figures/artifacts/epsilon_1.png). To see epsilon_2, change the filepath to epsilon_2.png. Except for the case of small μ, high σ², the error is essentially zero. Thus, in most settings we can treat DDPN as fully-heteroscedastic. Empirically, DDPN produces flexible, well-fit distributions (Fig. 3/4, Table 2).

We sincerely thank the reviewer for the recognition of our work.