Understanding Pathologies of Deep Heteroskedastic Regression

Thank you for the comments. We address your concerns below:

(main) Is there a sharp phase transition? … It would be interesting to calculate, at least under certain simple generative models, the free energy and check if there is a first-order/second-order phase transition.

We agree that the phase transitions are worthy of further investigation. Unfortunately, as seen in the cross sections through $\rho-\gamma$ space, though these phase transitions are continuous, they are not smooth in the order parameter, and we are unable to study this behavior analytically.

(main) Technically speaking, not sure which part of the paper is innovative---for example, Eqns 7--10, are they new or semi-new (i.e., similar derivations are obtained in a different context)? I would be skeptical that Eqns 7--10 are entirely new. Also, the proposed regularization is well-studied in the literature.

Yes, weight decay is a well-studied method for regularization, but we believe that our parameterization is unique, and splitting the regularization across two separate networks has not yet been studied in depth. [2] also looks at fitting heteroskedastic regression models with different levels of regularization applied to the two networks, and we will add this reference to the camera-ready version.

It is a bit handwaving when transitioning from parameter norm regularization to gradient regularization

We address this concern in the global comment above.

[2] Sluijterman, L., Cator, E., & Heskes, T. (2023). Optimal Training of Mean Variance Estimation Neural Networks. arXiv preprint arXiv:2302.08875.

We appreciate the positive feedback and provide replies to your comments below:

Below equation (7), the authors ‘consider the scenario in which the inner integral is approximated using a single MC sample’. I wonder if there is any justification for this approximation (experiment, argument, reference, etc.).

We address this in the global comment above.

I don’t understand the sentence below equation (9): ‘Interestingly, both resulting relationships include a regularization coefficient divided by the density of p(x).’ Does the word ‘regularization coefficient’ refer to the term with Laplace operator, which originates from the regularization term?

Yes, that is correct.

Typo: on top of equation (1)

$\Lambda$ represents the precision (inverse-variance), so $\Lambda^{-1/2}$ is the standard deviation.

Thank you for the helpful comments and suggestions. We address your concerns above in the global comment (regularization, single MC sample) and below (Dirichlet energy wrt $p(x)$):

There is a large conceptual leap from the weight decay formulation to using the Dirichlet energy as a regularizer. ... Other work has drawn (similarly loose) links to implicit regularization via backwards error analysis of predictive networks trained with SGD, and probably warrants mentioning https://arxiv.org/pdf/2209.13083.pdf.

Thank you for the useful reference–we will be sure to add it to the camera-ready version. We acknowledge that this was an ad hoc decision, and we will draw in greater ties from the literature. We address this in further detail in the global comment above.

A single Monte Carlo sample is used in the construction, without further discussion or investigation on the limitations of doing so.

We address this in the global comment above.

Taking the Dirichlet energy with respect to p(x) may be interesting and warrants discussion (or future work). Assuming p(x) to be uniform for the purposes of numerics is concerning, and doing so may help alleviate this issue.

The analysis we performed is still valid because, pointwise, the limiting cases still hold regardless of the underlying p(x) so long as it is non-zero. For the camera-ready version, we will take the data distribution into account.

Thank you for your comments. We address your concerns below:

...the "no regularization" regime in Prop. 1 does not describe any reasonable learning algorithms, all of which introduce regularization explicitly or implicitly

The setting of Proposition 1 is hypothetical in relation to finding a solution to the proposed NFE, which is an abstraction of the typical cross-entropy loss employed in heteroskedastic regression models. The implicit assumptions here are that the model being fit is overparameterized, a universal estimator, twice-differentiable, and is trained until convergence. For these reasons, the only regularization we are considering are the ones we impose through the gradient norm penalties.

If the authors wish to study a nonparametric estimator such as the one defined in their Eq (7), they should impose constraints on the functions (e.g. Sobolev) and carefully choose a rate of vanishing regularization strength in accordance with the function class.

In summary of our approach, we impose constraints via the gradient norms and take the limits of regularization strengths, $\rho$ and $\gamma$, going to zero and 1 in our analysis. We fear that we may not fully understand your concern. Could you provide some more detail or references to expand on your original comment?

If the authors wish to study estimators without explicit regularization -- as is common in the analysis of overparameterized models -- they should specify the form of implicit regularization (e.g. gradient descent / gradient flow; model parameterization).

We provide details for training and model parameterization in Appendix B.

Furthermore, the challenge of conditional variance estimation arises from overfitting, yet the main result is stated for a population objective without any account for sample size.

In our formulation, we can account for sample size through the $\rho$ parameter that scales the importance of the loss between the regularizers and the likelihood.

Additionally, the references to statistical physics appear completely unnecessary.

Looking at the phase diagrams was an idea inspired by statistical physics. The resulting phase diagram reveals a complex interaction between the regularization strength of the respective functions that cannot be easily summarized by a ratio or linear dependence. This new perspective over combinations of regularization strengths allowed us to notice that an efficient 1-dimensional search along the off-diagonal could be sufficient to find a well-calibrated fit.

The authors are encouraged to study the proposed learning objective in a relevant and non-trivial regime, and possibly to familiarize themselves with notions in learning theory and nonparametric statistics

We acknowledge that there are other avenues for approaching this problem, but do not consider that a reason to avoid other methods. Learning theory and nonparametric statistics do not relate to phase transitions to our knowledge.