If there is no underfitting, there is no Cold Posterior Effect

Thanks for your feedback. Let us try to clarify the confusion.

Taking the opposite, no underfitting basically means that there is no model with better train and test loss, hence if the Bayesian posterior is not underfitting then of course there is no cold posterior effect. Am I missing something here?

This kind of reasoning can be tricky, we will try to explain here why “no underfitting implies no CPE” cannot be directly proved by simply inverting the definition (without using Theorem 2).

It’s correct that no underfitting means there is no model with both better training and better testing loss simultaneously. Hence, the following must hold if there is no underfitting:

1. If another model has better train loss, it must have worse test loss.
2. If another model has better test loss, it must have worse train loss.

The situation, where increasing $\lambda>1$ gives a better testing loss (there is CPE), itself contradicts neither situation 1) nor 2). It is because it could be the case where the tempered posterior has worse train loss (situation 2). Therefore, no underfitting and CPE could co-exist.

However, situation 2) cannot happen as Theorem 2 shows that the train loss is also enhanced by increasing $\lambda>1$. If you do not use Theorem 2 and try to show the implication from the definition, you do not reach any contradiction.

In Proposition 3, how does the underlying data distribution enter the picture here?

Proposition 3 is a consequence of Proposition 6 in Appendix A.3 and the definition of CPE. To shed some light here, Proposition 6 says essentially that the gradient of the Bayes loss has an equivalent form as the difference between the Gibbs loss of the updated posterior (Eq. (6)) and the tempered posterior; where the “data-generating distribution” appears in the updated posterior. Next, the definition of CPE says the gradient of the Bayes loss is negative $< 0$, i.e, the Gibbs loss of the updated posterior is lower than the Gibbs loss of the tempered posterior. Hence, the Gibbs loss of the posterior cannot reach the minimum, which is the message of Proposition 3. Therefore, we don’t see the data-generating distribution in the conclusion of Proposition 3 since the related quantity is thrown away in some sense in the middle of derivation.

Theorem 4 is a very interesting result and I would have loved to see some empirical verification of it, i.e. if the posterior is optimal at lambda=1, does the train loss really not change anymore if you add a new sample to the posterior (of course one would need to average over adding the sample)? Can you basically predict optimality of lambda=1 from this property alone?

We are really glad that you can appreciate the relevance of this result. Yes, if $\lambda=1$ is optimal, then the training loss does not change for the updated posterior (in expectation). Please note Thm 4 establishes a “if and only if” condition.

We will try to highlight this result more in the new version. And we also plan to include a table like the next one illustrating this result for the linear regression model of Figure 1 under the four setups depicted there.

As can be seen, when there is no CPE the train loss does not change, as predicted by Thm 4. And, as Proposition 6 in the Appendix also predicts, when there is a “warm posterior effect”, the train loss of the expected updated posterior becomes bigger. And when there is CPE, the train loss of the expected updated posterior becomes smaller. We plan to add this discussion to the main paper. Thanks for bringing this up!

Thanks for your feedback. Let us try to clarify the confusion.

The argument is a bit unclear from my perspective. The authors argue that the problem is under-fitting which comes from misspecification. So is the problem the under-fitting or the misspecification which causes the under-fitting

The paper seems to be stepping on previous results and works, claiming that the misspecification of the prior or the likelihood lead to underfitting, and therefore underfitting is the problem that causes CPE. Well if CPE is present when under-fitting is present then the problem falls on either the prior or the likelihood.

My main question is what is the argument that the authors are making? Reading the paper it looks like you are arguing that misspecification is causing the CPE by causing underfitting. It is well known that any misspecification in Bayesian setups leads to lower performance.

I think I have understood the math and the experiments of the paper but the main argument does not seem very novel. Can the authors better explain?

To start, we would like to clarify that the main takeaway message of our work is the following: The presence of CPE implies that our Bayesian setup is misspecified and underfitting at the same time.

This is a simple, clean but actually novel way to interpret CPE.

1. Connections to previous work: The connection with underfitting has not been made before when trying to understand CPE. Most of the current literature focuses on the misspecification problem itself, but not on misspecification in the context of underfitting. We find it quite important that the ML community is fully aware of this point.
2. “it is well known that any misspecification in Bayesian setups leads to lower performance.”: We totally agree. But this “lower performance” can be due to underfitting or overfitting, i.e., you can have a misspecified model that is severely overfitting your training data, as we show in Figure 1 (b). Understanding which of the two is present when we observe CPE is quite relevant. Reviewer wHkC states that this question “ hold[s] profound implications.”

Having said that, we agree that we could do better in characterizing the relationship between misspecification and underfitting and will do so in the final version.

Some of the results are not convincing. A simple linear regression model on synthetic data is not enough in Figure 1. Figure 2 and 3 are also a bit unclear.

One tip that I can give on the presentation is to add the results of the big models in the main text and the toy experiments in the Supplement. Or have a gradual increase of difficulty in the presentation of the experiments. For instance you start with linear regression on synthetic data, then you have a ResNet experiment on CIFAR10 and then on Imagenet.

Thanks for this tip! We will consider your advice in the next version.

Thanks for your feedback. Let us try to clarify in the following.

According to the definition of CPE, as $\lambda$ increases, the test loss should decrease. Additionally, as $\lambda$ increases, the train loss always decreases irrespective of CPE. Up to this point, arguments in Theorem 2, Proposition 3 are accurate but rather self-evident.

As we have answered in the overall rebuttal at the top of the page, we do agree that the connection is, in some sense, straightforward. However, it's noteworthy that this link has not been recognized in the literature. The primary contribution of our study is to make the community aware of this connection. Because it is essential for comprehending and addressing the challenges associated with CPE.

Having said that, our formalization also provides new insights into CPE. Theorem 4 provides an interesting characterization not done before. Reviewer B5C9 is very positive about it. In our opinion, the connection with Data Augmentation does also provide new insights.

However, they have not demonstrated that the distribution, whose existence was proven, becomes a posterior distribution (which should be definable from a new likelihood or a new prior). Hence, according to the definition “underfitting” defined by authors, Insight 1 may not be correct.

Let us first clarify that, in our work, we refer to a “posterior distribution” to any distribution over the parameter space which depends on the training data. We refer to the “Bayesian posterior distribution” to a “posterior distribution” which is the result of the strict application of the Bayes’ rule. I.e., $p^\lambda(\theta|D)$ is a posterior distribution, while $p^{\lambda=1}(\theta|D)$ is the Bayesian posterior distribution. We will make this distinction crystal clear in our paper.

Therefore, Insight 1 is correct as we said “there exists another posterior, rather than another Bayesian posterior, that has lower training and testing loss at the same time.”

As you say, it would be great to know if this tempered posterior can be attained by some other Bayesian posterior coming from another likelihood/prior. We think this indeed is an interesting question. Although we think it is out of the scope to include this in our paper, where the focus is to highlight the connection between CPE and underfitting that has been overlooked so far, there are references showing positive answers, which we provide in the next section.

I believe at least one of the following two pieces of evidence should be provided. The original likelihood (or prior) is misspecified, compared to new likelihood (or prior). An inverse proposition also holds.(i.e., If there is underfitting, there exists CPE).

The first point is highly related to the previous question; it is clear that if CPE is present, prior and/or likelihood are misspecified, otherwise the Bayesian posterior would be optimal and CPE would not be present. As far as we know, at least in certain cases, the tempered posterior can be derived as the Bayesian posterior with a proper likelihood and prior (Wenzel et al., 2020 Wilson, Izmailov, 2020, Nabarro et al., 2022). Please refer to those for more details.

Regarding the inverse proposition; whether it holds is also a good question. However, it is not true in general; consider the extreme case where the prior distribution is a delta distribution centered at a model that does not explain either train or test data, let’s say $\theta_0$. In this setting, the Bayesian posterior is

$$p(\theta|D) \propto P(D|\theta)p(\theta),$$

which is also a delta distribution centered at $\theta_0$. The same happens with any tempered posterior. Even if the model $\theta_0$ is underfitting (there exist models with better training and testing loss), CPE does not appear in this context.

In the current manuscript, I believe that their main claim has not been theoretically clarified.

With the above clarifications, our claim “If CPE is present, there exist alternative posteriors with better train and test loss” theoretically holds. As said before, what we don’t show is that “If CPE is present, there exist alternative Bayesian posteriors with better train and test loss” theoretically holds. We are fully aware this would be a stronger result. But we do consider this paper as a first step in this direction. This work can help to bring attention to this relevant open questions.

Thanks for your positive review. We answer to your questions below:

Your results indicate that larger model capacities should be less susceptible to the CPE since they are less likely to underfit. However, Wenzel et al. (2020), figure 11, actually show the opposite, namely that increasing the depth of an MLP has no effect on the CPE, and that increasing the width makes the effect more pronounced. Do you have any thoughts on this?

Thanks a lot for pointing us to this figure. We did not realize this experiment was present there. One of the possible explanations is that they are using full-tempering while we are using likelihood tempering.

1. For full-tempering, Theorem 2 does not necessarily hold anymore. Intuitively, since \lambda works on both likelihood (data) and prior (regularization) together, the effect of increasing \lambda is mixed, thus not necessarily improving the fit on training data.
2. Thus, the CPE brought by full-tempering, as \lambda increases, does not necessarily come with a better training loss, i.e., training loss may not be improvable. Hence, full-tempered CPE can no longer be interpreted as just underfitting only.
3. As such, for full-tempering, increasing the model capacity may not achieve a lower degree of CPE as in our case. However, we believe it’s still valuable and will reproduce this experiment with likelihood-tempering to clear the argument.

I am trying to understand what the next steps for research on this topic might be. Did you observe any situations where underfitting could not explain the CPE?

If there is CPE, then your Bayesian posterior is underfitting. But, it might be the case that the posterior is underfitting and you do not have CPE. We haven’t said anything about the reverse implication. But, we do think this implication does not really hold in general. In the response to Reviewer wHkC we provide a counterexample.

For us, a relevant question is: is there any problem in working with tempered posteriors? What is preventing us from tuning this hyperparameter when using a Bayesian approach? This lambda hyper-parameter could be tuned using, for example, an independent validation dataset. Providing theoretical arguments in favor of this approach would be valuable. This work would support this, because if we do not do it, we know we are not squeezing out all the power of (generalized) Bayesian methods.

We thank the reviewers for the interactions and communications. During the discussions, however, we realized there was a fundamental misunderstanding in interpreting our work, which led to concerns about novelty. With respect to that, we would like to clarify here:

1. Novelty. Let us stress this again. Despite being simple, straightforward and intuitive, the connection of CPE and underfitting is not present in the literature. We have not found any reference. Only a few papers mentioned that it was traditionally known that Bayesian neural networks are known to underfit. But the link between CPE and underfitting is non-existent in the literature. Our paper should be judged by the novelty of the contribution, not by the complexity of the contribution, which, in hindsight, looks simple and straightforward.

2. Relevance. That CPE is a sign of underfitting is relevant, in spite of being simple to deduce. The community has mostly focused so far on CPE as a likelihood/prior misspecification problem. Being fully aware that this is also an underfitting problem is very relevant. Please note that misspecification and DA do not necessarily imply underfitting and CPE (Fig.1b shows how misspecification can also lead to overfitting and WPE; Fig.2e shows how DA can also lead to overfitting and WPE; (Domingos, 2000) also discusses this).

Our paper should be judged by the relevance to the community of the existence of this connection between CPE and underfitting, not by the complexity of the logical reasoning involved in arriving at this connection which, in hindsight, looks simple and straightforward.