Variational Learning Induces Adaptive Label Smoothing

We thank the reviewer for their review. The main issue seems to be related to the writing of the paper, for which we have given clarifications and promised a few fixes. We hope that the reviewer will consider raising their score.

Q1: I think the conclusions in Section 3.4 lack credibility. Reducing the expectation merely to the sampling of a single instance is unreasonable. The authors should provide results for multiple samples.

A1: There is a misunderstanding. In Sec 3.4, we show a single sample for simplicity but state that multiple samples are also possible (see line 244). In fact, in our experiments, we do use multiple samples. Hope this clarifies that there is no issue with credibility.

Q2 It would be preferable for the authors to apply some statistical methods to provide corresponding error estimates.

A2: Could the reviewer clarify what they mean by “error estimates”?

Q3: While for Thm.2, the author does not give a proof, which I think should be added.

A3: Thanks. We will expand the proof and make it easier to understand.

Q4: The issue of this paper lies in its writing logic.. The writing is confusing…Since Section 3.2 is a generalization of Section 3.1 and both follow the same approach, why not provide a detailed proof for Section 3.2 while treating Section 3.1 as a special case?

A4: It is an interesting suggestion, we will try to revise it with your suggestions.

Q5: Shouldn’t the target predicted by fi be the 0/1 labels?

A5: No. The $f_i \in \mathbb{R}$ takes real values, while the sigmoid $\sigma(f_i) \in [0,1]$ takes values between 0 and 1. None of them are 0 or 1.

Q6: Why does a symmetric situation arise?

A6: This is because we plot the absolute value of the noise $|\epsilon_i|$, which is the magnitude of the noise for clear visualization.

Q7: I did not understand the discussion after Thm 1 (lines 145-183).... Please provide a detailed explanation, as I do not understand the purpose, conclusion, or the meaning conveyed by Figure 2 in this section.

A7: This part explains the “adaptive” nature of the noise. Eq. 12 gives the exact expression for the noise and Fig. 2 visualizes it as the function of the posterior mean. We can rewrite this to make it clearer.

Q8: What is the role of Section 3.3? How does it help in deriving the conclusion of Section 3.4?

A8: Section 3.3 generalizes Section 3.2 to show that Newton's method provides more adaptive label smoothing. There is no direct connection with Section 3.4.

Q9: The proof appears to heavily rely on the linear term −yf(x) in the loss function. Does the result hold for other loss functions as well? Can the class of loss functions discussed by the authors encompass common loss functions such as cross-entropy loss? What effect do they have on different losses (such as different formats in exponential-family loss and some other loss functions beyond exponential-family loss)?

A9: Yes, the results hold for other loss functions including the cross entropy loss as explained in line 205. Please note that the linear term -y f(x) is not a limitation. All exponential-family losses have this term and most of the loss functions used in deep learning are derived from exponential-family losses.

We thank the reviewer for their review. As per reviewer's suggestion, we have made the code available at the following anonymous link. We also thank the reviewer for suggestions regarding writing. We will fix these in the next version. Below, we respond to other comments.

Q1: The authors… do not include the important paper [by Wei et al. 2022].

A1: Thanks for this. We will make sure to cite it in the next version.

Q2: [The authors] should include comparisons with other adaptive label smoothing techniques that are cited in the paper, like Ko et al. (2023) and Park et al. (2023).

A2: We did not compare to them because these papers are similar to Zhang et al. (this is stated in line 56-62) which we compared to in Fig. 4. We will add these other works but do not expect the results to change due to the similarity of these methods.

Q3: Is there a motivation why $q(\theta)$ is set to a Gaussian?

A3: There is no specific reason but the Gaussian case is simpler to explain. As shown in Fig. 1, the weight uncertainty in variational learning is expected to improve overconfidence in the predictions. The result is valid for all sorts of variational distributional forms.

Q4: Is there some reference defining what variational learning is?

A4: We will add a citation of this work: D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859–877, 2017.

Q5: Can you report all the steps for the proof of Theorem 1?

A5: Yes, we will fix this in the camera ready version.

Q6: Why in the proof of Theorem 1 do you write that the gradient of the KL simplifies to an equation which is instead reported to be the KL itself? Do you mean that the gradient of the KL is the gradient of the squared norm of ? How do you use the gradient of the KL in the proof?

A6: Perhaps the sentence is confusing. What we mean is that the expression for KL is simply the squared norm, whose gradient takes a simple form.

We thank the reviewer for their review. The reviewer found that “the extension to deep neural network part is unclear and not convincing” and also that the assumption on Gaussian variational posterior to be “slightly over-claim”. We do not agree with the reviewer's assessment and challenge it below.

First of all, the suggestion that the result “may not hold for other kinds of variational posteriors” is incorrect. We consider the Gaussian case because it is easy to show these connections, not because the calculation of KL divergence simplifies. In fact, KL divergence for any exponential family has a closed-form expression; e.g., see Page 15 onwards in Niesen and Gacia for such a list. In general, weight-uncertainty (or perturbation) in variational learning is expected to have a positive effect on the overconfidence of the label. Our paper shows cases where the connection is clear and it is not an overclaim.

Second, the reviewer also suggests that our empirical evaluations are not “apple-to-apple” comparisons because we compare variational learning to point estimates. This does not make much sense to us because such comparisons are routinely used in Bayesian deep learning, where the performance of the mean of the Gaussian posterior is compared to the point estimates. We request the reviewer to reconsider their assessment.

Third, the reviewer is critical of the analysis in Sec 3.4 calling it “too rough in terms of approximation”. Specifically, they dismiss the “approximation of the gradient at line 253” because it is a zeroth order approximation. This is not true because the result is simply a consequence of the first-order approximation, as shown below. $f_i(\theta) \approx f_i(\theta_t) + \nabla f_i(\theta_t) (\theta - \theta_t) \implies \nabla f_i(\theta) \approx \nabla f_i(\theta_t)$

Finally, the reviewer dismisses the experiments stating that “variational learning is often more robust than point-estimates” but they ignore the fact that we are comparing a point estimate with label smoothing to variational learning without an explicit label smoothing. For some reason, the reviewer then claims this to be “confounding” stating that “it is not clear if the superior performance is due to adaptivity label smoothing or variational inference”. This makes little sense to us because our main result is to show that variational learning induces the adaptive label smoothing. These two things are the same thing and there is no issue of one confounding the other.

We would also like to say that “atypical” is not a typo (as pointed out by the reviewer). We really mean to say that the higher noise is assigned to examples are not typical.

We thank the reviewer for their review. The reviewer finds that the work “may not be sufficient for an ICML-level publication” due to lack of “deeper theoretical insights” and “broader empirical validation”. We do not agree with the assessment. Our connections between variational learning and label smoothing are entirely new and precisely derived for generalized linear models. Through approximations, we show that they are expected to hold for neural networks as well (Sec 3.4) which is further validated empirically.

The reviewer has asked to include “other tasks such as out-of-distribution (OOD) detection and calibration” and “more recent baselines”, but we do not think that these add much value. This is because we are not claiming that variational learning induces a better smoothing. Instead, the goal of the experiments is to validate the approximations used in Sec 3.4 and confirm the induced nature of adaptive label-smoothing of variational learning. This is why we show similarities to Zhang et al.’s method in Fig. 4.

The reviewer also suggested “to incorporate additional classification benchmarks, such as Stanford Flowers” but they did not provide an exact source for this dataset and we were unable to find it by ourselves. We will be happy to include it if the reviewer can provide a citation or link.