Neglected Hessian component explains mysteries in sharpness regularization

We thank the reviewer for their questions, and answer some here.

The empirical results only include ResNets. May the empirical conclusion also depend on model architectures? How about the results of very simple models (FCN/LR) and very complex models (Transformers)? Note that they have very different loss landscape.

We agree that studying more architectures and datasets would be interesting; for this initial work we focused on very detailed experiments in our chosen settings rather than shallow experiments across a broad range of settings.

Training with weight noises only implicitly penalize the Hessian. This work often abuse Hessian penalties and weight noises.

In Sections 5.1 and 5.2, we review the links made in the literature between weight noise and Hessian penalties in order to frame our work. However, in all our experiments we make the procedure clear, and mainly focus on explicit regularization of the Hessian/Gauss Newton trace.

This work provides insights but failed to show how to improve or design better second-order-based methods. This can further support the conclusion of this work.

We agree that additional research is needed to further improve second order methods/regularizers; however our work showed that the Gauss Newton trace is another useful regularizer. In addition, our synthetic second derivative approach in Section 4.5 shows how we can overcome the poor second derivatives of ReLU in a practical, efficient way.

We thank the reviewer for their comments, and will fix the errors noted. We address some selected concerns below.

In terms of the use of second derivatives the manuscript sends a bit of a mixed message.

The overall point about second derivatives is a bit subtle; our work suggests that second derivatives in the update rule are helpful (get full NME information in HVP). In contrast, the full Hessian trace penalty has second derivative information in the regularizer. Minimizing this information during optimization reduces the effect of second derivatives in the update rule. It also introduces third derivatives into the update rule; these higher order derivatives in the updates seem to hurt training. It remains an open question whether or not other forms of higher order derivatives are useful.

143: Should this be SGD (there are not batches to be seen) or a normal gradient descent?

We wrote the rule for a single batch; in practice this update rule would be combined with batching/SGD.

226: "Minimizing the NME is detrimental" Shouldn't this state that "ignoring" the NME is a problem?

This should read “Minimizing the NME trace” is detrimental, as the GN trace penalty (no NME in the regularizer) works well.

We would like to thank the reviewer for their insightful review. We address the reviewer's questions and suggestions below.

How do you solve SAM? Have you also neglected the second-order term in your empirical analysis? What would be the effect if this term were not neglected in your situation?

Our experimental results for SAM are based on the SAM algorithm proposed by Foret et al. Specifically, the SAM update does not utilize any Hessian (but does evaluate an extra gradient). The reason the Hessian is absent from the SAM update is due to the approximation employed by Foret et al (using stop_gradient on the adversarial perturbation). Therefore, our implementation of SAM, which aligns with that of Foret et al, indeed neglects the Hessian. We have not investigated the implications of retaining the Hessian in the SAM formulation because this was already explored by Foret et al. They conducted this experiment and observed that by keeping the Hessian, SAM's performance slightly decreases compared to when the Hessian is neglected (which is consistent with our story). This is discussed in Figure 4 of their paper.

To derive PSAM from SAM Equation 6, we employ a first-order approximation as shown in Equations 9-11. This differs from the approximation used in the original SAM algorithm [Foret et al., 2021]. While exploring a second-order approximation could be a promising avenue, it falls outside the scope of this current work.

Could you provide some demonstrations regarding the off-diagonal elements in the NME? It is suggested that these elements may also play an important role.

The experiments in Section 4.4 show that the off-diagonal elements of the NME by themselves are not sufficient to obtain good results with PSAM. In these experiments we artificially zero out the diagonal elements of the NME of GELU. It would be interesting to see how well PSAM with ONLY the diagonal elements performs, but we did not have time in the rebuttal period to design and run this experiment.

I recommend that the theoretical analysis be expressed in a more formal and clear style.

Thank you for your suggestion. We will improve the style of the theoretical analysis and address the typos you identified.

We thank the reviewer for their thorough review. We address the reviewer's questions and suggestions below.

In section 2 when deriving the NME, can you include an explicit example where the NME is large? Either analytically or a plot demonstrating the evolution of, e.g., its Frobenius norm over the course of training.

The suggestion to identify specific examples where the NME is large is valuable. We are actively investigating this, but due to time constraints in preparing the rebuttal we don’t have definite results yet.

small thing: perhaps when you mention the 3 datasets, reverse the order to have them in increasing order of difficulty

We will change the order as you suggested.

one thing you do not discuss but I would want to know more about is — PSAM method only should work when rho is small. And indeed, when rho is small, even for ReLU the methods match. Of course, since the GeLU has the match even at the values of rho where there is a gap for ReLU, it seems that the size of rho is not the determining factor. But could you still comment on how to know whether the issue is just the size of rho?

Further evidence that the size of rho isn’t the sole factor can be found in Figure 3, which shows the gap for ReLU can be narrowed even at larger rho values by using a synthetic activation NME.