On Gradient-Weight Alignment

Thank you for your review of our submission and your feedback.

Your criticism is voiced assertively. While you raise some valid points which we are grateful for, we feel that we have to push back on a few aspects.

“[...] a promising position is to investigate how to use the estimated directional alignment and dispersion to determine an early stopping time without relying on any information from validation sets so that the trained DNN is more robust with respect to different validation sets in the downstream applications. The results presented for this direction so far are limited and not convincing enough. The authors are encouraged to continue along this direction.”

We agree that this proposed direction is of particular interest and a key motivator of our work. However, the purpose of our work is not solely to introduce a novel early stopping criterion but to find a measure that inherently links prediction performance to the training samples used during optimization. We thus see the evaluation of sample memorization and sample mislabeling in Section 4.2 and 4.3 as crucial for the reader to understand what GWA is measuring and how to better understand the metric for use in practice.

“[...] in Figure 3, the blue curve on the right sub-figure seems to be validation error rate, not validation accuracy. In addition, why not put the three curves (the validation accuracy curve, directional alignment curve, and directional dispersion curve) into one sub-figure?”

Thanks for catching the mistake, we will fix it. Regarding combining different figures, we have considered this but kept them separate for improved clarity.

“The observations from Figures 3 and 4 are not conclusive, and vague.”

The figures show the behavior described in line 258 and following, which is characterized by an increase in directional alignment when learning general features initially, and either a decrease or increase of directional dispersion after reaching peak alignment (with the later being associated with overfitting). As this is consistent across all experiments we would be happy to better understand which observations you are referring to.

“Contradicting to the statement made in Lines -3 and -2 on Page 6 (bottom of Page 6), the rightmost sub-figure in Figure 4 does not show any negative directional alignment.“

We agree that this is not worded properly and will update this. The focus should be on both directional alignment and dispersion staying consistently around 0 without showing any sign of learning useful information.

“Figures 3 and 4 contradict to Figure 5. Putting three together, one cannot conclude that the directional alignment curve tracks the validation accuracy closely.”

The main finding of Figure 5 is not how well directional alignment tracks validation accuracy but how well it is correlated to stiffness and how the two change during training. It thus does not contradict Figure 3 and 4, in fact, Figure 5 shows similar behavior to what we can observe e.g. in Figure 3 right. We will still remove the last phrase in the caption to avoid any confusion here.

Thank you for your responses. My concerns expressed in the weakness section remain the same. It is easy for the reader to understand your proposed metrics: the directional alignment and dispersion. The issue, however, lies in how to utilize the proposed metrics to derive something good, which is lacking at this point. I hope my suggestion can help you improve your future position and research along this line.

Thank you for your reviewing and the feedback for our submission.

“The paper provides many evaluations on C10 and C100 image classification tasks, which are pretty narrow analyses for evaluation generalization purposes; more experiments on ImageNet-1k would be beneficial.”

We focused on CIFAR-10/100 as we can use versions of the same dataset for showing a variety of behaviors (from compute intensive related work such as stiffness, to using memorization scores from Zhang et al., and labeling errors in CIFAR-N). However, we fully agree that more experiments on ImageNet-1k will strengthen our findings and will extend our analysis in an updated version, especially for Table 1.

“[...] will these metrics work on other tasks, such as object detection or segmentation.”

Yes, as long as we are able to compute a form of per-sample gradients, the method is applicable. Do you have any specific datasets in mind that you would be interested to see / that you think could be challenging?

“How do the proposed methods perform differently with those works? It would be good to compare the performance prediction ability by showing the correlation on specific NAS benchmarks [...].”

The main difference is that our metric is not zero-shot and still requires training the network - at least for a few epochs. This does seem like an interesting application though, and we will look into using GWA in this domain.

Thank you for your feedback on our submission.

“The idea of measuring gradient similarity is not new. Although this paper considers the alignment between gradients and weights, which is different from existing methods, it’s not clear why the former is superior to the latter in terms of generalization estimation. Regarding memory cost and computational complexity, I think some existing methods, such as Stiffness, can also be extended to a faster stochastic version, similar to the operation in Algorithm 1, line 9.”

While the idea of gradient similarity measures is not new - in fact it is the base of our work (see line 126) - these measures are so far not widely used outside the respective papers introducing them. This is mainly due to the substantial compute overhead associated with these former approaches. And while we would be happy to see extensions of previous methods with our proposed estimator, computing alignment scores based on pairwise similarity in comparison to a reference vector, such as the weights, will inherently always require more compute and introduce additional stochasticity.

“Apart from empirical evaluations, it would be better to see more insightful analyses of GWA’s effectiveness.”

We would be grateful if you could elaborate more on what you mean by measuring the effectiveness of GWA.

“[...] only a validation set is used as the baseline. Other popular metrics that do not rely on a validation set should also be evaluated. Additionally, the reported accuracies on both CIFAR-10 and CIFAR-100 are too low.”

Do you have any particular metrics in mind here? As mentioned in line 97, we consider sharpness as one of the most popular alternatives. However, not only does it have shortcomings discussed in Section 2 but we also do not see any additional benefit to using it in comparison to the more informative validation accuracy. We would be happy if you could elaborate more on the potential benefit here.

Regarding the accuracies on CIFAR-10/100, this is only for the ResNet-based models which need to be adapted to use group normalization in comparison to batch norm to compute per-sample gradients. We will look into changing these with inherently batch-free methods in an updated version.

“This could be evaluated in a more rigorous way, such as through quantitative analyses comparing model selection using GWA with baseline algorithms.”

Are there any specific experiments you would be interested to see? In particular delimited from the results in Table 1.

“The discussion in lines 412-416 seems unclear, as alignment is naturally easier when only a subset is used to train the model (making fitting easier).”

Could you please elaborate on this point? We use the whole dataset throughout training, only the model is influenced by different subsets of that dataset throughout training as part of the intrinsic bias introduced by SGD. In addition, we are not sure why alignment should be easier on a subset in general, as alignment is inherently based on similarity of data and not the dataset size.

“The second and third paragraphs in Section 4.4 lack clear logic. They initially suggest that directional alignment is a good predictor of generalization but then recommend using directional dispersion rather than alignment to decide when to stop training.”

Thanks for the feedback, we will focus on improving clarity here.

Thank you for reading the paper and reviewing our submission.

“Why does a good alignment of training sample gradients indicate good generalization?”

We suspect you mean alignment among gradients here. As mentioned in line 130 and following, gradients being similar in their direction across samples indicate that the weights need to change in the same manner for the loss to be reduced for all of these samples. Consequently, the features learned by optimizing with regard to these gradients are general.

“[...] intuition to use alignment between per-sample gradients and model weights?”

As shown in the work by Ji and Telgarsky (see line 147), the orientation, not the magnitude, of the weights is in theory the key property that describes the prediction capabilities of a model. Intuitively, taking the alignment between the gradient and weights allows us to capture this property, i.e. the change in orientation of the weights required to better predict a specific sample.

“[...] what if the training samples are noisy [...]”

Uncorrelated sample specific noise is theoretically orthogonal to the direction of general (useful) gradients. In practice, the impact (positive or negative) of noise is highly dependent on the type of noise (see e.g. [1]). The level of noise is thus reflected by samples being more or less aligned, not only among all gradients as shown in [2] but also with regard to the weight vector (as shown empirically by us in e.g. Section 4.2).

“[...] unclear what properties the lightweight GWA estimator satisfies.” and “[...] unclear whether the estimator has desirable properties such as being unbiasedness.”

Our estimator is based on the usual assumption of IID sampling and mini-batches being valid empirical estimates of the dataset statistics and the convergence of individual iterates (line 201). We will investigate this formally and will provide a proof when available.

[1] Chen et al., Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise, AAAI 2021.

[2] Fort et al., Stiffness: A new perspective on generalization in neural networks, arxiv.org/abs/1901.09491.