Great Minds Think Alike: The Universal Convergence Trend of Input Salience

Response to Reviewer We4a

We thank the reviewer for acknowledging our contribution. Regarding the reviewer's concerns, we clarify that there is some communication in the empirical verifications.

First, we would like to answer the clarification questions:

[Eq. 5 (W4)]

We thank the reviewer for pointing out this typo! As the empirical estimation of $\mu$, $\tilde{\mu}$ is computed as the normalized average of the normalized gradients throughout the experiments. This is the same as how $\mu$ is defined in L152 (where $u$ is the normalized gradient). It is defined as $\tilde{\mu}(k;x)=\frac{1}{M}\sum_{i=1}^M\frac{\nabla_x f_i(x)}{||\nabla_x f_i(x)||})/||\frac{1}{M}\sum_{i=1}^M\frac{\nabla_x f_i(x)}{||\nabla_x f_i(x)||}||\approx \mu(k;x),f_i\in\mathcal{F}(k).$ We will revise it in the manuscript.

[Fig. 3 and 5 (Q1)]

• Fig. 3 presents the results of $\rho(k_1,k_2)$, which is the average of the cosine similarity between two families. This is defined in L110-112 and serves as a starting point for the two hypotheses. The increase in the diagonal motivates and verifies H1. The increase in the rows and columns motivates and (partially) verifies H2 as mentioned by the reviewer.
• Fig. 5 shows the results of the cosine similarity between the averages of the normalized gradients of two families. This is the angle between the population means of two $\mathcal{G}$s. As also mentioned by the reviewer, this verifies H2.

[Gradient Similarities (W1)]

After clarifying the miscommunication, we now address the reviewer's major concerns:

The reviewer's observation of the two possible factors is insightful. Both of them are already considered separately in the manuscript. (ii) is discussed in the intra-family hypothesis. As for (i), there were some miscommunications due to the typos in Eq. 5. The angles between the average of the normalized gradients are exactly what we used to verify H2. The results are presented in Fig. 5, with definitions in the caption as $\mathbb{E}\_{x\in\mathcal{X}}CosSim(\tilde{\mu}\_j(x;\mathcal{F}),\tilde{\mu}_{j'}(x;\mathcal{F}))$. This is the dot product of normalized averages of the normalized gradients suggested by the reviewer. The results demonstrate an extremely high cosine similarity between the means and thus provide verifications to H2.

Furthermore, we reassure the reviewer with the results in Fig. 10. The mean attack is performed by the means estimated using the family of $k=160$ (L291-297). Thus if the concerned scenario is true, the mean attack would not lead to the best results when attacking other families.

[Saw Distribution (W2,W3)]

• The symmetry assumptions (W2). To study the degree of concentration of the gradients, we study the cosine similarity $t = u^T\mu$ between an individual gradient and the mean. This naturally leads to rotationally symmetric distributions since the distribution on the intersection between $S^{d-1}$ and the hyperplane does not affect the distribution of $t$. To resolve the concern, we carry out an empirical study of the distribution on the intersection (i.e. conditioned on $t$). Specifically, we train 1000 CNN models with $k=40$ and seeds 1~1000 on CIFAR-10 and compute $t$ regarding each test sample. The distribution of the first sample is visualized in Fig. R7(left). We partition the range of $t$ to 10 intervals by every 10 percent of the frequency, and inspect the direction of the mean of the gradients in each interval, each direction is estimated by 100 models. If these conditional mean directions are consistent with the population mean direction, then the gradients are symmetrically distributed on each $S^{d-2}$ hypersphere (R7(right)), thus verifying the rotational symmetry. We investigate the cosine similarities between the conditional and unconditional mean directions on the first 1000 samples. The 10*1000 similarity values have a mean and std at approximately 0.970 and 0.013 respectively. Thus the rotational symmetry is empirically verified.
• Marginalization Derivations (W3). The marginal distribution $p_{original}$ in Sec. 3.3 is obtained by taking the marginalization of Saw distribution through integration on the hypersphere. It was not claimed as a theorem or lemma since it is not novel. We will include a detailed derivation in the appendix.

[Gaussian Assumptions (W3, Q2)]

• The Gaussian distribution described in the appendix is not related to the Saw distribution by any means. The only assumption involved in the Saw distribution is the symmetry over the $S^{d-2}$ (e.g. dashed circles in Fig. 4).
• Appendix C serves as a sanity check or a null hypothesis on how to interpret the empirically observed cosine similarities. It is introduced in Sec. 3.1 L215, where the cosine similarities between the population means of different model families are studied. Fig. 5 shows that such cosine similarities are very high. However, the significance of these results may not be well interpreted given that in 2-D/3-D scenarios, this cosine similarity range may not be considered very high. Therefore, the analysis in Appendix C aims to uncover how difficult it is to achieve high cosine similarity as the dimension increases (e.g. Fig. 16).

[Generalizations & Population Means (Q0)]

It can be observed that as the model capacity increases, the distributions of models become more concentrated (illustrated in Fig. 4(a) and verified in Fig. 3,5,6, etc.). This suggests that as a single model approaches the population mean, the testing performance also increases. To verify this claim, we carry out the experiments on deep ensembles since they approach the population mean of the model directly instead of by increasing capacity like single models. It can be found in Fig. 8 that although approaching the population mean differently, deep ensembles and single models have similar trends. This provides support for the connection between the population means and better generalizability.

Response to Reviewer RiRS

We thank the reviewer very much for acknowledging our work! We now answer the reviewer's questions as follows to make sure that all remaining concerns are resolved.

[Figure 1 Clarifications (W1)]

We appreciate the reviewer for pointing out the potential ambiguity of Fig. 1. We clarify that Fig. 1 is for demonstration purposes and to provide a straightforward intuition for the hypotheses and the phenomena. We will update the manuscript to make sure there is no miscommunication. 1.1. The dots are plotted synthetically for a clear visualization. And 1.2. Fig. 1(b) is to illustrate the hypothesis of (i) shared means and (ii) convergence trend compared with Fig. 1(a).

[Model Architecture Clarification (W2)]

We thank the reviewer very much for the valuable point. For a better presentation, We will add examples to elaborate on how model architectures and families are defined, following the ways suggested by the reviewer.

[Definition of $\mathcal{F}$ in L96 (W3)]

L96 aims to define the space $\mathcal{F}$ of functions we focus on in this work. Note that given a fixed input dimension, all models can be seen as a function $f:\mathbb{R}^d\rightarrow\mathbb{R}$, regardless of the architecture, the capacity, etc. However, if the space of models $\mathcal{F}$ is simply defined as all such functions $\\{f\in\mathcal{C}|f:\mathbb{R}^d\rightarrow\mathbb{R}\\}$, many invalid models will be included. Therefore, we add a constraint to the functions in $\mathcal{F}$ that they fit the training distribution well. We will revise the definition to a mathematical formula like $\mathcal{F} = \\{f\in\mathcal{C}(\mathbb{R}^d)|\mathcal{L}(f;\mathcal{X}\_{train}, \mathcal{Y}\_{train}) < \epsilon \\}$.

[Definitions of $\rho,\rho_{ind}$ (W4)]

We thank the reviewer for the suggestion! It can indeed be a little confusing between the notations $\rho$ and $\rho_{ind}$. We clarify them as follows.

4.1. The subscript "ind" in the notation $\rho_{ind}$ refers to "individual". It is defined in L103 as the cosine similarity of two models. Note that the inputs of $\rho_{ind}$ are $f^{(1)},f^{(2)}$, which are two functions (models). Differently, $\rho(k_1,k_2)$ (defined in Eq. 1) represents the expectation of the cosine similarity between two arbitrary models from $\mathcal{F}(k_1)$ and $\mathcal{F}(k_2)$. It takes the width parameters $k_1,k_2$ as the input. Ideally, $\rho$ should be used to study the similarity. And we carry out the experiments by training 100 models for a given $k\in\\{10,20,40,80,160\\}$ for the four model architectures and three datasets. This already results in $100\times 5\times 4\times 3=6000$ trained models. To demonstrate the influence of the capacity $k$ in a higher resolution, we investigate $k\in\\{8, 10, 12, 14, 16, 20,\cdots, 384, 448\\}$ (L106). This is computationally infeasible for $\rho$. Therefore, we compare the results between $\rho_{ind}$ and $\rho$ in Fig. 6 to verify that $\rho_{ind}$ serves as a computationally efficient surrogate for $\rho$.

4.2. We will term $\rho_{ind}$ and $\rho$ clearly as "individual similarity" and "average similarity" to avoid ambiguity. And as suggested by the reviewer, we will move the definition of $\rho_{ind}$ (L103) and $\rho$ (Eq. 1) together for a better presentation.

4.3. Originally, Fig. 2 serves as the purpose of motivating the problem and appears before the definition of $\rho, \rho_{ind}$. Thus $\rho_{ind}$ was not used in the caption. As suggested by the reviewer, we will re-arrange the figure and text to include $\rho_{ind}$ in the caption of Fig. 2 for a better presentation.

4.4. We appreciate the reviewer for pointing this out. We will revise the labels in Fig. 2 and 3.

[$\mathbf{\mu}(k;\mathbf{x})$ in H1 (W5)]

Thanks for pointing this out! $\mathbf{\mu}(k;\mathbf{x})$ refers to the normalized population mean of the gradient, which is defined in L152. We will re-arrange this part and move the definition to "The Intra-Family Hypothesis." for a better presentation.

[$k_1,k_2$ in L158]

Note that in L158, we are discussing the cross-family similarity regarding the population mean of each family. Since it is studied through the cosine similarity, it is symmetric between $k_1$ and $k_2$. Note that for cross-family similarity, $k_1\ne k_2$. Therefore, the condition $k_1>k_2$ is equivalent to the condition $k_1\ne k_2$ in H2. We will revise the manuscript to make it consistent throughout the discussion.

[Depths and Widths (Q1)]

We appreciate the valuable suggestion from the reviewer. We acknowledge that the depth is also a factor that affects the model capacity. However, since the depth change usually leads to inconsistency in widths, we study the influence of depths by the -small and -large suffixes in the manuscript.

To address the reviewer's question regarding the influence of depths completely, we carry out additional experiments with different settings so that depths and widths can be altered independently by parameters $d$ and $k$, respectively. $d$ determinse the number of layers, and $k$ determines the width of the layers. Unlike the gradually increasing channel (e.g. $[k,2k,4k,8k]$) in the manuscript, here we set the same number of channels for all layers. In this way, the change in the number of layers does not affect the widths anymore. For example, when $d=3$, the layers are $[k, k, k]$. The results are shown in Fig. R1 of the attached PDF file.

It can be found that (1) Given a fixed depth or width, the influence of the other factor is similar when scaled up. (2) Depths are slightly different from widths. Larger widths lead to higher similarities, while closer structures in depths have higher similarities. This is verified by higher similarities near the diagonal entries for Fig. R1 (Right) compared with Fig. R1 (Left).

Response to Reviewer fWRt

We appreciate the reviewer very much for the acknowledgment of our work. To further strengthen the reviewer's confidence in our findings, we address the reviewer's remaining questions as follows.

[Fixed Initializations with Random (W1)]

We thank the reviewer for the suggestion of separating the stochasticity of random initialization of the model weights and the stochasticity of the randomization of batch orders in SGD.

In order to address the reviewer's concern, we carry out additional experiments, where models are all initialized with the same weights by setting seed = 0 before the initialization. After the initialization, we set again manually to 1~100 in the training process. The experiments are carried out using CNNSmall models with $k=20$ on CIFAR-10. It should be noted that this is actually equivalent to uniformly sampling from the original distributions. Therefore, we expect an almost identical distribution compared with the models used in the manuscript, where seeds are determined at the beginning of all steps. Since the normalized gradients $\mathbf{u}$ are of 3072 dimensions, it is infeasible to compare the distribution directly. To verify the uniformity, we inspect $t = \mathbf{u}^T\mathbf{\mu}(\mathbf{x})$ described in L177-190 of the manuscript. This marginal distribution is 1-D on $[-1,1]$ and we study the Wasserstein distance between the distribution of models with identical (random) initialization and the distribution of models with different (random) initializations. The distance is approximately $0.0225$, which means that the distributions are very close in terms of the dispersion degree.

[Other Sources of Stochasticity (W2, Q1)]

We agree with the reviewer that different training criteria such as the learning rates, the batch sizes, the solvers, etc. can affect the resulting models. However, it should be noted that a criterion determines a distribution of trained models. And the phenomenon discovered in this work holds for all kinds of criteria instead of the one used in the manuscript.

To completely resolve the reviewer's concern, we carry out additional experiments to investigate the influence of different training options, including learning rates, batch sizes, solvers, and data augmentations. The experiments are described in the global response in detail, and the results are presented in the attached pdf file.

[Other Data Types]

We thank the reviewer very much for the question regarding other data types. We clarify that text data is not commonly studied in the literature on benign overfitting. Additionally, conventional NLP training is typically conducted in an unsupervised manner, making it difficult to carry out similar experiments. However, since NLP data are now often trained using transformers, we investigate whether the transformer and attention mechanism exhibit this property as a starting point.

Specifically, we train vision transformer (ViT) models on CIFAR-10 with varying capacities controlled by $k\in \\{10, 20, 40, 80\\}$. CIFAR-10 has an input size of $32\times 32$ pixels, thus we set patch size to be $4\times 4$, resulting in $8\times 8$ patches. We set the embedding size to $4k$, divided by $k/2$ heads, and we set the depth to $8$. We then vary seeds in 1~100 and train 100 models of each $k$ and study the mean of the similarity $\rho(k_1,k_2)$ (i.e. the same experiments as Fig. 3 in the manuscript) and the similarity of the population mean (i.e. the same as Fig. 5 in the manuscript). The results are shown in Fig. R6. It can be observed that although distinct from convolutional layers, the transformer structure also has the discovered convergence trend. It can also be noted that the degree of dispersion of ViTs is much higher than CNNs.

Therefore, based on the additional experiments on transformers, we can deduce that the convergence trend may also hold for NLP tasks where transformers are extensively used.

Response to Reviewer YpLK

We thank the reviewer for the invaluable feedback. We address the questions as follows.

[Novelty (W1)]

We summarize the work in [5,6] and how the novel contributions of our work on deep ensembles differ from them. It should also be noted that exploring the mechanism of deep ensembles is not our main contribution, but an application and verification of our work.

• [5] studies the failure modes of deep ensembles and. However, [5] focuses on empirical observations of the performance of ensembles. This is never claimed as our contribution. Note that the comparison between the performance of single models and ensembles is not novel and can be found in many works discussed in our manuscript (e.g. [7]). However, the reason behind deep ensembles' performance gain is still mysterious [5]. Our work hypothesizes and verifies that (1) single models' performance gain is related to the convergence trend, and (2) the convergence is towards the shared mean of all capacities. This provides a novel perspective to understand deep ensembles and why an ensemble of many small models can achieve the performance of a single large model.
• [6] studies the double descent phase of DNNs using NTKs. The analysis based on NTK verifies similar results for deep ensembles as in [5]. As a result, the novelties of our work discussed above still hold. The similarity of mechanisms between models with different capacities and how the convergence is towards the shared population means across capacities are also novel compared with existing work. Additionally, NTK is a powerful tool yet with limitations and strong assumptions. The analysis is on the global loss regarding the data distribution (Eq. 4 in [6]). However, our results are on the direct prediction of any single input and its neighbors.

We will add these works to the related work section for a more comprehensive review of existing work.

[Experiment Setups (W2)]

To stay consistent across experiments, we follow the setups of [1] as claimed in L195-197. We use SGD solver with a learning rate $\gamma = 0.1/\sqrt{1+epoch}$. No weight decay, momentum, or data augmentation are included to minimize variants. The batch size is 128. All models are trained for 200 epochs.

[Sources of Stochasticity (W3, Q1, Q2)]

Training schemes do not affect the discovery. Please refer to the global response for details, including comprehensive additional experiments.

[Scale of Experiments (W4)]

This work is mostly based on the benign overfitting and double descent phenomena (e.g. [1-4]), where large models counterintuitively do not exacerbate overfitting. We aim to provide insights into this mysterious capability of DNNs from the perspective of XAI. Hence the settings of our work are mostly based on existing studies on this topic, too.

The complexity of the datasets used in our experiments is a strength, not a weakness. Unlike existing works, which mainly use CIFAR (e.g. [1]), we test our hypotheses on TinyImageNet, which includes 200 classes and 110k samples, offering a more complex and realistic data distribution. The same trends observed in CIFAR datasets are also evident in TinyImageNet, suggesting these trends will hold for even more complex datasets like ILSVRC.

We admit that carrying the experiments over ILSVRC can be infeasible for the studies of benign overfitting. As explained in Appendix B, with $k=64$, ResNetLarge is already equivalent to ResNet-18. We scale $k$ up to 448, which makes it impossible to train a group of models to estimate the empirical mean. For example, in [4], a study of overfitting carried out experiments on CIFAR and SVHN.

[Deep Ensemles and Single Models (Q4)]

As the capacity increases, the distribution becomes more concentrated (illustrated in Fig. 4(a), verified in Fig. 3,5,6, etc.). This suggests that as a model approaches the population mean, the performance increases. To verify this, we carry out the experiments on deep ensembles since they approach the population mean of the model directly instead of by increasing capacity like single models. Fig. 8 shows that although approaching the mean differently, deep ensembles and single models have similar trends. This provides support for the connection between the population means and better generalizability.

[Clarifications]

• Q3 [ResNet]. ResNets have higher expressiveness compared with CNNs and rely heavily on data augmentations. Since model performance is beyond our scope, data augmentation is not included. Fig. R5 shows additional results of ResNets with data augmentations. It improves the performance of ResNets greatly. However, the convergence trend is not affected. NLL is also affected by confidence. ResNets tend to be more confident in wrong predictions, leading to higher losses.
• Q5 [Input Salience]. Input salience is usually used by the XAI community to refer to the input gradient as a saliency map, and also used to refer to attribution-based methods. We will formally introduce the definition of the concept.
• Q6 [Logit]. Logits refer to the prediction of the target class before softmax. We also include the results of the post-softmax output in Appendix A. It shows that this does not affect the results. Note that taking the gradient of the output probability of the target class is equivalent to the gradient regarding the negative log-likelihood, normalized by the probability.
• Q7 [$u_1,u_2$]. We will use $u_1,u_2$ to refer to gradients of different sets and $u,v$ to refer to those of the same sets.
• Q8 [$\mu(k;x)$]. We will move the definition of $\mu(k;x)$ to H1 for a better presentation.
• Q9 [$\rho_{ind}$]. $\rho$ is defined in eq. (1). It refers to the similarity between models of capacity $k$ and is approximated by taking the average over 100 models each. $\rho_{ind}$ refers to the model-dependent similarity computed regarding the two models (in L233). We will define it more clearly to avoid ambiguity.