Untrained Networks' Class Bias: A Theoretical Investigation

Thank you for the responses. I do believe that the work has substantially improved since the very first version that I saw. With a bit more work, I think the authors are on track for a potential high-impact publication. I personally think it is worth putting in that effort rather than rushing a publication of limited impact. The current version of the manuscript seems ready for an ICLR workshop or a more specialised conference, which might be a good way for the authors to create some attention and engagement. Ultimately, I would be very excited to read a strong version of this paper (including a nice set of empirical results). I will raise my score accordingly, but would still rate the paper as marginally below the threshold for ICLR (which, admittedly, is very high).

Re: illustration of IGB vs. beneficial/detrimental. Thanks for clarifying - I had definitely read the paper as advocating for designing network architectures that are free from IGB (which does seem to make "intuitive" sense). Clarifying early and very explicitly in the paper that this is not the case would be very helpful. Apologies for the misintrepretation.

Re: quantifying IGB with a single value. Yes, thanks for pointing this out. Another reason to see if the IGB criterion can be formulated more generally. (The entropic formulation might also translate fairly straightforwardly to regression tasks, which is not true for the classification criterion of course) Since the current IGB criterion requires collecting lots of class assignments across datapoints and weights, experimentally estimating the conditional entropy by collecting logit vectors in addition to class assignments should only lead to minimal extra effort.

To avoid redundancy, we refer the reviewer to the general response, to which we'll now add further clarifications addressing more specific doubts.

Reply to the suggestion of the entropy formulation

We thank the reviewer for the insightful and helpful comment. We would like to take this opportunity to clarify a fundamental point. Through our simulations, our aim was to illustrate that the presence of IGB leads to a qualitatively different dynamic. It's not our intention to assert whether IGB is beneficial or detrimental; the answer to this question is entirely non-trivial and potentially problem specific.

We find the idea of formulating this in terms of entropy quite intriguing. An additional point not highlighted by the reviewer is that it would enable us to associate a functional with the distribution, allowing quantification of the level of IGB with a single value.

To avoid redundancy, we refer the reviewer to the general response, to which we'll now add further clarifications addressing more specific doubts.

Reply to W.1

• b) Yes, the interpretation of the reviewer is correct. $\mathbb{P}\_{\chi}(O^{(c)} | \mathcal{W})$ is a Gaussian distribution, or equivalently, the random variable $O^{(c)}$, conditioned on the fixed realization of the weights $\mathcal{W}$, is Gaussian distributed. We appreciate the reviewer's comment and, in the new version, we will opt for this second formulation to make the point clearer, i.e. $O^{(c)}_{| \mathcal{W}} \sim \dots$
• c) Again, the interpretation is correct: $\mathcal{N} (y \text{;} \Delta_{\mu} (f_0), 2 \sigma^2_{\infty})$ indicates a probability density function that we integrate. The used notation is extensively detailed in App. A. We will provide additional references that connect the various sections to the appendix throughout the text.
• d) In the paragraph 'Permutation Symmetry Breaking: the foundation of IGB,' we introduce the quantity $f_c$ by defining it as 'the fraction of datapoints classified as class $c$.' This definition applies to both finite and infinite datasets. The fraction, in general, is computed over the datapoints that comprise the dataset (not an expectation over the input distribution). However, we are particularly interested in the limit of datapoints approaching infinity, in which the fraction indeed converges to the value averaged over the input distribution.
  Considering the limit of infinite datapoints, we can then connect the quantity $f_c$ to the output distribution, derived in our analysis, through the law of large numbers (Eq.1).
  The datasets used in practice, although large, are clearly of finite size; hence, we encounter finite-size effects that perturb the asymptotic prediction. Nevertheless, this does not pose an obstacle in terms of predictability. Our analysis is indeed capable of effectively describing these effects by accounting for the aforementioned corrections (as in Appendix C.1 (linear case)). Various plots in the paper display these corrections. For instance, the theoretical curve overlaid on the histogram of the linear case in Fig. 5 accounts for finite-size effects, which, if absent, would lead the distribution to converge to a spike at 0.5.
  We kindly note that the notation is indeed grouped in App. A for ease of reference; however, we will follow the reviewer's advice and provide a more rigorous definition of the introduced quantities. If there are further unclear points in this regard, please, do not hesitate to highlight them. We would be more than happy to attempt to clarify.

Reply to W.3

We take advantage of this observation to clarify a fundamental point in our discussion that might have been misunderstood. It is not our objective to propose a narrative where IGB is unequivocally associated with a harmful effect. The article's objective is rather to highlight the presence of a non-trivial phenomenon and understand its nature through a rigorous analysis. We agree that the question of whether IGB is a desirable effect or not is inherently interesting; however, providing a complete answer to this question requires a dedicated, systematic study since the response is neither simple nor absolute (it might depend on the problem's nature whether the presence of IGB constitutes an advantage or an obstacle). From our standpoint, what our experiments aim to show is that certainly the presence or absence of IGB qualitatively modifies the dynamics (for example, by creating differences in the statistics of individual classes, which are not present in the absence of IGB). We focus on certain regimes that can naturally occur, for instance, during hyper-parameter tuning procedures or sometimes assumed as hypotheses (the assumption of a low learning rate is, for example, a fundamental assumption in many works). In such regimes, the difference between the two compared cases is evident.

The analysis with MaxPooling helps to illustrate, with a clear example, how the presence of an architectural element of this kind, by amplifying the asymmetry between nodes, in turn amplifies the presence of IGB. This allows us to understand, within the framework of our theory, where a profile similar to that observed experimentally (for example, Fig. 1 of the paper) originates.

Reply to W.4

We would like to point out that the experiments in Fig. 15 do not refer to a ResNet but to a CNN. However, as mentioned in the response to the above point, the message these experiments aim to convey does not relate to a comparison of the performances achieved in the two cases. Instead, we aim at highlighting differences that make the two dynamics qualitatively distinct. In the new version, we will be clearer about this point to avoid confusion.

To avoid redundancy, we refer the reviewer to the general response, to which we'll now add further clarifications addressing more specific doubts.

Reply to W.1

$\mathcal{W}$ indicates the entire set of weights. The weights are then organized into matrices that connect the various layers. In equations 7 and 8, we have expressed, for compactness reasons, the limit in terms of the divergence of the total number of parameters (cardinality of the set $\mathcal{W}$). As noted by the reviewer, and as we ourselves acknowledge in the text, this limit is ambiguous; we opted for this choice in order to maintain compactness and not burden the introduction of the phenomenon with technical details, which we have instead moved to more advanced sections of the document. Formally, for the single hidden layer network, the conditions we require (for the use of the Central Limit Theorem in our analysis) are $d \rightarrow \infty$ and$N_1 \rightarrow \infty$.

Reply to W.2

We appreciate the comments provided by the reviewer and acknowledge the need for a more rigorous treatment of the independence of the components in the given representation. In response to this feedback, we have rephrased the independence argumentation for clarity and rigor. Below, we present a concise summary of the key steps of the revised proof that will replace the arguments in Remark 1 in the updated manuscript. Consider the pre-activated nodes, $h^{(1)}\_{i} = \\sum\_j w^{(0)}\_{ij} \xi_j \\, .$ Given the independence of $\boldsymbol{\xi}$ components, we can employ the Central Limit Theorem (CLT) to derive the distribution of $h^{(1)}\_{i}$.

Since we also utilize the CLT for $O^{(c)} \left( \boldsymbol{\xi} ; \mathcal{W} \right) = \sum\_{m=1}^{N\_1} w^{(1)}\_{cm} g^{(1)}\_{m} \\, ,$ independence must also be established for the set $\\{ g^{(1)}\_{m} \\}$. As the activation function is an element-wise transformation of the set $\\{ h^{(1)}\_{m} \\}$, it suffices to ensure the independence of the latter. A well-known fact about jointly normally distributed random variables is that they are independent if and only if their covariance is zero. We then proceed to calculate the covariance $\textrm{Cov} (h^{(1)}\_i, h^{(1)}\_j)$ for $i \neq j$ . The covariance, in general, will be a function of the weights $w^{(1)}\_{i \cdot}$ and $w^{(1)}\_{j \cdot}$. However, we demonstrate that in the limit as $d \rightarrow \infty$, $\textrm{Cov} (h^{(1)}\_i, h^{(1)}\_j)$ converges with high probability (w.h.p.) to 0.
Indeed

Here, step $\boldsymbol{a}$ follows from linearity and the fact that $\mathbb{E}(\xi_m) = 0 \\; \\; \forall m$. Steps $\boldsymbol{b}$ e $\boldsymbol{c}$ directly result from substituting the covariance matrix of $\boldsymbol{\xi}$, given that $\boldsymbol{\xi} \sim \mathcal{N} (0, \mathbb{I})$. Regarding the final summation, each generic term comprises a product of i.i.d. variables distributed according to a Gaussian distribution with zero mean and variance $\mathcal{O} \left( \frac{1}{d} \right)$. Therefore, it follows from the CLT that, in the limit as $d \rightarrow \infty$, the summation converges, w.h.p. , to 0.

To avoid redundancy, we refer the reviewer to the general response, to which we'll now add further clarifications addressing more specific doubts.

Reply to Q1

Correctly noted, in the absence of IGB, we anticipate the distribution to tighten around a single value (in the case of a binary dataset like that in Fig.1, $f_0 = 0.5$). However, it's crucial to note that this convergence of the distribution to the true value formally occurs only in the limit of infinite datapoints (see also Definition 3.1). In our simulations on finite datasets, we never actually observe a spike at a single value but rather a distribution concentrated around it.

Through our analysis, we go further to quantify these finite-size effects, resulting in theoretical curve profiles to compare with empirical histograms (for instance, the linear case in Fig.5).
For more details regarding the analysis concerning corrections due to finite-size effects, refer to, for example, App.C.1, the paragraph 'Linear' (In Fig. 6, the effects of finite size are depicted as the number of data points increases, showing the curves derived in App.C1.).