Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensions

Thank you for your careful feedback and for having checked both the numerical experiments and the proofs. Your suggestions will definitely help to clarify the paper.

One potential weakness of this work is the relative simplicity of the model. However, I do not consider this a major weakness, as it is compensated by the originality of the results in relation to the existing literature, providing a baseline for investigating more complex settings.

Indeed, our data model is a simple single-index model. We decided to focus on single-index models in this paper because of the prevalence of Gaussian single index studies in the recent past (Ben Arous et al. JMLR '21, Bietti et al. NeurIPS '22, Damian et al. NeurIPS '23, Damian et al. arXiv:2403.05529, Bardone & Goldt ICML '24, Wang et al. NeurIPS '24). Here, we provide an extension to the non-Gaussian case, which is closer to realistic data. We are considering on an extension to multi-spike models for our future work.

Inconsistent notation in Fig. 1.B is used instead of (b).

Thanks for noticing, we fixed this.

I suggest writing explicitly what quantity is represented in the legends of Fig.4.

Thank you, we have added a title to the legends to stress that these are the widths of the patches extracted from ImageNet images.

Thank you for your detailed feedback and your useful comments, which spurred us to run an additional experiment to link ICA with deep CNNs, and to provide two additional plots to provide intuition on the effect of smoothing, and on the intermediate regime of FastICA. We start with these points before addressing your remaining comments below. We hope our reply alleviates any remaining concern; if not, please let us know, otherwise we would really appreciate it if you increased your score. Thank you for your time!

Readability of the introduction & explanations on why Independent Component Analysis learns similar filters as deep CNNs in Fig. 1.

Thank you for this suggestion -- we now realise the introduction jumped from deep CNNs to ICA too quickly. To make the connection more clearly, we have conducted an additional experiment: we trained three deep CNNs (AlexNet, Resnet18 and DenseNet121) on ImageNet and computed the excess kurtosis of the dot products $s$ between first-layer convolutional filters and ImageNet patches. We found that the excess kurtosis of $s$, which corresponds to the objective function of ICA, sharply increases after about 1000 steps of SGD, precisely when Gabor filters form in the first layer. This demonstrates empirically that neural networks seek out non-Gaussian directions when Gabor filters form, akin to what happens when running ICA. We summarise our results in a revised Figure 1 in https://figshare.com/s/0e72e797306c1a3f216a, and that we hope will clarify the motivation for studying ICA.

The explanation of why SGD benefits from smoothing the contrast function could be clearer [...]

The key intuition behind the smoothing operator is that, thanks to large $\lambda$ (than scales with the input dimension), it allows to evaluate the loss function in regions that are far from the iterate weight $w_t$, collecting non-local signal that alleviates the flatness of the saddle around $\alpha=0$ of $\mathcal L$, reducing the length of the search phase. A new plot, Fig 3 of https://figshare.com/s/0e72e797306c1a3f216a, shows the reduced flatness of the smoothed loss at the origin.

Choice of $\delta$ in the batch sizes $n=d^2, d^{3 + \delta}$, and $d^4$

We could have chosen any $\delta \in (0,1)$, since Theorem 2 implies that the 'intermediate' regime shown in Fig. 2 (middle) holds for a batch size of $d^2 \ll n \ll d^3$. We will add Fig 2 of https://figshare.com/s/0e72e797306c1a3f216a which shows the behaviour for $\delta = 0.5, 0.8$.

The Hermite expansion, square integrability and contrast functions.

We merely need the contrast functions to be square-integrable with respect to the standard normal distribution $P_0$, so we need that the growth of $G$ is less than exponential. In general, we don't require they are integrable with respect to the Lebesgue measure on the real line (e.g., excess kurtosis is not Lebesgue integrable).

In Figure 3, the learning rate $\eta$ is defined based on $k^*_1$, $k^*_2$, but how to tune it in practice?

Our theorem 4 offers a way to make a reasonable guess for $\eta$ as follows: $k^*_2$ is the information exponent of the contrast function and is therefore known. Meanwhile, $k^*_1$ depends on the likelihood and hence on the data distribution. However, we have the bound $k^*_1 \geq k^*_2$. Furthermore, if we assume that data has a non-trivial fourth-order cumulant (a mild assumption in practice), we can set $k^*_1=4$ and hence our theorem gives a recommendation on how to scale the learning rate in practice.

The analysis is focused on learning one non-Gaussian feature. A discussion on how multiple features interact would be valuable.

We decided to focus on single-index models in this paper because of the prevalence of Gaussian single index studies in the recent past (Ben Arous et al. (2021), Damian et al. (2023), Damian et al. (2024), Bardone & Goldt (2024)). We are considering on an extension to multi-spike models for our future work.

Ensuring present tense consistency

Thanks, we fixed this.

Highlighting the generality of non-Gaussian distributions (Cramér's decomposition theorem) [...]

Yes, we agree that Cramér's theorem emphasizes the importance of studying non-Gaussianities, we will discuss it.

Additional empirical justification to explain that "the growth of the excess kurtosis might compensate the poor sample complexity of FastICA in practice"

We will move the plot from the appendix to Fig. 4 with ImageNet results. We agree that it does not fully explain the behaviour of FastICA on ImageNet, but the growth of excess kurtosis with input dimension hints at important finite-size effects, which are out of the purview of our asymptotic theory.

Explicit emphasis in the title that this paper analyses sample complexity [...].

Thank you for this suggestion; we will consider clarifying the title should the paper be accepted. We have already made changes to clarify our objectives and motivations in the introduction (see below).

Thank you for your accurate comments and for the attention to the supplementary material, including the strategies of the proofs. Your suggestions offered valuable food for thought.

I don't see any glaring weaknesses to the paper, beyond the possible restrictiveness of the single non-Gaussian direction model -- however, this is rather trite, as algorithmic results for other multi-index settings are yet generally ill-understood.

Thanks for your observation. We decided to focus on single-index models in this paper because of the prevalence of Gaussian single index studies in the recent past (Ben Arous et al. JMLR '21, Bietti et al. NeurIPS '22, Damian et al. NeurIPS '23, Damian et al. arXiv:2403.05529, Bardone & Goldt ICML '24, Wang et al. NeurIPS '24). We are considering on an extension to multi-spike models for our future work.

The authors should remember to put in the Impact Statement.

Thank you for the reminder, we will add it.

Something that might be helpful is a discussion of why identifying non-Gaussian directions might be desirable or might happen automatically. A concrete mathematical statement here might be to show recovering these directions improves generalization error.

The reviewer raises a really interesting point, which is why it is desirable to learn non-Gaussian directions in the first place. Fascinating work in theoretical neuroscience has established that natural images have a highly non-Gaussian statistical structure: while pixel intensities themselves may follow roughly Gaussian distributions, the relationships between pixels are strongly non-Gaussian. Specifically, natural images are sparse in certain bases: edges, contours, and textures are more prevalent than random pixel noise. Gabor-like filters, i.e. non-Gaussian directions, are efficient at capturing these features that yield sparse image representations, see for example reviews such as Simoncelli & Olshausen, Annu. Rev. Neurosci. 24:1193–216 (2001) or the book by Hyvärinen, Hurri and Hoyer (2009).

More recently, there have been a series of works investigating the importance of non-Gaussian input structures in machine learning from a mathematical perspective, showing that neural networks will learn them if they are relevant for the task, i.e. if they improve generalisation; see in particular Ingrosso & Goldt, PNAS '22; Bardone & Goldt, ICML '24; Lufkin et al. NeurIPS '24.

We will add an extended discussion of this issue to the revised version of the paper, should it be accepted.