Spectral Analysis of Representational Similarity with Limited Neurons

We thank the reviewer for their thorough review. We hope our response addresses the reviewer’s concerns and look forward to discussing it further.

Strength and Weaknesses:

(Theoretical Insight) Moment-based unbiased estimators for CKA are plug-in estimators that correct for finite-sample bias arising from both input sampling and neuron sampling. While these methods are useful in practical settings, they offer limited theoretical insight—for example, they do not theoretically reveal how CKA varies with the number of neurons or how the eigenspectrum structure influences the finite-sample CKA.

Our work is complementary to these methods in that it enables an analytical understanding of representational similarity, specifically in the regime of limited neuron sampling. For example, consider recording $N$ neurons from the brain. An unbiased estimator can correct for finite-sample bias when comparing neural recordings to a model, ideally yielding a value close to the true similarity. In contrast, the random matrix theory allows one to analyze the spectral properties of the sample Gram matrix to infer, for instance, how many additional neurons are needed to resolve further eigenvectors. This type of inference is not possible with plug-in estimators.

(Asymptotic unbiasedness) We are grateful to the reviewer for raising this issue, which led us to improve our theoretical analysis. Here, we discuss the bias in our proposed algorithm and will provide a more detailed analysis in our manuscript.

The random matrix theory, which underlies our analysis, is valid in the limit $P, N \to \infty$ ($q = P/N \in (0,\infty)$) and is asymptotically unbiased by construction. In practice, for finite $N$ , the theoretical prediction for the self-overlap matrix $Q$ has order $1/\sqrt{N}$ fluctuations (see Ref A). Propagating this error to the estimated CKA, we also expect a similar $\mathcal{O}(1/\sqrt{N})$ fluctuation. We will provide additional experiments to demonstrate this.

In summary, we emphasize again that our method provides a predictive theory of CKA and complements unbiased estimators. In practice, we believe unbiased estimators are useful, especially for very small data, and should always be compared to RMT predictions. As the reviewer rightfully pointed out, the current version lacks a comparison between two methods, and we now include unbiased estimators in our experiments (see below).

In Section 4, we will elaborate on this in more detail and also include a separate subsection discussing the differences with unbiased estimators. We are also re-running all our experiments with unbiased estimators and will include the plots in the revised version (see below).

[a] Bun et al., On the overlaps between eigenvectors of correlated random matrices

Questions:

1: (illustrative example)

Consider two identical representations whose population eigenvalues follow a power law with exponent $\gamma = -1.2$ and $P = 100$. In this setting, the top $5 \times 5$ block of
$\frac{\tilde{\lambda_i}}{\sqrt{\sum_{j=1}^P \tilde{\lambda_j}^2}} \cdot \frac{\tilde{\mu_a}}{\sqrt{\sum_{b=1}^P \tilde{\mu_b}^2}} \cdot \tilde{M_{ia}}$ already accounts for about 95% of the CKA. While this is a simple toy case, the takeaway is that when eigenvalues decay quickly, the leading components dominate the CKA; with sufficiently many recorded neurons, these are precisely the localized components that can be estimated accurately.

(From heuristic to quantitative uncertainty.)

We agree that the above argument is still heuristic. Rather than attempting a full bias analysis on constrained optimization, we provide confidence intervals for $\tilde{M_{ia}}$ using a maximum likelihood estimation (MLE) approach.

First assume that we had the correct parametric form of the population eigenvalues, so that we estimated them accurately and thus obtained the self-overlap matrix $Q$ (as shown in Appendix C). Under a Gaussian ansatz(see Ref a where
$\ket{u_i^{(t)}} = \sum_{j=1}^P \epsilon_{ij}^{(t)} \sqrt{Q_{ij}} \ket{\tilde{u_j}} \quad \text{with} \quad \epsilon_{ij}^{(t)} \sim \mathcal{N}(0,1),$ (see [Ref A]) the squared overlaps follow $M_{ia}^{(t)} \sim \sigma_{ia}^2 \chi_1^2$, where $\sigma_{ia}^2 = \sum_{j=1}^P Q_{ij} \tilde{M_{ja}}$.

Using profile likelihood on the negative log-likelihood
$\ell_a(\tilde{M_{\cdot a}}) = \frac{1}{2} \sum_{i=1}^P \left[ \ln \sigma_{ia}^2 + \frac{M_{ia}}{\sigma_{ia}^2} \right],$ and running MLE, we obtain $(1 - \alpha)$ confidence intervals that translate directly to CKA/CCA via the delta method. (We added an Appendix for algorithms and also comparisons between true similarity, MLE estimates, and confidence intervals.)

In practice, we used constrained optimization rather than unconstrained MLE since each matrix element lies in $[0, 1]$, which yields slightly more stable results in extremely small neuron regimes (e.g., $N = 10$).

(How many components are well localized.)

We quantify the number of well-localized eigenvectors by analyzing when sample eigenvalue support components merge. For power-law population spectra $\tilde{\lambda_i} = i^{-1 - \gamma}$, we use a local two-peak approximation of the blue function
$\mathfrak{B}(x) = \frac{1}{x} + \frac{1}{N} \sum_{i=1}^P \frac{1}{\frac{1}{\tilde{\lambda_i}} - x}$ (see [Ref b]) to derive the critical neuron size $N_i^\star$ at which adjacent components coalesce.

This yields a delocalization threshold $i^\star \sim \frac{1 + \gamma}{\sqrt{8}} \sqrt{N}$, implying that only $O(\sqrt{N})$ leading eigenvectors remain reliably localized — precisely those whose diagonal overlaps $Q_{ii}$ stay near 1 before dropping sharply at $i^\star$.

[a] Monasson et al., Estimating the spectrum of large covariance matrices from random subsamples

[b] Bun et al., Cleaning large correlation matrices

2: We thank the reviewer for this suggestion. We now include the CKA predictions using unbiased estimators in all our experiments. Our method almost perfectly agrees with the unbiased estimator in synthetic dataset experiments.

In Fig.4, we found that our method tends to slightly overestimate CKA compared to the unbiased estimator, but it does not change the overall picture. We will include these figures in our appendix. We now confirm this behavior also for other regions (V1, V4, IT), and will include their plots in the appendix.

3.: The approximation $\beta \mathfrak{g}' \ll 1$ is mainly used to obtain an analytical solution (see ref.[3]). It essentially corresponds to the large eigenvalue $\lambda \gg 1$ and small $q\ll 1$ limit.

To see this, we note that $\mathfrak{g} \sim 1/z$ for $z\gg1$. Since $\mathfrak{g}'=z\mathfrak{g} \sim \mathcal{O}(1)$ remains order 1, the condition $\beta \mathfrak{g}' \ll 1$ implies $\beta \sim q\ll 1$. This can be also seen from Eq.S60 where the small $q$ correction to $\rho(\lambda)$ is given by an infinite power series in terms of $\lambda^{-1}$, which is negligible for large eigenvalues.

The limit $\lambda \gg 1$ can be relevant to neural data analysis since CKA is dominated by large eigenvalues, and the limit $q\ll 1$ is relevant when we study small deviations from the population density. Furthermore, we numerically confirm that our final formula leads to precise results for the power-law distribution, which seems to be relevant in many neuroscience experiments.

We will include a detailed explanation of the reasoning behind our assumptions in Appendix C.

**4.: ** (a) We thank the reviewer for this suggestion. We now state our main result as a proposition. (b) We made this choice because we mainly work with sample quantities, and with the hat, the equations looked too crowded. We can certainly change the notation if it would help readability.

Limitations: We thank the reviewer for pointing this issue out. Some works handle this using estimators, but an RMT treatment seems to be difficult and beyond the scope of this work. We will explicitly mention this issue in a separate Limitations section.

I thank the authors for their thorough reply to my questions. I think my major concerns have been satisfactorily addressed, and I will increase my score accordingly to a 5 when submitting my final recommendation. Given that I intend to increase my score, I would like to elaborate on why, though I think some of Reviewer iYBp's concerns are well-founded, I am still in favor of acceptance.

First, regarding the assumption of Gaussian random projections: I of course agree that the assumptions under which randomly-oriented projections and subsampling are equivalent are clearly not always satisfied in neural data, and that this inequivalence is an issue deserving of more theoretical attention (going back to Gao and Ganguli's "A theory of multineuronal dimensionality, dynamics and measurement"). However, I will echo the authors in offering a random matrix theorist's standard apologia for considering rotation-invariant ensembles: given an estimation problem of interest, it is worth first bringing the powerful analytical toolkit available assuming rotation-invariance to bear to see what surprises arise even in this highly symmetric, idealized world. Then, one can subsequently dive into the world of non-invariant ensembles, through what is usually a gradual process based on exploration of specific models for non-invariant random matrices. Identifying when and how the predictions of results derived assuming rotation-invariance break down is a useful part of in this process, and requires the results of the rotation-invariant computation to be known. Thus, I am not inclined to dismiss the results of this paper out of hand because the authors have assumed Gaussian projections.

Second, regarding whether static measures of representational similarity like CCA/CKA are worthy of study at all: The fact remains that a portion of the neuroscience community uses static measures to compare representations, no? (I would also strongly contest the claim that the neuroscience community at large has overlooked dynamics, but that is neither here nor there) Pragmatically, for those that do so---irrespective of whether the application of this static measure is appropriate for the question of interest---it seems worth knowing how their results are biased. Respectfully, I do not think that it is fair to place the blame for the sins (if they are indeed so) of some fraction of the broader community solely upon the shoulders of the authors of the present manuscript. Moreover, in the same way as I see the rotation-invariant assumptions of this manuscript as a first step towards an understanding of subsampling, I would see theoretical characterization of static measures as a first step towards an understanding of those measures that account for dynamics. I look forward to further discussion of this issue with the authors, and with Reviewer iYBp.

Thanks for following up! I think the addition of a formal "limitations" section will be helpful, and I like the idea of highlighting that one should choose a similarity measure with appropriate symmetries (especially when it comes to considering representational drift and related issues). Though I gather it's not yet visible to you as authors, I have raised my score.

We thank the reviewer for their positive comments and for raising important concerns. We hope our responses clarify the reviewer’s questions, and we are happy to address any follow-up questions.

Strength & Weakness:

We are aware that bracket notation is non-standard in the ML community, but we decided to use it because of how it was formulated in the original paper. We apologize for the inconvenience and will try to include a more detailed introduction to bracket notation in the appendix.

Questions:

1: This is an excellent question and one of the main limitations of our work. Indeed, it is not clear if this is a good model for neurophysiological recordings. We have several reasons to consider this:

1. It is analytically easy to study since there is ample work on Wishart matrices.
2. It has been debated whether neural data can be regarded as random projections. Often, neural data involves convolution and may potentially mix with each other. Also, large-area recordings such as fMRI automatically act as random projections as they involve averaging many units.
3. Neural recordings have been found to be high-dimensional in the sense that individual neurons represent population-level information, hinting that the recorded neurons are already mixed.

Although currently we struggle to generalize this to other sampling methods, we want to mention that this method may potentially be used to check whether the individual units are mixed or not. RMT is a powerful tool that can make precise predictions about how spectral quantities change as a function of sample size. One may employ cross-validation techniques to see if the change in the spectra by adding more neurons matches the RMT prediction and potentially informs practitioners. We leave this point to future work.

2: We thank the reviewer for raising this insightful point.

First, we can certainly use our method for estimating the noise ceiling by measuring the similarity between repeated trials of the same neural stimulus. Since both the noise ceiling (self-similarity) and neural similarity monotonically decrease with neuron sampling, it is not immediately clear whether their ratio still remains underestimated. On the other hand, normalizing with the noise ceiling may also result in inflated similarity scores since it is extremely sensitive to neuron sampling.

While we could not make this analysis due to time constraints, it will certainly be interesting to explore this problem.

Additionally, we aim to extend this theory to random projections in which the matrix elements are correlated instead of independent, as in the Wishart case.

We will discuss these points in detail in a separate Limitations section.

3: We thank the reviewer for bringing this interesting point to our attention. We certainly intend to extend such RMT analysis to other similarity measures, and will provide a more detailed discussion of these extensions in the future directions section.

While an RMT treatment of DSA itself seems difficult because of its complex nature, we would like to mention that this approach may potentially transfer to comparing delay-embedded representations with CKA when there are limited data points or the timeseries is sparsely sampled. However, we note that this still would not capture the entire non-linear nature of DSA.

Extension to Procrustes-based similarity measures is a very interesting idea! While CKA depends on the Frobenius norm of the cross-covariance, Procrustes distance relies on its nuclear norm (see Pospisil et al. [Ref a]), and we think it is reasonable to expect that a similar bias also exists in Procrustes distance.

Inspired by the reviewer’s question and the cited papers, we are currently conducting simple experiments on ring attractors with finite sampling and hoping to report empirical results on whether such biases exist in DSA and Procrustes.

[a] Pospisil et.al. Estimating shape distances on neural representations with limited samples

We thank the reviewer for their kind review and hope our responses addressed the reviewer’s questions.

Questions:

1: We thank the reviewer for catching this. In Fig.2, we had 20 trials of sampling neurons, and we will mention it in the caption.

2. We are sorry for the confusion. We have fixed the organization now.

3. We thank the reviewer for this suggestion and agree that application to other data would be valuable. We are actually aiming to release a codebase that can be generalized to any dataset. We will also try to include some new experiments on additional datasets in our revised manuscript.

We thank the reviewer for their thorough review. We hope our response addresses the reviewer’s concerns and look forward to discussing it further.

Weakness:

1&2: We first note that similarity measures are guided by the symmetries and structure present in the data. For instance, CKA and Procrustes treat two representations as similar if they are related by an orthogonal transformation. Depending on the underlying data-generating process, this may or may not be an appropriate symmetry to assume, and in some cases, alternative metrics may be more suitable.

We acknowledge that both CCA and CKA are static similarity measures and therefore do not account for temporal dynamics or nonlinear structure in the data. However, our work explicitly targets static representations, which remain widely used and scientifically relevant—for example, in studies of early visual cortex, where neurons are often modeled via their time-averaged firing rates. In such settings, the assumption of Poisson-distributed spiking and the use of rate-based representations are standard practice in neuroscience.

Regarding the concern about long-term relevance and impact, while our theoretical analysis focuses on CKA, the core phenomenon we study—sampling-induced spectral bias—is not unique to CKA or even to kernel methods. Any approach that summarizes high-dimensional neural activity using operators such as covariance matrices, Gram matrices, or cross-spectral matrices will exhibit analogous finite-sample biases in their eigenspectra. Thus, we believe our framework lays the foundation for extensions to more complex or dynamic similarity metrics in future work.

3: We agree that modeling neuron sampling as i.i.d. random projections is a convenient baseline but can be unsuitable when anatomical constraints induce spatial clustering and correlated activity, potentially violating the independence assumptions behind our RMT analysis. Our goal in this paper is to foreground—apparently for the first time—that neuron‑wise sampling itself reshapes the spectrum and can bias CKA/CCA; this effect has not been examined thoroughly. We now state this assumption explicitly and position our results as a baseline. Extending the analysis to structured, non‑i.i.d. sampling (e.g., spatially biased or block‑correlated selections) is an important direction, and we plan to pursue it in follow‑up work.

Please also see our response to Reviewer fmbc Q.1

Minor issues:

We thank the reviewer for pointing out this confusion. In our notation, tildes correspond to population quantities; however, we have another quantity, $\hat{\tilde M}$, which denotes the estimated value of the population $\tilde M$ from sample quantities. We will abandon this notation and instead refer to it as $\tilde M_{est}$.

Please also see our response to Reviewer HcPT Q.4b.

[a] Kriegeskorte et al. (2008) “Representational similarity analysis—connecting the branches of systems neuroscience.” Frontiers in Systems Neuroscience.

[b] Nili et al. (2014) “A Toolbox for Representational Similarity Analysis.” PLoS Comp. Bio.

[c] Kornblith et.al. (2019) “Similarity of Neural Network Representations Revisited (CKA).” ICML.

[d] Morcos et.al. (2018) “Insights on representational similarity in neural networks with CCA.”

[e] Sucholutsky et al. (2023) “Getting aligned on representational alignment.”

[f] Murphy et.al. (2024) “Correcting Biased Centered Kernel Alignment Measures in Biological and Artificial Neural Networks”

[g] Canatar et.al. (2023) “A spectral theory 346 of neural prediction and alignment.” NeurIPS.

[h] Yamins et.al. (2014) “Performance-optimized hierarchical models predict neural responses in higher visual cortex.” PNAS.

[i] Khaligh-Razavi and Kriegeskorte. (2014) “Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation”. PLoS Comp. Bio.

[j] Schrimpf et.al. (2018) “Brain-score: Which artificial neural network for object recognition is most brain-like?”

[k] Kar and DiCarlo. (2024) “The Quest for an Integrated Set of Neural Mechanisms Underlying Object Recognition in Primates.” Ann. Rev. Vis. Sci.

[l] Deitch et al. (2021) “Representational drift in the mouse visual cortex.” Curr. Biol.

[m] Roth and Merriam. (2023) “Representations in human primary visual cortex drift over time.” Nat. Comm.