Partial observation can induce mechanistic mismatches in data-constrained models of neural dynamics

We thank the reviewer for their careful critiques of our work, which have helped us further strengthen our paper. Below we respond to their comments on Weaknesses of our work:

• The reviewer brings up an excellent point---that both partial observation and the selection of a latent dimension much smaller than the number of observed neurons could contribute to why the feedforward chain in the experiment of Section 2 is misidentified as a line attractor. We set up the experiment to parallel how data-constrained models are often fit in practice, with both partial observation and an explicit bias towards smaller latent dimensions, but appreciate the concern that this leaves room for multiple explanations as to why the mismatch we found occurs. However, since partial observation even in the absence of an explicit bias towards small latent dimensions already bounds model dimension to be smaller than that of the teacher network, the quoted explanation we provide was intended to implicitly also support explanation (b). Nonetheless, we will provide further clarification on this point and our interpretation of the findings here.

• We agree that the experiment you propose would be fruitful in disentangling explanations (a) and (b), and as such have performed a sweep of LDS models of various latent dimensions and observability conditions, fit to the same functionally feedforward chain as in Fig. 1b.

  In particular, using the suggested notation, we fit models with a latent dimension $d = n-1$ (the $-1$ accounts for the fact that the LDS fitting procedure we use requires latent dimension to be strictly less than the observation dimension) to a functionally feedforward teacher circuit performing the same integration task (as in Fig. 1b). We performed fits at various values of $n$, up to the suggested fraction of $n = 0.1N$. In addition, as suggested, we fit LDS models of small latent dimensions $d \in \{2, 3, \dots, 6\}$ at full observability.

  The results demonstrate that the mechanistic mismatch we observe persists even if partial observability is the only driver of restricted model dimension. Further, the second experiment demonstrates that a model of small latent dimension will still learn a line attractor-like mechanism even at full observability. This finding is consistent with the quoted explanation of lines 127-132.

• We agree that an experiment with a fixed length feedforward chain while increasing $D$ could further clarify how the results of Fig. 3 should be interpreted. We interpret the suggested connectivity for such an experiment proposed by the reviewer as resembling synfire chain connectivity. Specifically, we consider teacher networks with connectivity $B = \frac{1}{k}\delta_{i+1,j} \otimes \mathbf{1}_{k\times k}$, where $k$ is the number of copies of each neuron in the chain. Here, the $\frac{1}{k}$ factor is included to ensure that all the nonzero singular values of $B$ remain equal to $1$ as the number of copies $k$ is scaled up. We compute the learned student matrix for a random selection of $d=50$ observed neurons in the limit of long observation time $T$, for fixed chain length $D/k = 50$ and values of $k \in \{2^0, 2^1, \dots 2^6\}$. For each value of $k$, we report the top two eigenvalues, time constants, and singular values of the learned student networks, averaged over $10$ random observed neuron selections, sampled without replacement.

  The results are consistent with the interpretation that neither a feedforward chain, nor a line attractor (or any persistent timescale mechanism) are learned when $D$ is increased in this manner. Specifically, as $k$ is increased, the learned timescales approach the intrinsic neuronal timescale $\tau$, ruling out persistent timescale mechanisms, and the learned singular values fall well below $1$, ruling out a feedforward chain-like structure. Thus, as in Fig. 3, the student still does not learn a feedforward chain, but fails to do so in a different manner.

Plots for the latest experiments are included in the global response.

We thank the reviewer for their appreciation of our work. Below we address some of the concerns and questions that they raise:

Weaknesses:

• We thank the reviewer for pointing out the work of Pandarinath's group, which we unfortunately missed citing in our initial manuscript. We will add the suggested references.

Questions:

• While the analytically tractable setup we describe in Section 3 is indeed constrained to a one-to-one mapping from units in the student and units in the teacher, we would like to emphasize that the motivating experiment described in Section 2 was actually performed with a latent variable model (LDS) that does not have such a constraint (See lines 109-113). Further, for our results on low-rank non-normal teacher dynamics (Section 3.1), we showed experiments in a supplementary figure that found that the spurious slow directions we derived analytically for the one-to-one mapping setup still occurred in LDS models (Supplementary F.5). We did not experiment with LFADS because it is a spiking model, whereas we focus on rate-based models throughout. Moreover, to the best of our knowledge, there's not a straightforward way of extracting spectral information from LFADS weights to make claims about attractor dynamics, whereas this can and has been done for LDS.
• We describe the limitations of data-constrained models in detail here to help frame why the insights that are derived from such models are often rather course-grained (attractor-like properties rather than individual synaptic weights). Further, it motivates our analytical setup in Section 2, which includes two of the four stated limitations (partial observation and neuronal noise), and justifies why we focus on recovery of spectral properties over other possible criteria (e.g., weight recovery). Lastly, the discussion of these limitations supports the overall message that even in a best-case scenario where the other two limitations are ignored, data-constrained RNNs are inductively biased towards fitting (possibly spurious) attractor-like solutions.
• In the white noise case, the students were only driven by noise; there are no known additional inputs. We will reword that sentence to make this distinction clearer.

We thank the reviewer for their careful reading of our manuscript, and are grateful for their positive assessment. Below we respond to their comments on the Weakenesses of our work:

• We are glad the reviewer found our formulation of the problem intuitive, and agree that the settings we consider are constrained. However, we would argue that they are not exceptionally so. In particular, we emphasize that the integrator networks in Figure 1 do in fact have multiple plausible mechanistic explanations; by examination of the activity and network outputs alone we could not discern that the example in 2a is a line attractor, while that in 2b is a feedforward chain. Given the mismatch in the fitted and ground-truth flow fields, one could experimentally distinguish these hypothesis based on perturbing the activity of the integrator neurons in this circuit. This could be accomplished, for instance, through optogenetic stimulation (see e.g. O'Shea, Duncker, et al. 2022, or very recent work by Vinograd et al. 2024 posted after the NeurIPS submission deadline). Carefully investigating when perturbations can sensitively distinguish between mechanistic hypotheses is an important topic, but lies beyond the scope of the present work.

• We agree with the reviewer that it would be interesting to expose possible misinterpretations of circuit mechanisms in existing biological datasets. However, for the purpose of the present paper we prefer to maintain our focus on synthetic data with known mechanisms, so that we can map out when mismatch in model mechanism arises. This is also motivated by the fact that conclusively demonstrating misinterpretation of mechanism in biology would require experimental manipulations; as we are not in a place to carry out such experiments, for the moment we prefer to take a more conservative approach rather than suggesting particular targets for future study. We will a discussion on this broader point, to our work.

• We thank the reviewer for drawing our attention to the simulation-based inference literature, and will expand the discussion section to reflect its connections with our work.