Discovering Temporally Compositional Neural Manifolds with Switching Infinite GPFA

We thank the reviewer for constructive feedbacks and overall positive recognition of our work. Please find responses to the raised questions and weaknesses below.

• The main criticism to the submitted manuscript is the absence of baseline comparison, specifically with respect to the Bayesian GPFA model (Jensen et al. 2021). We have implemented and evaluated the Bayesian GPFA model on both the synthetic dataset and the real neural dataset, and have revised the manuscript to include the comparison. The key takeaway for this comparison is summarised as following.
  • Averaging over multiple random seeds, Bayesian GPFA converges to similar asymptotic free energy value as infinite GPFA, given the synthetic data generated from both generative processes with and without stochastic binary loading mask. However, the Bayesian GPFA training is significantly less stable (indicated by the greater standard deviation across different random seeds, see Figure 2b in the revised manuscript). On the real neural dataset, Bayesian GPFA converges to a lower value as infinite GPFA, and the CCA analysis reveals that the infinite GPFA latents provides more accurate representation of the relevant behavioural variables (Figure 3c).
  • In addition to the different priors used for automatic model selection (ARD and IBP for Bayesian GPFA and infinite GPFA, respectively), a key difference between the two models lie in the approximate inference. The Bayesian GPFA model leverages circulant approximation to the full covariance matrix in the variational GP distribution, whereas the infinite GPFA model leverages the sparse variational approximation with inducing points. The ability to model the covariance matrix over all input locations enables Bayesian GPFA to capture information on a finer temporal resolution, which explains why the fitted latents directly capture the loaded latents ($\tilde{\mathbf{f}} = \mathbf{Z}\odot\mathbf{f}$; Figure 2e, bottom panel). However, modelling the loaded latents with only the GP latents using the circulant approximation raises concerns. Specifically, direct modelling of the loaded latents lacks interpretability, such that it is difficult to separate variabilities between the latent processes and the loading processes (Figure 2e, bottom panel, and Figure 3c).
  • The circulant approximation of the full covariance matrix induces greater approximation error as the latent dimension increases (Figure 2c, bottom).
  • The computational complexity for the circulant approximation scales supra linearly with the number of inputs, $T$, ($\mathcal{O}(T\log T)$). The resulting model is prohibitively expensive for practical application (Figure 2h).
• We appreciate the reviewer's sharp observation that the infinite GPFA model requires more training epochs to reach convergence. We have run additional ablations studies and show that the difference in the amount of training to reach convergence between standard and infinite svGPFA does not increase with the number of latent dimensions in the generative process (Figure S3 a-d in the revised manuscript). In contrast, the performance gap (in terms of asymptotic free energy objective) between the trained models under the two conditions with and without encoding variability grows significantly for the standard svGPFA model as the latent dimension increases, whereas the same gap for the infinite svGPFA model is minimally affected.
• The expected level of masking, or the expected level of sparsity in the binary masking matrix, $Z$, is $0.6$ in Figure 2 of the main paper. We define $\tilde{\alpha} = \langle\frac{\sum_{n, d}z_{nd}}{ND}\rangle$ for quantifying the expected level of sparsity. We quantify the difference (in percentage) of asymptotic free energy objective for the standard and infinite svGPFA models for varying levels of $\tilde{\alpha}$. From Figure S3f in the revised manuscript, we observe that the gap increases as the degree of sparsity increases (i.e., $\tilde{\alpha}$ decreases).
• The runtime comparison in Figure 2h in the revised manuscript now shows the average wallclock time for a single EM iteration during training for the different models (the previous version shows the average wallclock time for $2000$ EM iterations during training). Note that the revised Figure 2h now includes the comparison with bGPFA.

We again thank the reviewer for their instructive comments and feedbacks, and we hope our responses and revised manuscripts have addressed raised concerns. We are happy to engage in any extended discussion if there is any remaining question.

We thank the reviewer for constructive feedbacks and overall positive recognition of our work. Please find responses to the raised questions and weaknesses below.

• The main criticism to the submitted manuscript is the absence of baseline comparison, specifically with respect to the Bayesian GPFA model (Jensen et al. 2021). We have implemented and evaluated the Bayesian GPFA model on both the synthetic dataset and the real neural dataset, and have revised the manuscript to include the comparison. The key takeaway for this comparison is summarised as following.
  • Averaging over multiple random seeds, Bayesian GPFA converges to similar asymptotic free energy value as infinite GPFA, given the synthetic data generated from both generative processes with and without stochastic binary loading mask. However, the Bayesian GPFA training is significantly less stable (indicated by the greater standard deviation across different random seeds, see Figure 2b in the revised manuscript). On the real neural dataset, Bayesian GPFA converges to a lower value as infinite GPFA, and the CCA analysis reveals that the infinite GPFA latents provides more accurate representation of the relevant behavioural variables (Figure 3c).
  • In addition to the different priors used for automatic model selection (ARD and IBP for Bayesian GPFA and infinite GPFA, respectively), a key difference between the two models lie in the approximate inference. The Bayesian GPFA model leverages circulant approximation to the full covariance matrix in the variational GP distribution, whereas the infinite GPFA model leverages the sparse variational approximation with inducing points. The ability to model the covariance matrix over all input locations enables Bayesian GPFA to capture information on a finer temporal resolution, which explains why the fitted latents directly capture the loaded latents ($\tilde{\mathbf{f}} = \mathbf{Z}\odot\mathbf{f}$; Figure 2e, bottom panel). However, modelling the loaded latents with only the GP latents using the circulant approximation raises concerns. Specifically, direct modelling of the loaded latents lacks interpretability, such that it is difficult to separate variabilities between the latent processes and the loading processes (Figure 2e, bottom panel, and Figure 3c).
  • The circulant approximation of the full covariance matrix induces greater approximation error as the latent dimension increases (Figure 2c, bottom).
  • The computational complexity for the circulant approximation scales supra linearly with the number of inputs, $T$, ($\mathcal{O}(T\log T)$). The resulting model is prohibitively expensive for practical application (Figure 2h).
• We thank the reviewer for pointing out the typos and previously illegible axis labels in Figure 2a. We have now addressed them in the revised manuscript.
• We thank the reviewer for noting the figure rendering issue with Figure 2. This is due to high dpi associated with the scatter plots in Figure 2d. We have now replaced the figure with a copy with lower dpi, which should now improve the rendering speed.
• Regarding the question on how does the infinite GPFA handles overlapping/slightly differing latent factors, we emphasise that in our ablation studies on the synthetic dataset (see Figure S3 in the revised manuscript), some of the additional latent processes exhibit purely phase-shift relative to earlier latents (e.g., sin(3x) and cos(3x)). Under such conditions, the infinite GPFA still infers the ground-truth latents with high accuracy.

We again thank the reviewer for their instructive comments and feedbacks, and we hope our responses and revised manuscripts have addressed raised concerns. We are happy to engage in any extended discussion if there is any remaining question.

We thank the reviewer for constructive feedbacks and overall positive recognition of our work. Please find responses to the raised questions and weaknesses below.

• The main criticism to the submitted manuscript is the absence of baseline comparison, specifically with respect to the Bayesian GPFA model (Jensen et al. 2021). We have implemented and evaluated the Bayesian GPFA model on both the synthetic dataset and the real neural dataset, and have revised the manuscript to include the comparison. The key takeaway for this comparison is summarised as following.
  • Averaging over multiple random seeds, Bayesian GPFA converges to similar asymptotic free energy value as infinite GPFA, given the synthetic data generated from both generative processes with and without stochastic binary loading mask. However, the Bayesian GPFA training is significantly less stable (indicated by the greater standard deviation across different random seeds, see Figure 2b in the revised manuscript). On the real neural dataset, Bayesian GPFA converges to a lower value as infinite GPFA, and the CCA analysis reveals that the infinite GPFA latents provides more accurate representation of the relevant behavioural variables (Figure 3c).
  • In addition to the different priors used for automatic model selection (ARD and IBP for Bayesian GPFA and infinite GPFA, respectively), a key difference between the two models lie in the approximate inference. The Bayesian GPFA model leverages circulant approximation to the full covariance matrix in the variational GP distribution, whereas the infinite GPFA model leverages the sparse variational approximation with inducing points. The ability to model the covariance matrix over all input locations enables Bayesian GPFA to capture information on a finer temporal resolution, which explains why the fitted latents directly capture the loaded latents ($\tilde{\mathbf{f}} = \mathbf{Z}\odot\mathbf{f}$; Figure 2e, bottom panel). However, modelling the loaded latents with only the GP latents using the circulant approximation raises concerns. Specifically, direct modelling of the loaded latents lacks interpretability, such that it is difficult to separate variabilities between the latent processes and the loading processes (Figure 2e, bottom panel, and Figure 3c).
  • The circulant approximation of the full covariance matrix induces greater approximation error as the latent dimension increases (Figure 2c, bottom).
  • The computational complexity for the circulant approximation scales supra linearly with the number of inputs, $T$, ($\mathcal{O}(T\log T)$). The resulting model is prohibitively expensive for practical application (Figure 2h).
• We thank the reviewer for noting our current omission of citing the relevant Gokcen et al. study, we have now cited the paper in the revised manuscript with accompanying discussion.
• We thank the reviewer for noting the figure rendering issue with Figure 2. This is due to high dpi associated with the scatter plots in Figure 2d. We have now replaced the figure with a copy with lower dpi, which should now improve the rendering speed.

We again thank the reviewer for their instructive comments and feedbacks, and we hope our responses and revised manuscripts have addressed raised concerns. We are happy to engage in any extended discussion if there is any remaining question. If all raised concerns are resolved satisfactorily, we sincerely hope the reviewer could raise their score accordingly.

We thank the reviewer for constructive feedbacks and overall positive recognition of our work. Please find responses to the raised questions and weaknesses below.

• Regarding the reviewer's concern that the combination of GPFA with IBP prior is not revolutionary, we wish to emphasise that the infinite GPFA model is, to the best of our knowledge, the first model that explicitly incorporates the IBP prior in latent variable models with continuous-time latents. Hence, from the methodological perspective, we believe the idea behind the infinite GPFA model contains sufficient novelty. We understand that it is possible there are existing models that exhibit the above model features, and we respectfully ask the reviewer to inform us of such model, and we are happy to cite them in the revised manuscript. From the neural data analysis perspective, the infinite GPFA model provides a completely new way of interpreting population neuronal firing: the expression/encoding of latent (behavioural) information in both single-neuron and population activities is potentially not constant over time. The novel interpretation of neural coding enables us to better understand transient changes in neural activities (see, e.g., discussions regarding the motivations from the neuroscience perspective in l.72-l.80 and the Discussion section in the revised manuscript).
• The infinite GPFA model models the firing rate of each neuron (in log-space) as the weighted product of the binary loading ($Z$) and the latent processes ($f$). Binary expression of latent processes in the neural activities might happen at a might higher temporal scale than the temporal variations in the latent processes themselves (see, e.g., the discussion on exemplary datasets that exhibit such property from Kelemen and Fenton, 2010, and Jezek et al. 2011, in Section 1 of the revised manuscript). Hence, changes in transition alone (as in SLDS models) is not able to promptly reflect such contextual changes in the neural responses within the biological plausible timescale. We have now updated our discussion regarding comparison with SLDS models in the Related Works section in the revised manuscript to improve its clarity. We hope these additional statements have clearly conveyed our points, but we are happy to engage in further discussions if there is any point remains unclear.
• The number of feature, $D$, is identical for both standard and infinite svGPFA (and our new baseline comparison, the bGPFA model) for experiments with both synthetic and real datasets. Specifically, we set $D = 10$ for both the synthetic dataset and the real neural data. These hyperparameters are provided in Supplemental S3.1 in the revised manuscript.
• There are multiple future directions for the infinite GPFA model. From the methodological perspective, it is possible to extend the model to include non-linear generative and recognition networks to improve the modelling flexibility of the latent variable model (potentially under the variational autoencoder framework, see, e.g., Yu, et al. 2022). Moreover, despite introducing temporally varying loading process, the loading matrix, $C$, is still assumed to be stationary. Hence, another possible extension to the infinite GPFA model would be incorporating priors over the loading weight matrix $C$ with non-trivial temporal dependency (e.g., GP). From a neuroscience perspective, due to the linear assumption of the proposed model, it might be more suitable for modelling neural recordings from motor cortex, which have been shown to exhibit strongly linear tuning properties (see, e.g., Paninski et al. 2004 and Yu et al. 2008), and we leave this for future studies.

We again thank the reviewer for their instructive comments and feedbacks, and we hope our responses and revised manuscripts have addressed raised concerns. We are happy to engage in any extended discussion if there is any remaining question. If all raised concerns are resolved satisfactorily, we sincerely hope the reviewer could raise their score accordingly.

References:

Yu, C., Soulat, H., Burgess, N. and Sahani, M., 2022. Structured recognition for generative models with explaining away. Advances in Neural Information Processing Systems, 35, pp.40-53.

Paninski, L., Fellows, M.R., Hatsopoulos, N.G. and Donoghue, J.P., 2004. Spatiotemporal tuning of motor cortical neurons for hand position and velocity. Journal of neurophysiology, 91(1), pp.515-532.

Yu, B.M., Cunningham, J.P., Santhanam, G., Ryu, S., Shenoy, K.V. and Sahani, M., 2008. Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. Advances in neural information processing systems, 21.