Extracting task-relevant preserved dynamics from contrastive aligned neural recordings

Thank you for your clear summary of our work, and the constructive feedback. Please find below our point-by-point responses to your comments and questions. In all cases, we either added a new experiment or specific clarifications to the text to address your concerns. We look forward to your assessment!

Weakness 1: … it should be made clearer which specific loss terms were used to generate the results shown...

Thank you for this suggestion. We used only the model loss and contrastive loss for Figures 2-3 and S1-S2. We added behavior supervision loss for Figures 5 and S3-S5. So, in total, we use two variations of the algorithm. We will make it clear in the paper by using a separate name for when the behavior supervision loss is used or not: CANDY and CANDY-sup.

Weakness 2: … the paper doesn’t clearly demonstrate how LDS addition can be practically leveraged.

Thank you for making this point. Before discussing interpretability, we want to emphasize that the LDS component in CANDY is not just an add-on, but is central to the scientific question motivating this work: can we extract a preserved latent dynamical system (not just representations!) underlying neural activity across different animals performing similar tasks? While contrastive learning methods like CEBRA can yield behaviorally aligned embeddings, they do not model underlying temporal structure, and therefore cannot test whether shared latent dynamics exist across individuals. CANDY is designed to fill this gap. We chose LDS as a simple, interpretable, and computationally efficient way to model latent dynamics, and because LDSs are effective for local approximations of neural dynamics [1]. Figures 2 and 4 show that contrastive alignment is critical for recovering consistent dynamics, whereas behavioral supervision alone fails to do so, as indicated by weaker decoding from dynamics factors. Moreover, Section 4.2.4 and Figures 5 and S5 demonstrate that an LDS pretrained on multiple animals can be transferred to new animals without retraining. This provides compelling empirical evidence for the existence of shared latent dynamics across individuals.

In terms of interpretability, Fig. 3 shows meaningful topology in the embedding space. In Fig. 3B, latent embeddings colored by wheel velocity reveal a smooth, 1D manifold consistent with task structure. We added the same analysis (will be placed in the Appendix) using LDS dynamical factors and observed similarly structured, low-dimensional geometry aligned with both wheel velocity and trial type. Moreover, we added an analysis on the relationship between the dynamical factors and behavioral variables across sessions. Specifically, we computed the correlation between each dimension of the LDS latent state and the observed wheel velocity across sessions when latent dimension = 8. We found dynamical factors of CANDY exhibited correlations $\approx$ 0.68±0.04 with the behavior, whereas that of DFINE and SLDS exhibited correlations $\approx$ 0.23±0.034 and 0.12±0.001.

Q1

Good point! Beyond behavioral alignment (Figures 3B-C, S2, and S4), we find that CANDY’s latent embeddings also reflect meaningful temporal structure. We added an analysis of pairwise distances within trials and found that temporally adjacent time points remain close in latent space. Plotting Euclidean distance versus temporal separation shows clear temporal continuity in the embeddings, exactly what the LDS was designed to capture. We will add it to the Appendix.

Q2

Yes! We used a behavior supervision loss in the generalization experiments, since the pretrained sessions were unseen, so contrastive comparisons weren’t possible. In L257, we stated that both the LDS and behavior decoder from the pretrained model were frozen. To clarify, we will revise the sentence to “We froze the shared LDS and behavior decoder from the pretrained model and trained only the subject-specific encoder and neural decoder on a held-out third animal using both contrastive and behavior supervision losses (5 sessions).”

Q3

Thanks for the suggestion! We performed this control and the results are as expected. We randomly shuffled behavioral data relative to neural activity 20 times to break their correspondence. Under these conditions, the model fails to converge—the training loss remains high, embeddings lack coherent structure, and behavior decoding drops to chance. Visualizations show no session alignment or interpretable organization, in contrast to Fig. 3B-C. These results confirm that CANDY’s ability to extract structured, behaviorally meaningful embeddings depends on genuine neural–behavioral relationships, not just model architecture. We will include these findings in the Appendix.

Q4

Thanks for sharing this important suggestion! In the current manuscript, we tested whether the learned linear dynamical system and the behavior decoder can be generalized to unseen neural states from new subjects performing similar behaviors on both our mouse wheel-turning dataset (to be released if accepted) and a standard monkey center-out reaching dataset. Results show superior performance of the pretrained LDS and behavior decoder compared to training from scratch on the same sessions.

To address your question, we tested whether the model can generalize to new behavioral states by holding out a movement direction in the monkey dataset. Unfortunately, both CANDY (test $R^2$: mean -0.1209, s.e.m. 0.0319) and CEBRA (test $R^2$: mean -0.2138, s.e.m. 0.0248) failed to generalize to the unseen behavior category, which will be added to the Appendix. This reflects the broader challenge of out-of-distribution generalization in machine learning. We will add the following to the Limitations and Outlook section: "While CANDY shows strong generalization of the pretrained LDS and behavior decoder to unseen sessions, both CANDY and CEBRA failed to generalize to unseen trial types (Appendix), which can be explored in future work."

Limitation 1: … discuss whether preserving behavioral similarity is always desirable… when and where this method is appropriate, and what the limitations ...

Great point! Several recent studies have shown that different subjects can rely on similar neural dynamics within the same brain area to drive both well‑trained behaviors [2] and more naturalistic actions [3]. In our work, we likewise find that both mice in the wheel‑turning task and monkeys in the center‑out-reaching task exploit preserved dynamics. That said, it remains unclear whether such preserved dynamics generalize across all behavioral paradigms or brain regions, especially given that distinct internal processes can sometimes produce identical observable actions.

To address this, we added a synthetic experiment where two sessions were generated from the same spiral dynamics, while the third used limit-cycle dynamics. All three shared the same linear behavioral readout. When trained on all three sessions, CANDY successfully aligned the two spiral sessions, while the third remained misaligned in the latent space. With a shared behavior decoder, the decoding performance is only ($R^2 \approx 0.9, 0.6, 0.1$ respectively), revealing a mismatch in underlying dynamics. Based on the misalignment observed in the latent space, we identified the divergent session and excluded it. Retraining on the two aligned sessions improved decoding back to $R^2\approx 0.99$. This result suggests that a drop in cross‑session decoding performance can serve as an initial indicator of when latent dynamics are not shared.

We will make explicit in the Limitation and outlook section by adding: “While CANDY can facilitate the identification of distinct underlying dynamics as shown in the synthetic experiment, developing methods to distinguish truly distinct neural processes that yield similar behaviors remains an important direction for future work.”

Limitation 2: … How does the method perform on error trials? … whether the model distinguishes correct and incorrect trials … behaviorally similar but cognitively distinct trials are treated differently.

Thank you for this insightful suggestion. As our primary goal in this manuscript is to demonstrate CANDY’s ability to recover preserved, behaviorally relevant dynamics, we have not yet evaluated its performance on error trials. Nonetheless, we agree that probing whether CANDY embeddings can distinguish correct from incorrect trials, especially when the overt behavior remains similar, would be highly informative. Due to time constraints, we will reserve the full analysis of correct versus error trial dynamics for the period leading up to the camera‑ready deadline, if accepted, and include it in the appendix.

We appreciate your thoughtful and challenging questions, which have helped us strengthen both the conceptual and empirical aspects of our work. We hope that our explanation and the newly conducted experiments would address your concerns. We look forward to your reassessment!

[1] Abbaspourazad, Hamidreza, et al. "Dynamical flexible inference of nonlinear latent factors and structures in neural population activity." Nature Biomedical Engineering 8.1 (2024): 85-108. [2] Safaie, Mostafa, et al. "Preserved neural dynamics across animals performing similar behaviour." Nature 623.7988 (2023): 765-771. [3] Nair, Aditya, et al. "An approximate line attractor in the hypothalamus encodes an aggressive state." Cell 186.1 (2023): 178-193.

We thank the reviewer for their constructive feedback throughout their review. Below, we provide point-by-point responses to your questions and comments. In all cases, we either added a new experiment or clarification to the manuscript.

To begin with, we’d like to address a few misunderstood contributions in the summary, which we believe is the main reason behind confusions later in the review.

increase the generalizations across data: Our goal is not to improve behavior decoding generalization across datasets. Instead, we started with a biological question: how to extract the task-relevant preserved latent dynamics, motivated by evidence of shared strategy in certain brain regions shared across sessions while doing similar tasks [1]. We used CANDY as a framework for investigation and demonstrated its effectiveness on two datasets: a new 2p calcium imaging dataset on mouse wheel-turning task collected by ourselves (will be released upon acceptance) and a public monkey electrophysiology dataset.

demonstrates superior performance compared to existing baselines: We are not aiming to improve cross-session behavior decoding performance. Instead, we treat behavior decoding from neural embeddings and dynamic factors as a diagnostic way to evaluate alignment quality and whether a preserved dynamics is correctly captured. If our model matches or exceeds benchmark performance despite biologically motivated constraints, it suggests the preserved dynamics are well learned. Surprisingly, our model shows superior performance, further supporting both the existence of preserved dynamics and our model’s effectiveness.

To emphasize these distinctions, we will revise the contributions at the end of the introduction as:

“Our key contributions include:

1. Methodologically, we develop a new framework to extract preserved dynamics underlying continuous behavior across sessions and animals using a rank-based contrastive objective and a linear dynamical system.
2. Conceptually, we show that contrastive alignment is essential for uncovering preserved dynamical systems, not merely a decoding aid.
3. Biologically, while prior work shows preserved neural motifs across animals [1], none have tested if these can arise from the same dynamical system. We show that a dynamical model trained on one set of animals transfers to held-out individuals, indicating the existence of preserved dynamical structures in a specific brain region under a given task.
4. Experimentally, we collected a new cross-day multi-animal 2p calcium dataset and will release it upon acceptance.”

Weakness1 & Limitation2

Our primary goal is to build a model that extracts preserved, interpretable latent dynamics shared across sessions. Beyond its clear interpretability, CANDY leverages learnable nonlinear encoders to grant the latent space far greater flexibility than a standard LDS would allow [2]. Intuitively, if we map neural activities to latent variables via $z = f(x)$ and impose an LDS prior $\dot z = A z$, the induced dynamics on the observations become $\dot x = (\nabla f(x))^{-1} A\,f(x)$, where $\nabla f(x)$ is the Jacobian of $f$. Since $f(x)$ is learnable and nonlinear, the resulting latent variables induce nonlinear dynamics on the observed neural activity.

“While we focus here with an LDS prior to test our core hypothesis of the existence of preserved dynamics, whether replacing the LDS with different types of nonlinear dynamics in the CANDY framework can be beneficial is an interesting future direction.” We will add this to the Discussion section.

To address this concern empirically, we performed a new synthetic limit cycle experiment, involving nonlinear, nonstationary dynamics. Despite this complexity, CANDY successfully aligns the sessions and recovers the shared 2D limit cycle in the latent space when emphasizing on the contrastive loss (similar as Fig. 2C). This shows that the combination of a flexible nonlinear encoder with a linear dynamics prior is sufficient to recover complex underlying dynamics. We will add this to Fig. 2.

Weakness2 & Limitation1

We believe there is a misunderstanding about the primary motivation of our work and we apologize if the information in the Related work is misleading.

While benchmarking large-scale models on behavior decoding is important, CANDY is not designed to focus on maximizing behavior decoding performance, as we said above. We will add “These transformer-based architectures focus primarily on behavior decoding, whereas CANDY focuses on extracting interpretable preserved dynamics across sessions.” to the Related work.

It is a great suggestion to compare with Stitch-LFADS since it also models a shared dynamical structure. We grid-searched 800 hyperparameters for both datasets independently. The mean±s.e.m of testing $R^2$ are (table entry: using a universal behavior decoder \ session-specific behavior decoder):

Mouse dataset

Monkey dataset

These results indicate that stitch-LFADS failed to align the dynamics across sessions.

Q1

This figure is not meant to show CANDY’s superiority over CEBRA, but to highlight the advantage of those trained with contrastive learning over those without. Specifically, Fig. 3B shows that contrastive methods (CANDY and CEBRA) can align neural latent embeddings across sessions using behavior as an anchor, a property not observed in DFINE, which serves as a baseline without contrastive alignment.

Direct comparisons to CEBRA are shown in Figures 3C-D, S2, S3B and S4. CANDY produces latent embeddings with significantly stronger correlations to ground-truth behavior and higher behavior decoding performance, indicating more interpretable and behaviorally meaningful representations.

Q2

Due to limited access to high-performance GPUs, all experiments were run on CPU. On an AMD Ryzen 9 9950X, training CANDY on 8 sessions (~250 neurons, ~30 time steps each, batch size 2048) takes about 10 hours. For comparison, Stitch-LFADS takes ~8 hours and CEBRA ~6 hours under similar settings. With a high-VRAM GPU, we expect CANDY’s training could be accelerated by an order of magnitude or more.

Q3

Theoretically, both are important questions of current research. Empirically, the answer is yes.

Identifiability: Building on recent work on identifiable VAEs, CEBRA proves that latent variables learned with categorical contrastive loss are linearly identifiable. We apply a contrastive loss suited for regression, while preserving the same structure. We additionally fit a linear dynamical system to model these embeddings and LDS parameters are identifiable up to orthogonal transformations. Therefore, though there is still a gap in the broader literature regarding the identifiability in the case of joint fitting of contrastive objectives and their dynamics, our work builds on existing identifiability proofs in terms of the distinct parts of the model. We will include a discussion of this in the manuscript.

Interpretability: While LDS identifiability means that the exact matrices (e.g., dynamics matrix A) are not uniquely determined, the topology and temporal evolution of the latent trajectories are still meaningful in a task-aligned context, which is precisely our contribution. We aligned the latent variables using the contrastive loss and continuous behavioral signals, and illustrate their interpretability with examples:

• In Fig. 3B, neural latent embeddings colored by wheel velocity reveal a smooth 1D manifold aligned with task structure. To further highlight this, we will update the figure with an additional row colored by trial type (leftward vs. rightward cursor onset), where the two wings of the trajectory correspond cleanly to the two trial conditions, reinforcing the interpretability of the learned latent geometry.
• In Fig. 3C, the distance matrix of CANDY’s latent embeddings (top right) shows a block-diagonal structure that matches the behavioral similarity matrix (top left), indicating that the latent space faithfully encodes behavioral structure.
• The behavioral interpretability is also evident in the center-out reaching task (Fig. S4A), where the latent representations clearly organize the eight reach directions into distinct clusters.

In summary, while LDS and encoder parameters are expected to be linearly identifiable, empirically, the learned latent trajectories are behaviorally aligned, geometrically structured, and interpretable, supporting CANDY’s goal of uncovering shared task-relevant dynamics across sessions. In that regard, our work provides an important empirical contribution to the literature. We hope these clarify your question.

We hope our clarifications on the motivations and contributions of the paper and the choice of LDS address your concerns. If there are any other points we can clarify or further experiments we can conduct to strengthen your assessment, please let us know.

[1] Safaie, Mostafa, et al. "Preserved neural dynamics across animals performing similar behaviour." Nature 623.7988 (2023): 765-771. [2] Abbaspourazad, Hamidreza, et al. "Dynamical flexible inference of nonlinear latent factors and structures in neural population activity." Nature Biomedical Engineering 8.1 (2024): 85-108.

Dear Reviewer, we hope our explanations on the contribution of our work and interpretation of the LDS, additional experiments on stitch-LFADS and nonlinear dynamical systems, and the modification to the Contribution, Related work, and Discussion section address your concerns. Looking forward to your reassessment!

Thank you for this summary, which hits the mark perfectly. We appreciate your support of our manuscript and the constructive feedback throughout the review. Please find below our point-by-point responses to your questions and comments.

Weakness 1: … one dataset receives much more attention in the main text.

An important reason for this is because we collected this dataset ourselves specifically to evaluate generalization in a setting where we had full control over preprocessing and task design. Nonetheless, we show applications to the public monkey dataset, as is the convention in the field currently. We believe that the use and introduction of a new, cross-subject calcium imaging dataset, which will be made publicly available upon acceptance, is a strength of our study and part of the novelty. We hope it provides value to the community by expanding the types of datasets used to evaluate neural dynamics models.

… missing ground truth experiments on out-of-sample data.

We acknowledge this limitation, but emphasize that we showed in Figures 5 and S5 that the LDS and the decoder can be generalized to a held-out animal. To prove this on a dataset with ground truth, we added an additional experiment on the synthetic dataset by helding-out one session that shared the same spiral latent dynamics but different neural observations readout and noise perturbation. By fixing the LDS and tuning the encoder parameters only for the held-out session, we observe that the model can correctly infer the spiral latent trajectory on the held-out dataset, following the pretrained LDS flow field. We will add this result to the revised manuscript.

Weakness 2: The discussion … expanded to include more limitations and relationship to other work on latent dynamics inference and contrastive learning ... Limitations: … clearer about the limitations of their model and when they expect it to fail, … session-specific nonlinearities.

We agree with this assessment. In the initial submission, due to page limit, we had to shorten our discussion section. In the revised manuscript, we will include more work from the broader neuroscience literature that employs contrastive learning:

“Contrastive learning in neuroscience. Recent studies have explored contrastive learning for extracting meaningful neural latent representations. CEBRA [1] embeds neural activity with noncausal convolutional neural network by using contrastive objective on temporal similarity and discretized behavioral similarity. MARBLE [2] introduces a geometric deep learning framework that embeds local flow fields computed from neural activities into a latent space using contrastive objectives. However, it requires post hoc alignment for downstream behavior decoding across sessions. While both methods produce useful low-dimensional representations, they do not explicitly model shared latent dynamics across sessions or individuals.”

and add the following to an explicit Limitations and outlook section which will be included in the manuscript:

“CANDY offers a principled approach for aligning population activity and extracting shared latent dynamics across sessions and subjects, but it has limitations. First, it relies on a linear dynamical system (LDS), which may be too restrictive for highly nonlinear neural dynamics. While Koopman theory suggests that higher-dimensional LDSs can approximate complex systems, we lost the ability to truly reveal the dynamical structure and the latent dimension. Second, CANDY assumes a shared linear behavioral readout across subjects, which may overlook preserved dynamics with nonlinear behavioral readout or subject-specific readouts. Although we found this assumption sufficient in our experiments, incorporating distinct behavioral readouts is a promising direction for future work.”

Q1: Line 112, definition of S_h,j, I think that should be \leq, not \geq. Is that a typo?

No, it should be \geq. Please allow us to elaborate: $S\_{h,j}$ is defined as a set of variables whose distance to anchor $\hat{u}_h$ is larger than the distance between anchor $\hat{u}_h$ and $\hat{u}_j$, and hence are less similar to $\hat{u}_h$ than $\hat{u}_j$. Therefore in the contrastive loss, we pull $\hat{u}_j$ closer to $\hat{u}_h$ compared to other variables in $S\_{h,j}$.

Q2: The ground-truth analyses … where the model breaks down. … if the behavioral readout is highly nonlinear, or … the mapping varies from session to session?

Great question! In the model we presented, we always set the behavior decoder to be linear. To address your question, we added a few more synthetic experiments:

• Same spiral dynamics and same nonlinear behavior readout
• Same spiral dynamics but different linear readouts for each session

We observe that CANDY breaks in both situations. We believe this is an interesting future direction. One potential approach is that contrastive learning cannot just take in the behavior label naively and a behavior encoder might be needed for such a situation for recovering the preserved dynamics. We will add this to the Limitations and outlook section as written in the above response.

Q3: Lines 260-266: ... Is the model trained from scratch only trained on a single session?

We apologize for the confusion. No, the model trained from scratch was trained on all held-out sessions. We will change L263 from “model trained from scratch on the same sessions” to “model trained from scratch on all held-out sessions”.

Q4: ... Is the model robust to different latent dynamics across sessions? Can it be used to discard sessions that are deemed too dynamically different from the other sessions?

Great question! To explore if the model can distinguish sessions with different dynamics, we added a synthetic experiment where two sessions were generated from the same spiral dynamics, while the third used limit-cycle dynamics. All three shared the same linear behavioral readout.

When trained on all three sessions, CANDY successfully aligned the two spiral sessions, while the third remained misaligned in the latent space. The decoding $R^2$ value for each dataset is only $\approx 0.9, 0.6, 0.1$ respectively, revealing a mismatch in the underlying dynamics. Based on the misalignment observed in the latent space, we identified the divergent session and excluded it. Retraining on the two aligned sessions improved decoding back to $R^2\approx 0.99$.

This result suggests that CANDY can not only recover shared latent dynamics but potentially also serve as a diagnostic tool to flag sessions or subjects that deviate in their dynamical strategies. We see this as an exciting direction for extending this work. We will include the new synthetic experiment in the Appendix and add the following to the above Limitation and outlook section: “Third, while CANDY could facilitate the identification of distinct underlying dynamics as shown in the synthetic experiment, distinguish different neural processes that yield similar behaviors is an interesting extension to this work.”

We appreciate your thoughtful and constructive questions, which have helped us strengthen both the conceptual and empirical aspects of our work. We hope that our explanation and the newly conducted experiments would address your concerns. We look forward to your reassessment!

[1] Schneider, Steffen, Jin Hwa Lee, and Mackenzie Weygandt Mathis. "Learnable latent embeddings for joint behavioural and neural analysis." Nature 617.7960 (2023): 360-368. [2] Gosztolai, Adam, et al. "MARBLE: interpretable representations of neural population dynamics using geometric deep learning." Nature Methods (2025): 1-9.

Great point! We apologize for the confusion in the text and will add the following sentences (bold) to the paragraphs in L110-115.

Given an anchor $\hat{u}_h$ with the associated behavior $y_h$, for any other embedding $\hat{u}_j$, we define a set of embeddings that have higher ranks than $\hat{u}\_j$ in terms of the behavioral label distance with respect to the anchor as $S\_{h, j} := \\{\hat{u}\_k \mid k\neq h, d(y\_h, y\_k)\geq d(y\_h, y\_j)\\}$, where $d(\cdot, \cdot)$ is the $L_1$ distance between the two behaviors. In other words, $S\_{h,j}$ is defined as a set of embeddings whose distance to the anchor $\hat{u}_h$ is larger than the distance between the anchor $\hat{u}_h$ and $\hat{u}_j$ , and hence are less similar to the anchor $\hat{u}_h$ than $\hat{u}_j$. Define the "likelihood'' of $\hat{u}_j$ given the anchor and the higher rank set as:

Eq. (1)

where $\text{sim}(\cdot, \cdot)$ is the similarity measure between two latent embeddings (e.g., negative $L_2$ norm) and $\tau$ is the temperature parameter. Here, the likelihood will be maximized to push $u_j$ closer to the anchor $u_h$ in the embedding space than any other embeddings in the rank set.

We sincerely thank the reviewer for the thoughtful engagement with our work and for the updated score. We’re especially grateful that you found the results convincing and recognized the potential impact of our model in neuroscience. Your suggestions have significantly helped us strengthen both the conceptual framing and empirical validation of our work. We truly appreciate your support and the time you put into the review process!

Thank you for the summary and your constructive feedback. Please find our point-by-point responses to your comments and questions below. In all cases, we either performed new experiments or added clarification to the text to address your concerns. We look forward to hearing your assessment.

Weakness 1: The various components of the method are well-justified: … so I consider CANDY to have modest novelty.

While these are indeed important building blocks of our work, we respectfully note that CANDY’s contributions go well beyond simply combining standard modules:

First, our methodological advance lies in developing a new framework to extract preserved dynamics underlying continuous behavior across sessions and animals using a rank-based contrastive objective and a linear dynamical system. This shows two specific novelties compared to previous methods: (1) this goes beyond the limitation of other behaviorally-anchored neural embedding methods, such as CEBRA, which have to discretize behaviors before applying the contrastive objective; (2) the behaviorally-anchored neural embeddings are also constrained by a linear dynamical system, which assists the learning of the preserved dynamics by maintaining the temporal structure of the embeddings within a trial.

Second, as a conceptual novelty, we show that contrastive alignment is essential for uncovering preserved dynamical systems, not merely a decoding aid. This is especially important for neuroscience as neural recordings show different degrees of heterogeneity, including number of neurons and the exact location of the recording. In Figures 2 and 4, removing the contrastive term causes the latent trajectories and the LDS to collapse. This highlights a new role of contrastive learning: as an alignment tool for uncovering preserved dynamical-structure across sessions, rather than just representation learning as in CEBRA.

Third, our work addresses an important biological question: while earlier work has shown that neural activity motifs are preserved across animals performing the same tasks [1], no work to date has investigated whether these motifs can be supported by the same dynamical system equations. Here, we show that an LDS trained on one set of animals transfers directly to held-out individuals, indicating the existence of preserved dynamical structures in a specific brain region during a specific task. In Figures 5 and S5, we froze both the learned LDS and the behavior decoder and still achieved high decoding performance on unseen animals/sessions. This demonstrates that the latent dynamics inferred by CANDY are not only interpretable, but also consistently preserved across subjects in the mouse striatum during the wheel-turning task and in the monkey motor cortex during the reaching-out task. This is important because this shows a unique cross-subject generalization of dynamical models of the animal brains.

Finally, on the experimental side, we designed and collected a new 2p calcium cross-animals dataset ourselves, which includes 3 animals with 5 sessions each (15 sessions total), and each session contains ~200 trials with recordings from ~200 neurons in the striatum. We will make the dataset available to the community if accepted.

We hope these clarifications would address your concerns of the originality of this work.

Weakness 2: MARBLE … should be cited. … discussion of the specific ways in which MARBLE and CEBRA differ from CANDY… the related works section.

Thank you for referencing these papers. We agree that contrastive learning is a rapidly evolving area in neuroscience and appreciate the opportunity to better position our method in this context. We will add the following paragraph to the Related Work section:

“Contrastive learning in neuroscience. Recent studies have explored contrastive learning for extracting meaningful neural latent representations. CEBRA [2] embeds neural activity with noncausal convolutional neural network by using contrastive objective on temporal similarity and discretized behavioral similarity. MARBLE [3] introduces a geometric deep learning framework that embeds local flow fields computed from neural activities into a latent space using contrastive objectives. However, it requires post hoc alignment for downstream behavior decoding across sessions. While both methods produce useful low-dimensional representations, they do not explicitly model shared latent dynamics across sessions or individuals.”

Weakness 2: BLEND also shares similar aims to CANDY.

BLEND develops a model-agnostic paradigm to distill behaviorally relevant neural latent embeddings from an existing model for use on new datasets lacking behavior signals. In contrast, CANDY develops a framework to extract preserved dynamics across animals by using behavior as an anchor to align heterogeneous neural populations (e.g., number of neurons and location of the recordings). We will add the following sentences to the Related work section: “The BLEND framework [4] uses a teacher-student setup to distill behaviorally relevant information to student networks when there is a lack of behavior data in a new dataset, but does not focus on extracting shared task-relevant dynamics across animals.”

Weakness 3 & Q1: Is the LDS useful at all? …

This is a critical aspect of our work and you are correct that the LDS is not designed to improve behavior decoding performance. Below, we highlight two distinct reasons to address why LDS is useful:

First, the LDS module is essential for addressing an important scientific question: Can we extract a preserved and latent dynamical system (not just representations!) underlying neural activity across animals performing similar tasks? While contrastive learning methods like CEBRA can produce behaviorally meaningful embeddings, they do not model the underlying dynamics and thus cannot determine whether shared dynamical motifs exist across subjects. This is precisely what CANDY was designed to address. More importantly, in Section 4.2.4, Figures 5 and S5, we demonstrate that a pretrained LDS learned from several animals generalizes to new animals with high decoding performance, without retraining the LDS. This provides biological evidence that the neural activities in the same brain region can exhibit shared dynamics across individuals in similar well-trained behaviors.

Second, LDS is a key component because our methodological novelty lies in extracting a preserved, interpretable dynamical system via behaviorally aligned neural embeddings, not in boosting behavior decoding performance using LDS. As shown Figures 2 and 4, omitting the alignment step prevents recovery of a coherent LDS. Fig. 4 further illustrates that contrastive objective is critical for uncovering preserved dynamics: replacing contrastive loss with behavior supervision achieves similar decoding from embeddings but fails to recover meaningful dynamics, as seen in much worse decoding from the latent dynamical factors. To our knowledge, this is the first demonstration of behavior guided alignment is essential for isolating latent dynamical systems that generalizes across both sessions and animals.

In summary, the LDS module is essential for our scientific goal of testing shared dynamical motifs across individuals. In the revised manuscript, we will add these clarifications to directly address this concern. We welcome further questions and look forward to your reassessment.

Q2: Will the wheel turning data be made public upon publication?

Yes! We are committed to open science and will release the striatal 2p calcium imaging data upon acceptance. We hope you will consider this as an additional crucial contribution of the paper to the community.

Q3: Fig. 4a and 4b … for easy comparison.

Great suggestions! For easy comparison, we will put the first column of Fig. 3B to Fig. 4A and the top right similarity matrix in Fig. 3C to Fig. 4B. Then we will move the C-E to the third row in Fig. 4.

Limitations

Training a session-specific encoder is not a limitation but a necessary design choice due to the substantial heterogeneity in neural recordings across sessions. Behavior signals serve as a key for aligning latent neural dynamics across sessions. Without them, embeddings drift arbitrarily in latent space. As shown in Fig. 4, when both contrastive and behavioral losses are removed, thus removing behavioral supervision, the embeddings fail to align, resulting in failure to recover consistent underlying dynamics. Existing post hoc alignment methods similarly depend on behavioral information to align latent embeddings across sessions [1,3], further underscoring the necessity of behavioral signals. Moreover, adapting to new sessions with a new encoder, as demonstrated in Figures 5 and S5, only requires a few trials of data.

We hope our clarifications on the methodological, conceptual, biological, and experimental novelties, along with the added content in the Related work and Discussion sections, and the explanation of the usefulness of LDS address your concerns. If there are any other points we can clarify or further experiments we can conduct to strengthen your assessment, please let us know.

[1] M. Safaie, et al. "Preserved neural dynamics across animals performing similar behaviour." Nature 623.7988 (2023): 765-771. [2] S. Schneider, Steffen, J. H. Lee, and M. W. Mackenzie. "Learnable latent embeddings for joint behavioural and neural analysis." Nature 617.7960 (2023): 360-368. [3] A. Gosztolai, et al. "MARBLE: interpretable representations of neural population dynamics using geometric deep learning." Nature Methods (2025): 1-9. [4] Z. Guo, et al. "BLEND: Behavior-guided Neural Population Dynamics Modeling via Privileged Knowledge Distillation." The International Conference on Learning Representations (2025).