Neural Representational Consistency Emerges from Probabilistic Neural-Behavioral Representation Alignment

We sincerely appreciate the reviewer's thoughtful feedback and the positive assessment. We address each point as follows:

Q1. The langrangian arguments could do with more detail.

A1: Thanks for this suggestion. In our revision, we will add a more comprehensive derivation of the Lagrangian multiplier method, explicitly clarify how constraints are transformed into optimization objectives, and provide mathematical explanations for key steps in the derivation process.

Q2. The authors state that the neural representations exhibit minimal correlation with non-corresponding pairs of neural data and behavioural data in figure 3b. But this doesn’t seem so? The correlations between all trials are somewhere between 0.8 and 0.95, including unmatched neural and behavioural data (from my understanding of the plot).

A2: Thank you for this observation. Correlations in Figure 3b are indeed generally high due to our compact latent space. Our intention was to emphasize trial discriminability, with diagonal elements forming distinguishable patterns in most cases. In our revision, we will better characterize successful cases and emphasize trial-level discriminability.

Q3.I don’t see how the method would be ‘present significant potential for advancing calibration-free neural decoding systems’ (line 464). There would still need to be trials from a new animal or session to internally learn the alignment.

A3: PNBA extracts preserved neural representations in zero-shot individuals (Sections 4.3-4.4), allowing us to train both PNBA models and BCI decoders on known individuals and directly apply them to new individuals without any fine-tuning or calibration (Section 4.5). Thus, we believe this demonstrates potential for calibration-free BCI applications.

Q4. The downside is that the full trial must be input to match with behaviour, hindering online decoding that is important in BCI applications.

A4: In experiments from Section 4.5, we can adopt a sliding window approach with most BCI decoders, especially Linear and MLP, supporting online decoding. While not our primary focus, this would address the concern. We will add this discussion in our revision.

Q5.Unclear how correlations are performed, are they on latent space embeddings?

A5: All correlation calculations are indeed performed on the low-dimensional latent representations z, as they have the same spatial dimensions. We will clarify this explicitly in our revision.

Q6.Recently published Gosztolai et al. MARBLE. (2025.3) paper seems relevant.

A6: Thank you for this recommendation. While both MARBLE and our PNBA aim to reduce dimensionality of neural activity, there are fundamental differences. MARBLE is a single-modal method focusing on geometric constraints and temporal dynamics on neural activity, while PNBA is a multimodal strategy directly incorporating behavioral data as constraints in latent space.

We'll add this related work in our revision.

Q7. I wonder whether there is some combined approach with sequential methods like LFADS or GPFA that could be proposed? Since at the moment it does not explicitly model temporal dynamics.

A7:We agree and believe such a combination is entirely feasible and would address the current limitation in modeling temporal dynamics. PNBA provides multimodal alignment constraints while methods like LFADS/GPFA offer complementary temporal modeling capabilities. These represent independent constraints that could be directly combined in future work.

We'll include this as limitation and state this future work in our revision.

Q8. It was unclear to me what the normalized representation values are? Are these the latent space activations? Unit vector normalized?

A8:'Normalized representation values' refers to z-score normalization applied independently to each trial's representation, making different modalities comparable by eliminating scale differences. We'll clarify this in our revision.

Q9. I wonder how the authors would approach such 'multiple solutions' with PNBA? Since in such a case, I would imagine that the low-dimensional neural representations wouldn’t align?

A9: PNBA directly addresses the "multiple solutions" challenge. Rather than forcing identical representation for the same behavior, our approach:

1. Avoids assuming one-to-one correspondence between neural activity and behavior, instead employing distributional matching that preserves neural diversity.

2. Ensures that PNBA must generate different neural representations for the same behavior (Theorem 1, Property 1), otherwise it would result in representation collapse.

3. Establishes a lower bound on representational similarity that ensures differences and an upper bound to promote feasible similarity (Theorem 1, Property 3).

Therefore, these multiple considerations make PNBA particularly suited for studying the many-to-one mapping between neural activity and behavior.

We thank the positive assessment and constructive feedback. We address each point as follows:

Q1. Decoding behavior would better demonstrate captured behavioral information. Authors should discuss limitations on real-time inference and computational demands for BCI.

A1: Behavior decoding results are as follows: M1, R²=0.89, PMd, R²=0.83 (training subjects); M1, R²=0.78, PMd, R²=0.71 (testing subjects), indicating capture of behavioral information. These are expected as PNBA aligns neural-behavioral representations. To avoid circular reasoning, Section 4.5 presents movement decoding using V1 data, where the neural encoder was only trained with stimulus in PNBA, excluding movement data.

For BCI discussions, training takes ~1 hour (M1/PMd) and ~8 hours (V1), detailed in SI Line 1075, with inference requiring 0.3ms and 1.4ms, respectively (averaged over 1000 runs on an A100 GPU), showing potential for BCI.

We will add these discusions in our revision.

Q2. Simulations could reveal model limitations. Can the model generalize across missing behavioral conditions or recover ground truth parameters?How robust is it to latent space dimensionality, trial misalignment, or individual variability?

A2: Thanks for this suggestion. We agree with the usefulness of simulation. However, building simulations for neural-behavioral modeling with ground truth representation constraints is inherently difficult due to the complex, unknown nature of true neural encoding mechanisms. We therefore validated PNBA across three diverse real datasets.

Regarding specific questions:

a. No, PNBA cannot generalize to missing behavioral conditions in the raining. This is a current limitation, as discussed in Section 6.

b. PNBA doesn't aim to recover ground truth parameters, as these are difficult to define in real neural data. PNBA is data-driven, learning effective representation alignment instead, similar to CLIP.

c. We show robustness through theoretical guarantees (Theorem 3) and empirical validation. Figure 4b shows the results of varying latent space dimensions. Figures 3b,6c show effective handling of trial-to-trial variability in unseen subjects.

Q3. The authors missed a reference, Herrero-Vidal et al. NeurIPS 2021.

A3: Thanks for suggesting this work.

Herrero-Vidal et al. assumes similar neural trajectories across individuals responding to identical stimuli so as to align single-modal neural activity through individual-specific optimization. In contrast, our multimodal neural-behavioral alignment doesn't presuppose neural encoding similarity, and can be tested across individuals. This methodological difference allows us to validate preserved neural representations in unseen subjects.

We believe our findings provide empirical support for assumptions in Herrero-Vidal et al., supporting shared neural representations under the same behavior.

We will add these discussions in our revision.

Q4. Compare with Procrustes alignment methods (Williams et al. 2021, Safaie et al. 2023).

A4: Thanks for this suggestion. We didn't directly compare with Williams et al. and Safaie et al. for several reasons:

a. Our work proposes cross-modal (neural-behavioral) alignment, while these methods focus on unimodal (neural-neural) alignment, making direct comparison potentially unfair.

b. Our method requires no fine-tuning on unseen individuals, whereas these methods require individual-specific optimization.

c. Different foundational assumptions (A3).

We will include these discussions in our revision.

Q5. Add a section discussing limitations.

A5: Limitations are currently discussed in Section 6, paragraph 1, noting PNBA cannot train with neural activity alone. We will add temporal dynamics discussion based on Reviewer DJVs's suggestion.

Q6. Compare with linear alignment methods regarding decoding performance, data demands, and computational costs for BCI applications.

A6: For computational costs and linear alignment comparisons, please see A1 and A4. Regarding data requirements, PNBA is currently trained with 2 monkeys (M1/PMd) or 8 mice (V1), indicating moderate demands. While BCI applications aren't our current focus, our results show potential, and we plan dedicated BCI research in future work.

We will add these discussions in our revision.

Q7. How much data is needed for initial training? How does the transformation for handling different neuron counts work, and how much data is used for pre-alignment?

A7: Our model requires no pre-training, but trains directly on data from multiple individuals in the training set and directly tests on new individuals without fine-tuning.

To handle varying neuron counts, we use a shared convolutional projection head followed by pooling to only unify dimensions in our network. This requires no separate pre-alignment data or process, as it's learned directly during end-to-end training.

We sincerely thank the reviewer for the constructive feedback. We address each point as follows:

Q1. Line 382-384’s claim is confusing. How is the calcium imaging encoder trained?Is the zero-shot experiments still the cross-subject zero-shot experiments?Is the behavioral encoder frozen?If both encoders are, the evidence does not suggest broader preservation.

A1: We appreciate the reviewer's careful examination.

a. Our conclusion summarizes preservation properties observed in three independent experiments (M1, PMd, V1) during zero-shot cross-subject conditions. We will clarify this in our revision.

b. The calcium imaging encoder follows the same PNBA framework but with visual stimulus encoders. The zero-shot evaluations are also performed on unseen subjects without any fine-tuning or further alignment process.

c. During training, all encoders are optimized.

d. We understand the concern about preservation claims. However, our conclusion is supported by (1) consistent preservation observed in functionally different brain regions (M1, PMd, V1); (2) such validation are performed on unseen subjects without further finetuning or alignment. We believe this rigorous validation provides evidence for the preservation property.

Q2. The proposed matching objective in Section 3.1 is only one necessary. Is reversibility of encoders a sufficient condition for alignment?The theoretical guarantees seem too loose for sufficient alignment.

A2: We agree with the reviewer's insight. While reversible encoders would provide an optimal sufficient condition for alignment, this becomes impractical for neural data requiring dimensionality reduction. Our generative constraint approach offers a practical sufficient condition that balances theoretical guarantees with implementation feasibility, achieving effective representation alignment on multiple datasets without requiring strict encoder reversibility.

Q3. What assumptions underlie using Pearson correlation for alignment quality?(1)How is it applied to single-trial data?Is it per coordinate?Are vectors centered?(2)Did you try other metrics like L1 distance?Are all baseline methods optimized based on the given evaluation metrics?

A3: The core assumption behind using Pearson correlation is that aligned neural and behavioral representations should exhibit similar distributional structures in latent space.

(1) We compute correlation between complete neural-behavioral latent vectors for each trial, not coordinate-wise. Pearson correlation centers the vectors and normalizes by standard deviation, distinguishing it from cosine similarity.

(2) We considered alternatives (L1, Euclidean) but selected Pearson correlation because it's established in neuroscience for representational similarity analysis (e.g., Safaie et al., Nature 2023) and effectively captures structural similarities in high-dimensional spaces.

All methods (baselines and our PNBA) were optimized using their original objective functions, not optimized the evaluation metrics.

We will include these details in our revision.

Q4. Limited details are provided in the main text regarding how to handle different neuron counts across sessions and how to transfer encoders given different input sizes. Are encoders re-trained when initialized on the data from a new animal? How do you deal with unseen neurons in new animals/sessions?

A4: Thank you for highlighting this important issue. We addressed varying neuron counts through a neuron-adaptive encoder architecture incorporating projection layers with pooling layer (SI Line 1091-1093) to get unified dimensional latent space. This enables direct application to new animals with different input sizes.

No, encoders are not re-trained when applied to new animals.

Our approach handles unseen neurons effectively because: (1) the neural encoder unifies the latent space dimensionality, and (2) based on PNBA, training across multiple subjects with diverse neural populations encourages extraction of behaviorally-relevant features rather than memorizing individual neuron characteristics, enabling generalization to completely new neural populations. We will incorporate these details into the main text.

Q5. Suggested SwapVAE

A5: Thank you for suggesting this reference. While both approaches address neural representations, fundamental differences exist between SwapVAE and our PNBA framework. DropVAE operates within a single modality (neural spikes), using data augmentation and swap operations predicated on within-trial similarity, without directly incorporating behavioral constraints. In contrast, PNBA draws inspiration from CLIP's multi-modal paradigm, explicitly aligning neural and behavioral representations through generative constraints. This cross-modal approach directly establishes neural-behavioral associations, enabling zero-shot generalization to unseen subjects, as detailed in A4. We will add this in our related work.

We appreciate the reviewer's detailed feedbacks! We address each point as follows:

Q1. How does zero-shot generalization work with varying neuron counts across subjects?

A1: Our model uses convolutional and pooling layers to standardize input activities of varying counts into a fixed-size latent space(detailed in SI Line 1088-1094). Based on PNBA, training on multi-subject further forces the model to learn behavior-relevant neural representations rather than individual-specific neuron characteristics, enabling cross-individual generalization without any fine-tuning on zero-shot subject.

Q2. Safaie et al, 2023 was not compared in Table 1.

A2: Table 1 excludes Safaie et al. because they focus on neural representation alignment requiring individual-specific optimization, whereas we evaluate cross-modal correlations on unseen subjects without fine-tuning. A direct comparison would be unfair to theirs.

Q3. Figure 3a: Was histogram from aggregated data of two mice?What do individual mouse histograms look like?

A3: Yes, the histogram aggregates data from 2 mice viewed identical stimuli. Individual mouse histograms show the same patterns for stimulus representation, with subtle peak variations in neural representation due to individual differences.

Q4. Figure 3a:What samples construct the histogram?How are normalized values computed and varying trial lengths/population sizes handled?

A4: The histogram uses latent representations z (d×T) from each trial, reshaped into one-dimensional vectors and z-score normalized. Neuron count variations are addressed by pooling neural features into a fixed-dimensional latent space (SI Line 1088-1094). Variable trial lengths are handled through sliding window (t=16) processing (SI Line 1067). All vectors across trials are then concatenated into a one-dimensional array, whose frequency distribution forms the histogram.

Q5. Figure 3b: Why is the correlation matrix not symmetric?

A5: The correlation matrix is asymmetric due to trial asymmetry. Specifically, corr_matrix[i,j] measures correlation between neural activity of trial i and visual stimulus of trial j, noted as pair (i,j), while corr_matrix[j,i] measures pair (j,i), a different combination, naturally yielding different values. Similarly in Figure 6c, pairs (i,j) and (j,i) represent different trial combinations.

Q6. Line 247: How are the 4 pairs chosen from the 8 mice?Shouldn't there be 8 choose 2 total number of pairs?

A6: We apologize for the confusion. Our experiment included 5 pairs of mice (10 mice total), with each pair viewing identical stimuli. 4 pairs (8 mice) were used for training, while the remaining pair (2 mice) served as an independent test set to evaluate zero-shot performance (see SI Table 4). We will refine this in the revised version.

Q7. Figure 5a: is mean R calculated across all trial pairs regardless of behavior condition?How are different trial lengths handled?What is the standard deviation calculated over?

A7: No, we calculate mean R only between trials with matching behaviors, following Safaie et al, 2023. For trials with different lengths (d×T1 vs d×T2), e.g. T2>T1, we downsample T2 by uniformly discarding timepoints. This length handling only applies to monkey data, as all V1 trials have consistent lengths.

Standard deviation is calculated across all possible same-behavior trial pairs.

We will add these clarifications in our revision.

Q8. Figure 5b and 5c: Similar question to the above.How are the mean R and standard deviation calculated for across-session and across-subject?

A8: We use identical strategy as A7, but comparing trials from different sessions or different subjects while maintaining matched behavioral conditions.

Q9. Section 4.5: How can the model handle zero-shot behavior decoding with varying neuron counts and time steps?What's the detailed training and inference procedure?

A9: Neural representations have the same spatial dimensions across subjects (A1). V1 trials have consistent lengths, but we can use sliding windows to handle varying lengths (SI Line 1067).

The full procedure:(1) train PNBA on training subjects (2) generate representations via the frozen neural encoder, validate preservation (3) train V1 BCI decoders using training subjects' representations and behavior (4) apply frozen neural encoder and BCI decoder to unseen subjects.

Q10. Section 4.5: Could the authors also provide results of behavior decoding on the primate datasets?

A10: Our primate decoding results on unseen animals: M1 achieved R²=0.78; PMd achieved R²=0.71. We note that these results are expected as PNBA aligns neural-behavioral representations for these datasets. To avoid circular reasoning, we showed independent movement decoding using V1 data in Section 4.5, where the neural encoder was only trained with stimulus in PNBA.

Others: We have fixed all typos.