Task-Optimized Convolutional Recurrent Networks Align with Tactile Processing in the Rodent Brain

Thank you for taking the time to review and provide your valuable feedback! We address the weaknesses and questions as follows.

Passive sweeps vs. active whisking and including neural fit shapes during training

This is a great question. We are already generalizing because the shapes that are used in the neural evaluation are not present, even in distribution, in the ShapeNet training data. The reason why we chose to do passive sweeps for the large sweeping dataset on ShapeNet is that, especially with the temporal flipping used in the tactile augmentations, active whisking is isomorphic to doing multiple passive sweeps. Therefore, passive whisking is a more scalable implementation of active whisking, which is important for generating pretraining dataset diversity, which is in turn is important for matching brain data by the “Contravariance Principle” [1, 2, 3].

[1] R. Cao and D. Yamins. “Explanatory Models in Neuroscience, Part 1: Taking Mechanistic Abstraction Seriously.” Cognitive Systems Research, vol. 87, Sep. 2024, p. 101244. ScienceDirect, https://doi.org/10.1016/j.cogsys.2024.101244.

[2] D. Yamins and J. DiCarlo. “Using Goal-Driven Deep Learning Models to Understand Sensory Cortex.” Nature Neuroscience, vol. 19, no. 3, Mar. 2016, pp. 356–65. PubMed, https://doi.org/10.1038/nn.4244.

[3] A. Nayebi*, N. C. Kong*, C. Zhuang, J. L. Gardner, A. M. Norcia, and D. L. Yamins. “Mouse Visual Cortex as a Limited Resource System That Self-Learns an Ecologically-General Representation.” PLoS Computational Biology, vol. 19, no. 10, Oct. 2023, p. e1011506. PubMed, https://doi.org/10.1371/journal.pcbi.1011506.

Low number of stimuli and confidence intervals for the noise ceiling

The mouse neural dataset we are evaluating against only has 2 different shapes and 3 distances which we can use as unique stimuli. This was the most recent viable candidate for somatosensory cortex data that was available. We agree this is an unfortunate limitation that reduces the generalizability of our findings; though we note that low numbers of stimuli are a common occurrence in neuroscience and our task-optimized modeling approach, along with our inter-animal consistency analysis, highlights the need for future neural datasets to move forward in this concrete direction of collecting more stimuli. This was noted in the “Limitations and Future Directions” paragraph in the Discussion section: “Most importantly, current tactile neural datasets remain limited in stimulus diversity and the number of object conditions tested, restricting the captured neural variability and could be the reason why inter-animal consistency values are lower for the statistical average between animals than the current pointwise empirical maximum of 1.3.” (Note: Statistically, Pearson's correlation value (r) is within the range [-1, 1] but the split-half noise correction can result in values >1 when self-consistency is low.) Finally, since we train the models on the large-scale tactile whisking task rather than fit to the neural data directly, we note that our models can be evaluated without change on future neural datasets that collect more stimuli. Therefore, our models and results here are not the final say, but a platform for future exploration and evaluation.

Statistical significance of model comparisons across mice

The error bars in Figure 4 show the SEM of the median neural fit score across animals. Additionally, we find that the Wilcoxon significance on the neural score across 11 mice for Inter+SimCLR (best neural score model) & Resnet+Mamba+SimCLR (best task score out of self-supervised models) gives statistic=2.0, p=0.0425 which is significant (p<0.05). We will include this additional result in the paper!

Search budget and computational efficiency

We already report the GPU hours in line 191 in the main text and the training details (e.g., hyper-parameter tuning strategy) in appendix A2 Model Training. We will provide a more detailed description on the search and training cost in our revised paper.

Missing details in main text & relevance of robotics in introduction

With the additional page provided for the camera-ready version, we are happy to include the augmentation hyper-parameters, exact RSA equation, IntersectionRNN definition in the main text. We will also revise the robotics-hardware history in the introduction section to clarify the primary message and result of our work.

Impact on mainstream ML and contribution to architectural insights for temporal tactile modeling

One of our main contributions is the Encoder-Attender-Decoder (EAD) framework, which enables comprehensive and systematic model search with diverse neural network architectures. Despite the similar architectural conclusion, our tactile processing is different from prior vision works as the input of the model is forces/torques other than pixels that are easier to understand. We also found that we had to create our own data augmentations for self-supervised tactile learning, as typical image augmentations did not scale for this dataset [Figure 5c]. Furthermore, to the best of our knowledge, we are the first to compare tactile models to brain data for neural alignment test; hence, our findings are well-supported by both the EAD framework and the neural alignment test for temporal tactile processing.

The EAD architecture as a unified search framework

As mentioned in abstract line 3-4 and line 166-168, we agree that the EAD framework offers a comprehensive solution that allows searching through different model architectures other than a new modeling concept for tactile processing.

Attention module analysis and performance gains

Compared to the choice of encoders and augmentations, the performance gain of different attention modules are indeed relatively small, as discussed in paragraph “GPT-based Attenders Provide Modest Improvements in Task Performance and Neural Alignment” in line 274-285, hence, more analyses are focused on the encoders and augmentations.

Thank you for taking the time to review and provide your valuable feedback! We address the weaknesses and questions as follows.

Low number of stimuli

The mouse neural dataset we are evaluating against only has 2 different shapes and 3 distances which we can use as unique stimuli. This was the most recent viable candidate for somatosensory cortex data that we could find.

Model neural fit scores compared to a2a baseline

All models are indeed above the mean animal-to-animal (a2a) fit, but below the maximum a2a baseline. This is noted in the caption but we will update the figure to make this clearer. (Note: Statistically, Pearson's correlation value (r) is within the range [-1, 1] but the split-half noise correction can result in values >1 when self-consistency is low.)

We noted both of these limitations in the “Limitations and Future Directions” paragraph in the Discussion section (lines 351-354): “Most importantly, current tactile neural datasets remain limited in stimulus diversity and the number of object conditions tested, restricting the captured neural variability and could be the reason why inter-animal consistency values are lower for the statistical average between animals than the current pointwise empirical maximum of 1.3.”

Thank you for your question! Randomly initialized networks are highly non-random functions as they are architectures selected to perform the task well when trained. Therefore, architecture matters a lot, and this is a well-noted phenomenon in NeuroAI across brain regions [1, 2, 3]. By doing this comparison in our manuscript, it allows us to isolate the contributions of the architecture vs. loss function for every pair of (architecture, loss function) tuples. The models that saturate our noise ceiling (bars on far right in Fig. 4b) are noticeably improved when trained. Thank you for asking this, we will include this explanation in our final revision.

[1] Yamins, D. L., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014). Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23), 8619-8624.

[2] A. Nayebi*, N. C. Kong*, C. Zhuang, J. L. Gardner, A. M. Norcia, and D. L. Yamins. “Mouse Visual Cortex as a Limited Resource System That Self-Learns an Ecologically-General Representation.” PLoS Computational Biology, vol. 19, no. 10, Oct. 2023, p. e1011506. PubMed, https://doi.org/10.1371/journal.pcbi.1011506.

[3] Schrimpf, M., Kubilius, J., Lee, M. J., Murty, N. A. R., Ajemian, R., & DiCarlo, J. J. (2020). Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron, 108(3), 413-423.

Thank you for taking the time to review and provide your valuable feedback! We address the weaknesses and questions as follows.

Low number of stimuli

The mouse neural dataset we are evaluating against only has 2 different shapes and 3 distances which we can use as unique stimuli. This was the most recent viable candidate for somatosensory cortex data that was available. We agree this is an unfortunate limitation that reduces the generalizability of our findings; though we note that low numbers of stimuli are a common occurrence in neuroscience. Our task-optimized modeling approach, along with our inter-animal consistency analysis, highlights the need for future neural datasets to move forward in this concrete direction of collecting more stimuli. This was noted in the “Limitations and Future Directions” paragraph in the Discussion section (lines 351-354). (Note: Statistically, Pearson's correlation value (r) is within the range [-1, 1] but the split-half noise correction can result in values >1 when self-consistency is low.) Finally, since we train the models on the large-scale tactile whisking task rather than fit to the neural data directly, we note that our models can be evaluated without change on future neural datasets that collect more stimuli.

Discrepancy between model-to-animal and animal-to-animal alignment

The model-to-animal neural score is obtained by using the maximum alignment score across the model to each animal. As a result, the models indeed achieve a score above the mean animal-to-animal (a2a) fit, but all are also below the maximum a2a baseline. (This is noted in the caption but we will update the figure to make this clearer.) In other words, the variability between animals is very high but there is always an animal pair that do very closely match. The inter-animal consistency score calculation is included in the Appendix (A3, Eq. 1). It is the RSA score between the response patterns for each pair of animals.

Distinguishing different types of stimulus and variations

No, we find that matching the neural data is not simply explained by distinguishing convex from concave; but rather, the models more reliably decode near-medium-far, suggesting that there is more variability along this axis for neural predictivity, than merely convex vs. concave. Below, we show that the decoding performance of the best models is above chance for near-medium-far. For each of the 6 stimuli, we train a logistic regression model on the 5 other stimuli and test on the 1 held out stimuli. While the number of stimuli is too low to make a definitive claim, interestingly our most task performant self-supervised model had the highest stimuli decoding score.

Finally, a figure of the RDMs for the best neural fitting supervised and self-supervised model is already included in the appendix (Figure A3.)

Task performance measurement other than “top-5 accuracy”

For self-supervised learning, all the models are first pre-trained with self-supervised losses. After we have the best pre-trained checkpoint, we freeze the model weights and add a trainable classification head, which is a common approach for evaluating downstream task performance for self-supervised models [1]. This assembled model is then finetuned with labels in a supervised fashion and we save the best model based on its accuracy on the validation set and report the test accuracy (see line 187-189); hence, reporting the task performance under “top-5 accuracy” is consistent for both supervised and self-supervised learning. In our preliminary experiments, we tried self-supervised learning with the instance recognition loss [1], however, the training is highly unstable and the results are not satisfactory, which suggests that in instance recognition as an objective/measurement is less promising for tactile processing. Thank you for bringing this up, we will mention this in our revised paper.

[1] Wu Z, Xiong Y, Yu SX, Lin D. Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018. p. 3733–3742.

Brain regions for rodent neural data & potential more fine-grained analyses

It is all from S1 (barrel cortex specifically). Our current neural dataset does not parametrize convexity and concavity for us to run a tuning curve analysis. However, our focus is on building the models which will be useful for such future datasets and subsequent downstream analyses to further separate models, since our model parameters are optimized on a task (rather than our specific neural dataset) and can be readily evaluated in new settings.

Identifying the best-fitting layer

The best neural fitting layer is the layer with the highest neural score, so a separate split of data was not used.

Explanation of the IntersectionRNN

The following equations define the recurrent computations at layer $\ell$ and time step $t$ for IntersectionRNN, more details about the computation of different RNNs used in our paper can be found in [2], and we plan to include these details in our revised paper.

$$m_t^\ell = \tanh(W_m^\ell * x_t^\ell + U_m^\ell * s_{t-1}^\ell + b_m^\ell)$$ $$n_t^\ell = **relu**(W_n^\ell * x_t^\ell + U_n^\ell * s_{t-1}^\ell + b_n^\ell)$$ $$p_t^\ell = \sigma(W_p^\ell * x_t^\ell + U_p^\ell * s_{t-1}^\ell + b_p^\ell + 1)$$ $$y_t^\ell = \sigma(W_y^\ell * x_t^\ell + U_y^\ell * s_{t-1}^\ell + b_y^\ell + 1)$$ $$s_t^\ell = p_t^\ell \circ s_{t-1}^\ell + (1 - p_t^\ell) \circ m_t^\ell$$ $$h_t^\ell = y_t^\ell \circ x_t^\ell + (1 - y_t^\ell) \circ n_t^\ell$$

Notation

• $x_t^\ell$ is the input at time $t$ and layer $\ell$
• $s_t^\ell$ is the state at time $t$ and layer $\ell$
• $W^\ell$ and $U^\ell$ represent learnable weight matrices
• $b^\ell$ are bias terms
• $\circ$ denotes element-wise multiplication
• $\sigma$ is the sigmoid function
• $\tanh$ is the hyperbolic tangent function
• $\text{relu}$ is the rectified linear unit function
• $*$ denotes a linear transformation or convolution depending on context

[2] Aran Nayebi, Javier Sagastuy-Brena, Daniel M. Bear, Kohitij Kar, Jonas Kubilius, Surya Ganguli, David Sussillo, James J. DiCarlo, Daniel L. K. Yamins bioRxiv 2021.02.17.431717; doi: https://doi.org/10.1101/2021.02.17.431717

Ethological validity of the SimCLR loss

The augmentation for SimCLR is an approximation for head motion of the rodents (see in Fig. 2 b, middle, for example, the rotation augmentation mimics head rotation of the rodents seeing the same object). From a training perspective, it enforces invariance of the same object for the model; hence, the “sample-twice” mechanism of SimCLR together with the proposed tactile augmentation ethologically approximating how rodents rotate their heads and provide different views of the same object to improve the learning results.

Inductive biases and prior work in rodent tactile processing

As mentioned in line 98-101, prior work in rodent tactile processing is only limited to supervised learning, a small number of model architectures, and crucially does not compare the models to brain data. With our Encoder-Attender-Decoder framework, we can systematically explore and search for the inductive biases necessary to rodent tactile processing, which is further strengthened by our results in neural alignment, as discussed in line 320-324. To sum up, our work is fundamentally different from previous works and the findings are well-supported by both task performance and neural alignment of the models under comprehensive search with a broad range of different neural network architectures (e.g., SSMs, RNNs, transformers).

Integration of insights into robotic tactile systems

There is currently no established or widely-used method to process tactile data in robotics, due to so much variety in hardware implementation. Yes, our intention is to convey that recurrent EADs would be a good candidate for such tactile hardware as they become more capable. At the moment, animals and simulation environments are more capable than current robotic hardware; therefore, our motivation for this work is to better understand a biological tactile sensing system so that, down the line, we can try to replicate biological tactile capabilities in robots using the inductive biases discovered in this paper. As such, the primary result of this work is in a neuroscientific scope and currently not deployed on robots. Still, our work (1) provides new tactile augmentations which may also be used to train self-supervised policies on robot tactile data, and (2) serves as a validation of low-dimensional force sensors as sufficient for object discrimination, as opposed to requiring high resolution images as tactile data (visuotactile sensors like GelSight). Both provide strong evidence for developing future robotic tactile systems based on force/torque sensors, other than existing sensors purely rely on vision, and can be potentially adapted for building such models.

Thank you for taking the time to review and provide your valuable feedback! We address the weaknesses and questions as follows.

Transferability to robotic tactile sensors

Our motivation for this work is to better understand a biological tactile sensing system so that, down the line, we can try to replicate biological tactile capabilities in robots using the inductive biases discovered in this paper. As such, the primary result of this work is in a neuroscientific scope and not deployed on robots. Still, our work (1) provides new tactile augmentations which may also be used to train self-supervised policies on robot tactile data, and (2) serves as a validation of low-dimensional force sensors as sufficient for object discrimination, as opposed to requiring high resolution images as tactile data (visuotactile sensors like GelSight).

Simulation-to-reality gap in tactile modeling

We used the most biologically accurate simulation model of whiskers available based on real mouse morphology in order to reduce the gap, but without real whisker force sensors, we are unable to evaluate the extent of the gap ourselves. Especially with tactile sensors, the wide variety in sensor implementation (optical, capacitance, mechanical, magnetic, pneumatic, etc.) and high amount of noise and variability even within sensor types means that collecting quality real data is very difficult. A discussion of the limitations of current hardware is included in the Introduction section– current bio-inspired robotic whisker sensors are limited by hardware complexity, poor multi-stimuli discrimination, and mechanical shortcomings that hinder accurate tactile sensing compared to biological whiskers. While the creation and validation of a real whisker array force sensor is outside the scope of our current neuroscientific evaluation, we definitely agree that this would be beneficial to evaluate our models on real whisker-like hardware once available.

Exploration of alternative recurrent architectures

Thank you for your suggestion! Currently, the RNN architectures explored in our paper include: time-decay (the original Zhuang’s model), GRU, LSTM, UGRNN, IntersectionRNN. We didn’t consider any bi-directional architectures for supervised learning as it breaks the biologically plausible temporal unrolling mechanism, where each layer is sequentially activated as time increases. For self-supervised learning, we incorporate the idea of time reversing in our proposed tactile augmentation (see Fig. 2b, middle, temporal), which flips the input temporally and is similar to the high-level idea of bi-diretional RNNs.

We further replace the recurrent cell with vanilla RNN, and conduct experiments on the following architectures:

1. Zhuang+GPT (best task-performant and neural-fit architecture for supervised learning)
2. Zhuang+IntersectionRNN+SimCLR (best neural-fit architecture for self-supervised learning)

The best task-performant architecture for self-supervised learning is ResNet+Mamba+SimCLR, which doesn’t include any RNNs; hence, it’s not included in the additional experiments. We present the task performance (top-5 classification accuracy) and neural fits of the vanilla RNN variants below.

We can observe from above that vanilla RNN doesn’t provide further improvement in either task performance or neural fit compared to existing RNN architectures.

Ecological relevance of ShapeNet for neural alignment

Although ShapeNet is not ecologically relevant for mice, we note that the untrained models are worse than the trained models at neural predictivity, suggesting our choice is beneficial. There are no suitable rodent-relevant datasets, so we would have to hand-craft one by selecting a few objects. Arguably, the dataset diversity in ShapeNet should prepare the model for more downstream tasks, while a smaller but rodent-specific dataset may not be large or diverse enough to do well. Finally, in NeuroAI more broadly, it has been shown that having dataset diversity is critical for model-brain alignment, not just task performance– this is known as the “Contravariance Principle” [1, 2, 3].

[1] R. Cao and D. Yamins. “Explanatory Models in Neuroscience, Part 1: Taking Mechanistic Abstraction Seriously.” Cognitive Systems Research, vol. 87, Sep. 2024, p. 101244. ScienceDirect, https://doi.org/10.1016/j.cogsys.2024.101244.

[2] D. Yamins and J. DiCarlo. “Using Goal-Driven Deep Learning Models to Understand Sensory Cortex.” Nature Neuroscience, vol. 19, no. 3, Mar. 2016, pp. 356–65. PubMed, https://doi.org/10.1038/nn.4244.

[3] A. Nayebi*, N. C. Kong*, C. Zhuang, J. L. Gardner, A. M. Norcia, and D. L. Yamins. “Mouse Visual Cortex as a Limited Resource System That Self-Learns an Ecologically-General Representation.” PLoS Computational Biology, vol. 19, no. 10, Oct. 2023, p. e1011506. PubMed, https://doi.org/10.1371/journal.pcbi.1011506.