Credit-based self organizing maps: training deep topographic networks with minimal performance degradation

Dear reviewer,

Thank you for your comments and suggestions. We are actively working on preparing a response to your comments. However, we noticed that the information about references 2 and 3 in your comments (related to topologically organized GMMs) may be missing or can't be viewed by us. We would appreciate it if you could share the information about these references with us.

Authors

We sincerely thank the reviewer for their thoughtful feedback and for recognizing the effectiveness of our proposed approach and for acknowledging the clarity of our writing and the validity of our experiments. In response to the reviewer’s comments, we have further refined our arguments and addressed all concerns to strengthen the manuscript.

Inference without a loss function. The BMU selection process is only used at training time to induce a topographical organization across the model filters. However, during inference time the model simply computes the output of all filters without the need for calculating the BMU unit and in that sense operates similar to a non-topographical network. For that reason, the network doesn’t require a particular loss or reward function to operate after it is trained. We clarified this point in section 3 (Model Training) of the revised submission (lines 237-239).

Reconcilitation between AB- and CB-SOM. We were unsure whether we understood the reviewer’s point correctly but we considered two possible interpretations: We assumed that by “reconciliation between AB-SOM and CB-SOM occurs” the reviewer hypothesizes that the two models AB- and CB-SOM eventually converge to the same representation after sufficient training. Several of our experimental results support the view that such reconciliation does not occur in these models: 1) The visualization of weights in the first convolutional layer in Fig 3a shows drastic qualitative differences in the type of filters emerging in the first layer of the network that would very likely have substantial effect on the type of representations emerging in later layers of the network; 2) Related to the previous point, the distribution of frequency selectivity in the two models are substantially different between the two models as shown in Fig. 3e; 3) Unlike in CB-SOM, units in deep layers of AB-SOM have significantly lower selectivity to categories such as Face, Body, and Place; 3) The Representational similarity to macaque and human visual cortices is quite different between the two models judging by the results in Fig. 5; 4) Finally the performance of the two models on object recognition is substantially different according to Fig. 2b.

Alternatively, the reviewer may have suggested that while the representations are different, the probability of units being selected as BMUs may converge to similar values in the models. As shown in Fig. A1 in the appendix, we showed that this point is also not true as the entropy of the distributions are quite different between the two models with CB-SOM having a higher entropy (more uniform likelihood for units being selected as BMUs) compared to AB-SOM.

We hope our explanation adequately answers the reviewer’s question and we remain open to clarifying further.

Related work by Hartono et al. Thank you for bringing these two highly-related works to our attention. In both works, topographically organized feature maps are learned via the context-relevant SOM approach which combines the SOM learning rule (including both BMU selection and update rule) with backpropagation of error given a loss function. In that sense, we believe that this method is effectively similar to the AB-SOM baseline considered in our work that combines the original SOM update rule with backpropagation using the cross-entropy loss. We included and discussed both references in the Related Work section of the revised submission (lines 99-101).

I appreciate your response in such a short time. You have clarified all of my questions very well. This is a good paper.

I only have some minor issues that need further clarifications and modifications.

1. There is no neighborhood function during the operation stage. All of the filters generate their outputs without any topological constraints. Is my understanding correct?
2. Regarding the difference between your model and the rRBF in [Hartono 2015, 2020]: while their model did not explicitly select the BMU based on the loss gradient, the loss gradient is included in the filters' modification. Hence, the filters with large gradient losses are the ones that are modified the most. In contrast, filters with low gradient values (credits) will be less modified. Doesn't this make your model practically equivalent to theirs?
3. Line 634 exceeds the frame of the paper.
4. [Hartono 2015, 2020] are journal papers, not conference papers, so please fix line 596 and line 602,

We thank the reviewer for recognizing the significance of our work and its potential interest to both neuroscience and deep learning communities, as well as for acknowledging our extensive comparisons between model responses and neural recordings. Below we have carefully addressed each of the reviewer’s concerns and questions to further enhance the manuscript.

Evaluating alternative architectures. We thank the reviewer for their helpful suggestion. We agree with the reviewer that showcasing the generality of the proposed algorithm is critical. To address the reviewer’s comment, we trained an alternative neural network architecture (CORnet-S [1]) following similar settings as those used for ResNet18 architecture. We found strong topographical organization in the early and late layers of this network, mimicking those we had obtained in our ResNet18 experiments. We added these new results to the new Fig. A6 in the appendix.

[1] Kubilius, Jonas, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj, Elias Issa et al. "Brain-like object recognition with high-performing shallow recurrent ANNs." Advances in neural information processing systems 32 (2019).

Clarification of “units”. We thank the reviewer for bringing this issue to our attention. We agree that the language used in our original manuscript was not completely clear. Indeed, by “units” we refer to channels in each convolutional layer. We revised Figure 1 and text in section 3 and clarified this point by replacing “unit” with “filter” throughout.

Relation to lateral connections. We appreciate the reviewer’s interest in understanding the relationship between our proposed mechanism and lateral connections in the brain. Local lateral connections in the brain are indeed known to connect neurons as an exponentially decaying function of physical distance, which helps to shape topographical organization. This concept is explicitly embedded in Kohonen’s original Self-Organizing Map (SOM) model (Kohonen, 1982), although it is implemented in a modified form in later variations.

Briefly, in Kohonen’s original formulation, the activity of each unit $\eta_i$ is computed as follows:

$\eta_i = \sum_{k\in S_i} \gamma_k \phi{\prime}_k$

where $S_i$ denotes the set of neighboring units around the Best Matching Unit (BMU), $\phi{\prime}$ represents the weighted average of inputs (i.e., the feed-forward computation performed by each unit), and the coefficients $\gamma_k$ are determined based on the distance between each neighboring unit and the BMU. This formulation reflects a form of lateral influence, where increasing the match between the BMU and the input pattern (via a Hebbian learning rule) also leads to updates in neighboring units, with each unit’s update weighted according to its distance from the BMU (i.e., modulated by the $\gamma$ coefficients).

In later implementations of SOM, the explicit role of lateral connections in calculating unit activity was often omitted, while their influence on the update rules was retained through the use of a distance-dependent neighborhood function. This simplified formulation retains the impact of lateral connectivity on learning, even though the computation of activity no longer directly involves lateral connections.

Our proposed Credit-Based Self-Organizing Map (CB-SOM) utilizes the same update rule as the classic SOM, preserving this connection to lateral connectivity in the brain. The primary difference lies in how the BMUs are selected; in CB-SOM, BMUs are chosen based on the gradient magnitude, rather than activation levels. This modification aligns CB-SOM with gradient-based learning methods commonly used in deep neural networks, while still incorporating the influence of lateral connectivity principles observed in the brain through the neighborhood function applied during weight updates.

We included the above information about the relation between lateral connections and the SOM models in a new appendix section A.1 of the new submission.

We thank the reviewer for their thoughtful comments and recognition of our work’s strengths, particularly the novelty of CB-SOM in integrating topographical organization into deep neural networks and its implications for neuroscience and AI. We have carefully considered the reviewer’s questions and comments and provide detailed responses addressing each point:

Comparison of computational efficiency. Thank you for your comment. We would like to first emphasize that as the architecture of the model used in our experiments is the same as the original model (ResNet18), the computational cost of our proposed CB-SOM model is exactly the same as the original ResNet18 model architecture at inference time. The competitive algorithm is only used during training and does not apply at inference time. In that regard, the only difference in computational cost is during training as we introduce additional steps for selecting BMUs and performing the SOM-based weight updates. For transparency, we measured and compared the training step-times for each of the baselines and added the information in a new appendix Figure A5.

Highlighting the remaining gap in performance. To address the reviewer’s concern, we updated the “main contributions” statement to better reflect this remaining challenge. We further updated the “discussions and limitations” section to appropriately emphasize the significance of the remaining gap (line 537).

Impact of the performance trade-off on the algorithm’s appeal outside neuroscience. Thank you for the comment. We acknowledge that the lower performance of topographical models may make them less attractive as practical tools. We would like to emphasize that the motivation of our study is to replicate a missing characteristic of the cortex in existing deep neural networks, cortex-like topographical organization, in order to improve the existing models of visual cortex. This is an important characteristic missing from existing models that our project aims to address. Topographical neural network models are thought to be potentially immensely useful for guiding therapeutic interventions such as stimulation or surgeries [1]. For example, planning resection surgeries in patients with epilepsy could benefit from predictions from topographical models that match the organization of the cortex as surgeons often plan their surgeries to avoid cortical patches that serve critical functions such as face recognition. We updated the introduction and discussion sections to better incorporate the motivation as well as potential benefits of our research (lines 50-58, 515-522).

[1] Schrimpf, M., McGrath, P., Margalit, E., & DiCarlo, J. J. (2024). Do Topographic Deep ANN Models of the Primate Ventral Stream Predict the Perceptual Effects of Direct IT Cortical Interventions?. bioRxiv, 2024-01.

High-level summary of performance drop in previous approaches. We appreciate the reviewer’s comment and agree that including this information would provide better context to our readers. We revised the “Deep topographical neural network” subsection and reported the range of relative drop in object recognition accuracy based on prior models of cortical topography (lines 153-154).

Providing code for reproducibility. We included all the code required for replicating our results in the supplementary material. We will also publicly release the code upon publication.

Other baselines in Fig2A. To address the reviewer’s concern, we added the non-topographical ResNet18 architecture to Fig 2a as an additional baseline. We did not include the TDANN model as the supervised-trained variation of this model does not elicit strong topographical organization and only the unsupervised trained variation replicates topography. However, we felt that it would be unfair to compare the learning speed between models that are trained, supervised and unsupervised.

Discussing the implications of improved brain alignment in CB-SOM. We appreciate the reviewer’s comment. Our results show that in particular when compared to data from the human visual cortex, our topographical CB-SOM model is superior to the non-topographical ResNet18 model. This result suggests that inducing topographical organization in deep neural networks through overrepresentation of units with maximal effect on task performance (similar to CB-SOM) may lead to more aligned representations with the brain. We revised the Discussions section and expanded on the implications of these results (lines 496-500).

(Continues in the next comment)

Dear Reviewer,

Thank you again for taking the time to review our work. We’ve provided detailed responses to your comments and would greatly appreciate any further feedback or clarifications from you regarding our rebuttal. Your input is invaluable in helping us improve our manuscript, and we are eager to engage in a constructive discussion. Please let us know if there are any additional points we can address.

Thank you for your time and consideration!

Authors