Transplant of Perceptrons

• While the idea is novel and interesting, the major weakness of the paper lies in its implementation, applicability, and lack of empirical validation.
• The authors chose variance as the criterion for activeness and drew parallels with Hebbian learning. However, the chosen approach fails to take into account the class label, as it is expected that some neurons would specialize for a certain class. The proposed approach might also hinder the specialization of neurons, which is a desirable trait.
• The theoretical analysis needs to be more concise and clearly state actionable insights. The chosen approach makes it seem like an expected result that mutual information would increase.
• I am not convinced that simply copying the weights of the active neurons to inactive neurons is an optimal approach for implementing transplantation. The authors also fail to provide a convincing justification for this choice.
• The biggest concern lies with the empirical validation and the applicability of the method. The primary analysis is conducted on MNIST, which is not an issue in itself. However, the baselines are much lower than those of traditional MLPs. The decision not to utilize activation layers is concerning and lacks justification. This raises questions about whether their findings hold with activation layers, which are essential for the applicability of the method.
• While the authors claim that their method is applicable to different architectures, they have not demonstrated this. They only considered different numbers of hidden layers in MLPs, which does not truly constitute a different architecture. The applicability of their approach to CNNs, for instance, remains unclear. Furthermore, it's important to consider how activation layers and batch normalization would affect their findings.
• Extending the study to more complex datasets like CIFAR-10 is also left unclear.

My biggest concern in this paper is the performance of the transplanted multi-layer perceptron: in the ablation results for $\eta$ and $k$, the accuracy does not surpass $90$% for MNIST, including $\eta=0$ (vanilla multi-layer perceptron). It is widely accepted that MNIST achieves an accuracy of $>90$% for MNIST on logistic regression. Hence, the multi-layer perceptron described in the paper (with a hidden layer of width 100) should comfortably achieve $>90$% for MNIST.

My second biggest concern is the strong abuse of the term perceptron. The perceptron refers to a single-neuron such that $y=wx + b$, where $w$ is a vector of real-valued weights. In the paper, the authors define a perceptron as having a weight matrix $W \in \mathbb{R}^{m \times n}$, which is rather a multi-layer perceptron. These are fundamental concepts that should not be abused. In particular, Hebbian learning in a perceptron is very well understood, as having the update rule $w_{t+1} = w_t + yx$, it would have been advantageous to highlight this to describe hebbian learning in perceptrons. However, I believe in the manuscript, the authors were largely referring to multi-layer perceptrons and not the perceptron.

Generally, I don't find that the paper highlights or addresses a problem with artificial neural networks beyond seeking inspiration from neurobiology. Why would we need transplantation in neural networks?

I found the mutual information analysis lacks clarity: the authors compute the mutual information between the data $x$ and the output $y$ and compare the different transplantation methods. However, I don't find that the authors sufficiently motivate why this analysis is important and how this relates to transplantation.

Minor feedback:

• Section 2.2 is not necessary, and can be moved to the appendix.
• Figures 4, 7, 9, and 11 are very small, it would be good to make them larger to make them more legible.
• The algorithm is relatively simple, would be nice to make the code available.

The connection to forms of biological neuronal replacement are overstated. In particular, even if neurons could be replaced 1 for 1 in a brain, the goal is to replace a previously valuable but damaged neuron. This is a different program than the paper pursues in ANNs.

To reset the weights feeding into or out of units to high values, after those weights have been trained into irrelevance, is highly counterintuitive, and thus requires a very strong argument to justify. In this case, a non-trivial increase in accuracy would be needed (this does not happen).

The results are for NNs with linear activation functions. How the proposed method would work in NNs with non-linear activation functions is important but not addressed.

Reviewer limitation: I did not read section 3 (Theoretical Analysis).

1. It would be great to see a more in-depth analysis of Fig. 8: the proposed method seems to degrade the quality; how is this justified?

2. Sec. 4.2 is titled 'RESULTS IN MULTIPLE ARCHITECTURES' which for a conventional reader is a bit misleading - the architectures evaluated are 1-4 layer perceptrons.

3. It would be great to see the application of the approach on a more modern task/DNN - e.g. sota image segmentation or 3D reconstruction / egomotion estimation. Given the relative simplicity of the method, what would be the technical challenges implementing such evaluation?