From Overconnectivity to Sparsity: Emulating Synaptic Pruning with Long Connections

We thank the Reviewer for taking the time to examine our work and for providing valuable insights and comments. We will now address the questions raised:

1. “Please explain why your method can have effect.”

We present a detailed mathematical analysis of the training dynamics of our method in Appendix G. Our analysis of the gradient flow dynamics in LCNs supports our empirical observations. In Appendix G4 we also include visualizations of the gradient norms of LCNs during training, which are also aligned with our analysis and observations. Essentially, long connections impose a structure that modifies the gradient dynamics of the network (compared to FFN or residual) and supports "layerwise training".

2. “Please compare with some state-of-the-art pruning techniques.”

We provide a reference to a recent work (ICLR ‘24) [1] that focuses on Vision Transformers pruning. We refer to Tables 4 and 5, where the authors report results on ImageNet using ViT-Base, allowing for direct comparison. In Table 4, they provide the FLOPs and accuracy drop with their method, while in Table 5, they report similar metrics for ToME [2], which is not a pruning technique but rather a method for reducing the number of tokens and, consequently, the model's FLOPs. With our method, we achieve 10.30 FLOPs (G), outperforming both approaches without sacrificing accuracy. We provide the table below based on ViT-Base on ImageNet, for comparison (the numbers of the two methods are extracted from Table 5 in [1]):

[1] https://arxiv.org/abs/2403.17921, [2] https://arxiv.org/abs/2210.09461

Thank you for your follow-up comments. We acknowledge the concerns you have raised:

• We apologize for the discrepancy in our copying from (https://arxiv.org/pdf/2403.17921) in our initial reply. We corrected our table.

• We understand the Reviewer's concern that a single comparison point may be insufficient to demonstrate the method's superiority. To address this, we expand our comparison by referring to Figure 6 of [4], which presents several pruning methods for BERT-Base. Notably, our method, which removes entire layers, results in a 0.25 relative FLOPs with no performance degradation, outperforming all methods shown in the figure. Additionally, since our approach can be viewed as a natural compression method, we include results from [1], a 2024 survey on general transformer compression. In Table 4, the authors compare works that compress BERT-Base using knowledge distillation. Since we provide results (Appendix F) only for 3 datasets of the GLUE Benchmark (SST-2, QNLI, QQP) we provide the original papers' references which contain the analytic results on GLUE. We observe that:

From Table 1 of [3], TinyBERT with 14.5M parameters exhibits an accuracy drop for 2 out of 3 datasets (from ~1% to ~3%) depending on the task. TinyBERT with 66.7M parameters performance is almost on par with the full BERT-Base. For DistilBERT (66M parameters), from Table [1] of [2] we see that there is an accuracy drop of ~1-2.5% on the 3 datasets. Our model converges on 3 layers (45-46M parameters) without accuracy drop compared to the residual baseline (full model), striking a balance between efficiency and performance.

[1] https://arxiv.org/pdf/2402.05964

[2] DistilBERT: https://arxiv.org/pdf/1910.01108 (Table 1)

[3] TinyBERT: https://arxiv.org/pdf/1909.10351 (Table 1)

[4] https://arxiv.org/pdf/2204.09656

We hope that this provides a more complete picture of our method's performance relative to existing pruning/compression approaches.

Finally, we would like to emphasize that our work is not solely focused on pruning; instead, it ultimately introduces a new architecture, Long Connection Networks, which you also highlight in the strengths section of your initial review. So, while pruning is certainly one valuable outcome, we believe that our work also makes significant contributions to neural network design, theoretical understanding, and practical implementation, all of which are central to the paper. The architecture seems to naturally induce several desirable properties (namely, natural sparsification, enhanced robustness to input noise, efficiency in low-data settings) that could be valuable for making neural networks more efficient, robust, and biologically plausible.

We appreciate your thorough review process as well as your feedback and will await the consolidated feedback from all reviewers. We are committed to addressing all concerns raised to strengthen the paper's contribution to the field.

We thank the Reviewer for taking the time to examine our work and for providing valuable insights and comments. We will now address the questions raised:

“The evaluations are too simple. Like many things in machine learning, something that works for simple versions of tasks might break down for larger tasks. I am most knowledgeable about vision. I would like to see similar results for imagenet rather than just CIFAR-10 to be really sure that these results are actually going to be impactful. (Apologies if I missed it if this was included.) I am not competent to juge the meaningfulness of the language results.”

We presented our proposed architecture and validated it through both theoretical analysis (Appendix G) and experimental studies across text and vision modalities. Our experiments included simpler datasets such as IMDb (Fig. 5), Amazon (Fig. 4b), and CIFAR-10 (Fig. 6). Additionally, in the original submission we evaluated LCNs on ImageNet (Fig. 4b, Fig. 7, Table 1). We believe this aligns with the reviewer’s suggestion.

Furthermore, in the revised version we include another large-scale experiment on BERT pre-training and fine-tuning (see Appendix F). We note that across all these experiments, LCNs were trained efficiently, delivering performance on par with their residual network counterparts and showing improved robustness against noise, while consistently demonstrating their ability to prune redundant layers, achieving a 30-80% reduction in the number of layers compared to baseline models.

We thank the Reviewer for taking the time to examine our work and for providing valuable insights and comments. We will now address the questions raised:

1. ‘I did not feel that the comparison to similar models was adequately made. There is a related work section which many relevant models are correctly brought up, but 1. there is no explicit description of how LCNs are different from these models and 2. apart from residual connections, none of these models are quantitatively compared to this previous work. One is left to ponder whether LCNs makes conceptual/quantitative advances in the field.”

• Qualitative comparison of LCNs and Related models

We revised the paper to further highlight our contribution in relation to the existing literature (Section 2 line 120). There, we discuss the difference between LCNs and original Residual Networks, Deep Supervision, DenseNets and DenseFormers, which are models referenced in our Related Work. The key distinction is that in all these models each layer is connected to all preceding layers (for Residual and Deep Supervision check [1] eq.1 and also [2], for DenseNet see in [1] eq.2 and for DenseFormer see in [3] Fig. 1 and equations in page 3), whereas LCNs possess a fundamentally different structure, connecting each layer only to the output, as demonstrated in Equations 1–4 and detailed in Appendices A and G.

[1] https://arxiv.org/abs/1608.06993, [2] https://arxiv.org/abs/1605.06431, [3] https://arxiv.org/abs/2402.02622

• Quantitative comparison of LCNs and Related models

We quantitatively compared LCNs with Residual Networks across various settings in the main paper and also showed that Deep Supervision does not exhibit the same behavior (Appendix C).

2. “the link to biology and the brain is interesting, but after the initial abstract it is hardly addressed. The authors report longer training times as more akin to 'biological brain networks', but I'm not sure which result even supports this (in Fig 8 it trains faster?). That LCNs handle noise better is interesting, but an intuitive explanation/link to previous literature as to why this occurs would be very helpful in its interpretation.”

We revised the paper to better highlight the connections between LCNs and biological brain networks. Specifically we highlighted the connection between LCNs and layer-wise training (Appendix G4), a method that is employed in NN training and is motivated by cognitive neural development in the prefrontal cortex [1]. We further detailed the macroscopic similarities of LCNs and biological NN in our conclusions. We also briefly note that studies have indicated layer V in the neocortex receives inputs from all other neocortical layers and functions as the primary output layer [2,3], much like the output layer of an LCN.

We direct the Reviewer to reference [4], which demonstrates that architectures interpolating between residual and feedforward networks exhibit enhanced robustness to noise, as compared to standard residual networks (intuitively skip connections transfer input noise to the entire network).

In the majority of the experiments (Mixer on Cifar-10 Sec 3.4 footnote 9, ViT on ImageNet Sec. 3.2, BERT pre-training Appendix F) LCNs required longer training times compared to residual baselines.

[1] https://www.frontiersin.org/journals/neuroanatomy/articles/10.3389/fnana.2011.00029/full
[2] https://www.frontiersin.org/journals/cellular-neuroscience/articles/10.3389/fncel.2015.00233/full [3]https://www.frontiersin.org/journals/neuroscience/articles/10.3389/neuro.01.1.1.002.2007/full

[4] https://arxiv.org/abs/2006.05749

3. “It appears from the figures that all results are derived using a single random initilisation seed? If so, I would strongly recommend re-running these analyses over several seeds, otherwise it is very difficult to draw any strong conclusions.”

We added in the manuscript the results from multiple runs (Table 1).

4. “Some of the segways between sections could be smoother. For example short introductions to sections 3.1/7 would help. Some figures could also be clearer. For example a few instances where the axis labels were confusing. E.g. what are the axis labels in Fig 2? I presume w and the error gradient with respect to w? in fig 3 is weight of hidden1 = w1, and wouldn't it be interesting to look at w2/w3 as well to check if they are sparser for LCN, and maybe absolute values instead of raw values to better show sparsity? In Fig 7 I would prefer an explicit mathematical term relating to an equation than 'std' and 'noise_level'. In section 3.1 a schematic (even if for the appendix) would be helpful to the reader”

We included a short introduction at the beginning of section 3. We also clarified Fig.2 and changed x-axis in Fig.3 and Fig.7 for clarity, as per the reviewer’s suggestion. For section 3.1 we included a reference to Fig.1 to visualize the subnetworks we describe.

• We continue our response in the comment below due to characters limit.

Answering Reviewer's questions :

• “line 54: 'Contemporary deep learning architectures aim for overparameterization' - what does this mean?”

We direct the Reviewer to references [1–5]. Modern neural networks are overparameterized, meaning they contain a significantly larger number of parameters than required to fit the training data. This design exhibits intriguing properties and has been the subject of extensive analysis.

[1] https://arxiv.org/abs/1912.02292, [2] https://arxiv.org/abs/1812.11118, [3] https://arxiv.org/abs/1811.04918, [4]https://arxiv.org/abs/2206.04569, [5] https://arxiv.org/abs/2311.14646

• “is the residual network given by equation (3) a specific type of residual network? It is introduced there rather abruptly for a non-expert reader”

We added the equation of the residual network in Appendix A. Since in 2.1 we perform a simple mathematical analysis we are considering a residual network given by the equation:

$x_{l+1} = f_{l+1}(x_l) + x_l = w_{l+1}x_l + x_l$

• “for the toy example the models are trained for 500 epochs (1000 examples per epoch). This seems like a lot for such a simple problem?”

We trained with simple SGD and we used a small learning rate (=0.001), leading to slow convergence. Re-running the experiment with larger lr (=0.01) we get similar results with 100-150 epochs.

• “line 259: 'possible indication of overfitting' - is Fig 5 showing the training or validation accuracy?”

Fig.5 is showing the validation accuracy. In Fig. 5c the intermediate layers of the residual Transformer exhibit an irregular non-smooth behavior (compared to 5a and 5b of the same model)- that is why we think it might be an indication of overfitting.

• “typos: line 210: brackets around citation line 326: -> architectures line 411: -> directly paragraph 460: brackets around (multiple) citations line 492: -> effectively”

We revised the manuscript and corrected the mentioned issues.

Thank you for raising your score and for these additional points.

Regarding ImageNet accuracy: The difference in reported accuracies is due to batch size and associated training settings. We use a batch size of 256 due to computational constraints, while the original ViT paper uses 4096; all other hyper-parameters are identical. This impact of batch size is well-documented - for example, the same ViT-Base implementation achieves 74.6% with the original training setup versus 72.6% with batch size 1024 (see https://github.com/ssuncheol/vision-transformer-pytorch). Our results are consistent with these findings. Note that this performance gap could be decreased with extensive hyperparameter tuning, but most importantly, we compare LCN and vanilla (residual) architectures under identical training conditions, ensuring a fair comparison of the architectural contribution. We will add a note about training conditions vs performance in the final paper for clarity.

On performance differences: The small variations in performance between LCNs and traditional architectures can be explained by their different generalization characteristics. LCNs might show slightly lower performance when a model is overfitting to a dataset (as the additional capacity of traditional architectures can help memorize the training data), but tend to outperform when better generalization is needed. This is consistent with their adaptive layer usage and explains the small variations you note in Table 1. We use "lossless" to convey that LCNs achieve comparable performance while automatically using fewer layers, but we're happy to use "performance-preserving" if you feel it would be clearer.

We thank the Reviewer for taking the time to examine our work and for providing valuable insights and comments. We have taken them into account resulting in a much improved manuscript.

1. “The validity of the structure is verified only on the base ViT, but there is a lack of experimental and theoretical validation for larger model structures.”

• theoretical validation

We provide, in Appendix G, a comprehensive theoretical analysis of the training dynamics of LCNs based on gradient flows for different layers. We also validate these theoretical claims experimentally by visualizing the gradient dynamics of LCNs during training (see Appendix G.4).

• experimental validation

We provide further evidence, in Appendix F, on the broad applicability of the proposed architecture by running a pre-training experiment on BERT (followed by fine-tuning on three tasks). The results are consistent with the rest of the training recipes in the paper.

2. “If the author claims that this is a fine-tuning method, the time spent during model training is typically included in the overall time assessment. Fine-tuning often involves adjusting a pre-trained model on a specific dataset, and this process can require significant computational resources and time. However, if the focus is on inference speed or deployment efficiency, the discussion (time, GPU memory and other computation cost) might be reported during the fine-tune process .”

All experiments in the main body of the paper consist of models trained from scratch. We clarified that the reported metrics and results related to inference speed and memory usage pertain to the training process. Moreover, to show the general applicability of our architecture we also experimented on fine-tuning (Appendix D) as well as pre-training+fine-tuning (Appendix F). In all cases we provide details on the training and evaluation protocol we followed (computational resources, epochs). We note that:

Our experiments show that while LCNs require ~2x longer training time (700 vs 300 epochs for ViT), they achieve:

• 40% faster inference (13.9ms → 8.3ms)
• 41% reduction in parameters (86M → 51M)
• 41% smaller storage size (330MB → 195MB)

Importantly, the longer training is a one-time cost that enables permanent inference benefits. The pruned architecture can be distributed for deployment without requiring end users to perform the longer training"

3. “It sounds like you' re noting that the determination of the final model layer relies heavily on experimental validation, which can indeed be resource-intensive and time-consuming. This approach can lead to significant computational costs, especially when iterating through different configurations to find the optimal structure.”

For determining the final model’s number of layers we indeed use a validation set. However, this set is small (5% of the full dataset in our ImageNet experiment), allowing for quick validation. Our method requires only a single pass per batch to compute the output of each layer (as detailed in Section 3.1), and a single evaluation is sufficient, without the need for multiple iterations (as opposed to methods proposed in [1], [2]). We are also working on super-light algorithms for determining a sample-specific layer depth (see Future Work).

[1]https://proceedings.neurips.cc/paper_files/paper/2022/file/3b11c5cc84b6da2838db348b37dbd1a2-Paper-Conference.pdf

[2] https://arxiv.org/pdf/2402.11187

4. “While verifying the effectiveness of a structural design on a classification task provides important insights, it doesn't necessarily guarantee that the same design will perform well across different tasks, such as object detection, segmentation, or generative modeling. To better justify the generalization to other tasks, the authors could provide theoretical justifications or include empirical evidence.”

We present some evidence supporting generalization:

• Appendix F demonstrates successful application to masked language modeling and transfer learning (BERT). The results are consistent with our observations when training LCNs from scratch, delivering performance on par with the residual baselines while utilizing 25% of the full model’s layers.
• The theoretical analysis in Appendix G shows that the benefits arise from fundamental gradient flow properties that are task-independent. The analysis is performed for an arbitrary task modeled by a loss function $L$ that we aim to minimize and it is further supported by empirical evidence.
• Empirically, we show robust transfer learning capabilities from ImageNet to CIFAR-10 (Appendix D).

Nonetheless, we agree more validation on diverse tasks would be valuable and have explicitly noted this as a limitation in Section 9. We are currently working on scaling up our experiments to billion-parameter LLMs, as discussed in our Future Work section.