We sincerely thank the reviewer for recognizing our contributions, particularly acknowledging our effort to demonstrate the effectiveness of Lp-Convolution through the Sudoku Challenge. Additionally, we deeply value the constructive feedback provided, which offers significant opportunities to enhance the quality of our paper. Below, we address the reviewer's comments and provide detailed clarifications.

Weakness 1) Inconsistent and Confusing Terminology for Key Components

First, we would like to clarify the terminology:

• Lp-Mask: trainable mask $\mathcal{M}$, applied to convolutional weights $\mathcal{W_i}$ (Eqn. 2).
• Lp-convolution: the overall convolution process incorporating the Lp-mask (Eqn. 3).

To address this, we will revise the manuscript as follows:

• Clearly highlight the distinction between the two terms in Section 3.
• Use “Lp-convolution” consistently throughout the text, reserving “Lp-mask” for contexts where it is explicitly relevant.

We believe these changes will significantly enhance the clarity of the manuscript and minimize any potential confusion. Thank you for highlighting this important point.

Weakness 2) Irrelevant Mention of Vision Transformers (ViTs)

While ViTs are not the primary focus of our study, we included them in the introduction to provide broader insight—that CNNs can be advantageous over ViTs in data-hungry regimes due to their inductive bias. However, as the reviewer pointed out, without subsequent experimental results to support this point, we agree that our intention may not be effectively conveyed.

To resolve this we compared ViTs with Lp-Models as following table:

As shown in Table, Lp2-AlexNet achieves comparable performance to ViT-16x16 on TinyImageNet with significantly lower parameter counts and computational cost, demonstrating its efficiency. We will include this result to clearly relate the ViTs and CNNs. Thank you for raising this important point.

Weakness 3) Lack of Supporting References for Biological Comparisons

We agree that lines 108–110 should be carefully addressed with supporting references, as they represent a key premise of our work. Additionally, the observation that CNNs can exhibit sparse weight patterns during biological modeling is an important point that warrants further discussion.

To address these points, we will revise the manuscript lines 108-110 as follows:

• "Standard CNN architectures are typically designed with rectangular, dense, and uniformly distributed connections [6-9], in contrast to the circular, sparse, and normally distributed connections commonly observed in biological neuron [10-12]. Early studies in biological modeling using CNNs have shown that task-specific adaptations can lead to sparse weight patterns [1-5]. These insights demonstrate the adaptability of CNNs and highlight the potential for bridging artificial and biological connectivity patterns."

These revisions will provide stronger support for our claims and improve the manuscript’s precision.

References

[1-5] See reviewer's references.
[6] LeCun et al. "Gradient-based learning applied to document recognition." IEEE (1998)
[7] Krizhevsky et al. "ImageNet classification with deep convolutional neural networks." NeurIPS (2012)
[8] Simonyan and Andrew. "Very deep convolutional networks for large-scale image recognition." ICLR (2015)
[9] He et al. "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification." ICCV (2015)
[10] Lerma-Usabiaga et al. "Population receptive field shapes in early visual cortex are nearly circular." J Neurosci (2021)
[11] Seeman et al. "Sparse recurrent excitatory connectivity in the microcircuit of the adult mouse and human cortex." eLife (2018)
[12] Hage et al. "Synaptic connectivity to L2/3 of primary visual cortex measured by two-photon optogenetic stimulation." eLife (2022)

Weakness 4) Undefined Variables and Ambiguity in Key Equations

We are grateful for the reviewer bringing this to our attention. We will ensure that the variable $\sigma$ (defined in the legend of Fig. 1e) is clearly stated in the main text to enhance clarity and precision in our presentation. Moreover, we agree that it needs to be explicitly defined in the main text as well. We will also properly cite the following reference to Eqn 1.

• Goodman, Irwin R., and Samuel Kotz. "Multivariate θ-generalized normal distributions." Journal of Multivariate Analysis (1973)

We appreciate the reviewer’s sharp observation regarding the undefined variables and ambiguity in key equations.

Question 1) Evidence for Gaussian Sparsity as Biological Constraints

Thank you for raising this important question. Since Gaussian sparsity is a core assumption in our model as a biological constraint, it is essential to address this thoroughly. Below, we provide both theoretical and empirical evidence supporting Gaussian structured sparsity.

1.Theoretical Evidence

• Sparse Coding Theory: Sparse Coding Theory posits that neural systems optimize sensory representations by minimizing redundancy. Learning a sparse code for natural images leads to the emergence of simple-cell receptive field properties [1]. This process can be linked with Gaussian priors, where synaptic weights follow a Gaussian distribution with most connections being weak and a few strong, promoting efficient information encoding [2].

• Effective Receptive Field (ERF) Theory: In convolutional neural networks, the actual influence of input pixels on an output neuron decreases in a Gaussian manner from the center of the theoretical receptive field [3]. This means that while the theoretical receptive field defines the maximum possible area of influence, the ERF is effectively smaller and Gaussian-shaped, with central pixels contributing most significantly to the neuron's output.

2. Empirical Evidence

• Supporting References: These two references demonstrate both anatomical and functional distribution of synapses predominantly following a Gaussian-like distribution in the visual cortex [4-5].
• Analysis Result: We have demonstrated Gaussian distribution with the in vivo functional synapse data [5] in alive mouse V1 in Appendix A.3.

These foundations affirm Gaussian sparsity as a biologically plausible constraint within our Lp-convolution framework. We will include this additional section in the manuscript to reflect these points comprehensively. We thank the reviewer for their valuable feedback, which has enhanced the clarity and robustness of our manuscript.

Refrences

[1] Olshausen andField. "Emergence of simple-cell receptive field properties by learning a sparse code for natural images." Nature (1996)
[2] Olshausen and Millman. "Learning sparse codes with a mixture-of-Gaussians prior." NeurIPS (1999)
[3] Luo et al. "Understanding the effective receptive field in deep convolutional neural networks." NeurIPS (2016)
[4] Hellwig et al. "A quantitative analysis of the local connectivity between pyramidal neurons in layers 2/3 of the rat visual cortex." Biological cybernetics (2000)
[5] Rossi et al. "Spatial connectivity matches direction selectivity in visual cortex." Nature (2020)

Question 2) What are the trends in p and their biological meaning?

We refer the reviewer to Appendix A.16 for detailed analyses and provide the following summary.

1. Distribution and Convergence of p

The learned p values, as shown in Appendix A.16.1 and A.16.2, are dispersed around distinct values such as 2, 4, 8, and 16 without overlapping. While these distributions vary slightly across datasets and model architectures, a general trend is observed: p consistently converges towards smaller values compared to its initialization, suggesting a decreasing tendency throughout training.

2. Layer-wise Trend

Appendix A.16.3 and A.16.4 further reveal layer-specific trends. In early layers, p predominantly decreases, likely reflecting the refinement of basic feature representations. Conversely, in late layers, p tends to increase, indicating enhanced integration of higher-level features and abstraction. These contrasting trends align with the hierarchical processing patterns observed in neural networks and biological systems.

3. Biological Meaning

The convergence of p towards smaller values supports our hypothesis that p learns to reinforce biological constraints. This is consistent with effective receptive field theory and our findings in Figure 1, where sensory input receptive fields in trained models resemble Gaussian distributions, a natural phenomenon in biological sensory systems. However, as p remains sensitive to its initial values, ensuring convergence to globally optimal values remains a challenge, which we aim to address in future research.

Weakness 5) Limited Improvements and Not SoTA

We agree with the reviewer’s observation that the performance gains seem modest and far from SoTA. However, we kindly request the reviewer to consider the distinct values of our work beyond achieving SoTA. Specifically, our contributions lie in:

1. exploring the potential of novel biologically inspired inductive biases, and
2. developing a new, easily pluggable module for CNNs.

1. Novel inductive bias

We acknowledge that the prior works that explore biological ideas focus on providing novel insights and, hence often fail to surpass original ML methods in raw performance [1-3]. Nonetheless, we have put ourselves into situations to provide not only novel insights but also practical improvement to the ML community. We have demonstrated consistent improvements in various architectures and tasks, underscoring the robustness and versatility of our approach. While our results are not SoTA, we believe our work has still meaningful contributions to the field by introducing novel inductive bias to the community.

2. Easily pluggable module

Historically, CNNs have evolved through the introduction of innovative modules. For example, a depthwise convolution module was originally proposed to improve computational efficiency not SoTA in MobileNet [5] and now plays a pivotal role in SoTA architectures like ConvNeXt and RepLKNet [6, 7]. Similarly, CoAtNet, a precursor to Astroformer, leveraged a hybrid module combining depthwise convolution and self-attention, enabling Astroformer to achieve SoTA solely through architectural refinements [8]. In this context, our proposed Lp-convolution module offers practical value as a pluggable component that is easily integrated into existing architectures, facilitating flexible and efficient deployment (See Appendix A.21).

We believe that exploring novel bio-inspired algorithms and providing new convolutional modules enrich the ML community by offering both theoretical insights and practical tools. By contributing an additional design choice that enhances robustness and versatility across architectures, our work supports innovation in ML model design.

References

[1] Pogodin, Roman, et al. "Towards biologically plausible convolutional networks." NeurIPS (2021)
[2] Liu, Yuhan Helena, et al. "Biologically-plausible backpropagation through arbitrary timespans via local neuromodulators." NeurIPS (2022)
[3] Kao, Chia Hsiang, and Bharath Hariharan. "Counter-Current Learning: A Biologically Plausible Dual Network Approach for Deep Learning." NeurIPS (2024)

Weakness 6) Weak Justification for Large, Sparse Kernels and Unclear How RSM Benefits Contemporary Vision Tasks

1. The justification for using large, sparse kernels

The use of large kernels enables the model to cover the input space more effectively with fewer layers compared to smaller kernels [1, 2]. However, simply increasing the kernel size does not guarantee performance improvements, as shown in Table 1 (Base vs. Large). This is presumably due to larger kernels inadvertently incorporating irrelevant global information, which can hinder performance compared to smaller kernels that rely on locality inductive biases to extract local features hierarchically. This is where sparsity plays a key role. We introduce sparsity constraints to optimize the usage of large kernels, ensuring they focus on only relevant global information while mitigating the disadvantages of naïvely expanding kernel sizes, supported by Sudoku experiments.

We will include this discussion in the revised manuscript to better justify our approach and clarify its effectiveness.

2. RSM for Contemporary Vision Task

While RSM analysis itself is not directly tied to contemporary vision tasks, it serves as a critical tool to explore biologically inspired design principles that can inform about AI models. In this study, we used RSM analysis to measure the alignment between AI models and the brain with our novel inductive bias idea, rather than relating with contemporary vision tasks such as classification, detection, or segmentation. The essential question driving this work is whether CNNs behave like the brain and, more importantly, what insights can be gained by applying brain-derived inductive biases to AI models.

References

[1] Ding, Xiaohan, et al. "Scaling up your kernels to 31x31: Revisiting large kernel design in cnns." CVPR (2022)
[2] Luo et al. "Understanding the effective receptive field in deep convolutional neural networks." NeurIPS (2016)

Thank you for your valuable feedback—it’s been incredibly helpful in improving our work. As today is the final day for reviews, please don’t hesitate to ask if you have any remaining questions. We’d be happy to clarify anything to ensure the work is as strong as possible.

We sincerely thank the reviewers for their overwhelmingly positive and encouraging feedback. We deeply appreciate the recognition of our efforts to craft a comprehensive narrative bridging neuroscience and AI. The absence of identified weaknesses and the reviewers' enthusiasm for our approach reinforce our confidence in the significance and robustness of this work. This feedback inspires us to further explore and advance the potential of biologically inspired neural networks.

Question 1) Connections with Anisotropic Diffusion

Thank you for your insightful and constructive comment. We greatly appreciate the suggestion to explore connections between Lp-convolution and anisotropic diffusion (Perona & Malik, 1990), as well as the broader scale-space theory outlined by Koenderink (1987) and Lindeberg (1994). While our current work emphasizes practical and empirical aspects of Lp-convolution, we agree that its theoretical mapping to diffusion processes could be a fascinating direction for future research.

In particular, the parameterized adaptability of p in Lp-convolution provides a natural mechanism to bridge isotropic and anisotropic processes. For example, varying p could function similarly to the diffusion coefficient in anisotropic diffusion, dynamically controlling feature emphasis and smoothing based on local structures. This adaptability could also extend scale-space representations by offering a flexible, multi-scale feature extraction framework. Thank you again for this stimulating idea, which offers valuable guidance for future exploration. We will carefully consider this perspective in our ongoing and future work. Again, we are truly delighted to receive such positive evaluations of our research.

Thank you for your valuable feedback—it’s been incredibly helpful in improving our work. As today is the final day for reviews, please don’t hesitate to ask if you have any remaining questions. We’d be happy to clarify anything to ensure the work is as strong as possible.

Weakness 3) Experiment on Representational Similarity Not Convincing

We appreciate the reviewer’s feedback and understand the concerns regarding the representational similarity (RSM) results, specifically:

1. small effect sizes and modest significance
2. inclusion of non-ventral stream regions in the analysis.

However, we respectfully argue that our RSM results are compelling and well-supported in addressing the primary objective of our study: to explore whether introducing biologically inspired constraints into CNNs enhances their alignment with the brain. Below, we provide detailed responses to each concern.

1. Small Effect Size and Modest Significance
While the observed effect sizes in RSM analysis may appear modest, we believe our results are robust, meaningful, and well-aligned with prior research. Key points supporting this include:

• Robustness Across Architectures:
  The observed trends consistently demonstrate that CNNs incorporating Gaussian sparsity (a biological constraint) achieve better alignment with brain representations compared to others. These trends are consistent across various CNN architectures, despite the inherent variability and complexity of biological data.

• Meaningful Effect Sizes Relative to Accuracy Gains:
  The SSM improvements for Lp models (e.g., AlexNet) reach approximately 3%, significantly larger than the corresponding Top-1 accuracy gains of around 1%. This disparity underscores that the observed RSM differences reflect meaningful biological alignment rather than statistical noise.

• Reproducibility with Prior Work:
  While the absolute SSM values may seem modest, their range (0.2–0.4) aligns closely with prior findings, such as those reported by Shi et al. (2019) [1]. Moreover, we executed the same codebase as previous work [2], ensuring methodological consistency and the validity of our comparisons.

These points collectively demonstrate that our RSM results, while modest in absolute terms, are biologically relevant and sufficiently robust to support one of our main objectives.

2. Inclusion of Non-Ventral Stream Regions In response to the reviewer’s concern about including non-ventral stream regions, we note that our methodology follows established practices from prior studies [1–3], which analyze a broad range of visual areas in the mouse cortex. Our rationale for including both ventral and dorsal regions is as follows:

• Holistic Evaluation of Representational Capacity:
  While ventral stream regions like VISp and VISl are directly relevant to "what" tasks such as image classification, dorsal regions (e.g., VISam, VISpm, VISal) offer insights into the broader representational capacity of CNNs. By analyzing both streams, we aimed to provide a more comprehensive evaluation of CNNs in relation to biological systems.

• Focused Analysis on Ventral Stream Regions:
  To ensure clarity, region-specific SSM analyses are provided in Appendix A.12. These results clearly show that the ventral stream region VISl consistently achieves the highest SSM values across all CNN architectures. Consequently, the maximum SSM values reported in Figure 6 are derived from VISl, confirming that our primary analysis is appropriately focused on ventral stream processing and directly relevant to tasks like image classification.

By incorporating both ventral and dorsal regions, we situate our findings within a comprehensive framework for evaluating CNNs in the context of biological visual systems. The dominance of VISl in our results further substantiates the alignment of CNNs with ventral stream processing, reinforcing the validity and significance of our conclusions.

References

[1] Shi et al. "Comparison against task driven artificial neural networks reveals functional properties in mouse visual cortex." NeurIPS (2019)
[2] Bakhtiari et al. "The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning." NeurIPS (2021)
[3] Shi, Jianghong et al. "MouseNet: A biologically constrained convolutional neural network model for the mouse visual cortex." PLOS Computational Biology (2022)

Weakness 2) Evaluation Limited to "Toy" Datasets

Given the availability of well-established large-scale datasets like ImageNet, smaller datasets such as CIFAR-100 and TinyImageNet are often perceived as 'toy' datasets. However, these smaller datasets are pivotal for examining novel inductive biases, as highlighted in numerous prior studies [1–5]. This is because, as data size increases, the impact of inductive biases tends to diminish due to scaling effects [6, 7]. For example, CNNs outperform ViTs in data-scarce regimes due to their strong locality inductive bias, while enriched data can obscure the benefits of this bias [1, 7]. Thus, the choice of these smaller datasets was essential for effectively investigating the potential of our proposed inductive biases.

At the same time, we fully recognize the importance of evaluating our method on closer-to-real-world datasets like ImageNet-1k to better understand its broader applicability and potential impact. As this study represents an initial exploration of a novel inductive bias, our primary goal was not to compete for SoTA performance but to demonstrate the conceptual strength and practical utility of our approach. Rather than training models from scratch on ImageNet-1k, we deliberately adopted a transfer learning strategy to integrate our method with ImageNet-1k pretrained SoTA models (Figure 3).

The transfer learning results highlight both the conceptual and practical advantages of our method (Table 4). Although the observed effect sizes were modest, our method seamlessly integrated with pretrained models, enhancing their adaptability without diminishing their performance. This outcome underscores the utility and flexibility of our approach, demonstrating its potential to advance transfer learning applications and contribute to the study and application of novel inductive biases.

References

[1] Lee et al., "Vision Transformer for Small-Size Datasets." arXiv preprint arXiv:2112.13492 (2021)
[2] Verma et al., "Manifold Mixup: Learning Better Representations by Interpolating Hidden States." ICML (2019)
[3] Zagoruyko et al., "Wide Residual Networks." BMVC (2016)
[4] Feng et al., "Conv2NeXt: Reconsidering ConvNeXt Network Design for Image Recognition." CAIT (2022)
[5] Hu et al., "Unlocking Deterministic Robustness Certification on ImageNet." NeurIPS (2024)
[6] Bachmann et al., "Scaling MLPs: A Tale of Inductive Bias." NeurIPS (2024)
[7] Zhang et al., "ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond." IJCV (2023)

We appreciate the constructive critique from the reviewer expecting the high standards of ICLR. We acknowledge the reviewer’s concerns regarding the lack of the ImageNet-1k benchmark and the absence of SoTA results, especially in light of multiple revision opportunities. While we share the ambition of presenting such results, at this early exploratory stage of our novel idea, we regret that we were not yet able to achieve those outcomes. That said, we firmly believe that our research still offers meaningful contributions to the ML community, and we hope the reviewer will appreciate the broader value of our work in advancing novel directions within the field. Below, we address the concerns raised and provide clarifications where needed.

Weakness 1) Small Effect Sizes and Limited Practical Impact

We agree with the reviewer’s observation that the effect sizes might appear small compared to the state-of-the-art (SoTA) models. Indeed, it is impressive that Astroformer achieves 93.4%, outperforming the second-best PyramidNet (89.9%) by more than 3% solely through architectural modifications [1]. However, we kindly request the reviewer to consider the distinct values of our work beyond achieving SoTA performance. Specifically, our contributions lie in:

1. exploring the potential of novel biologically inspired inductive biases, and
2. developing a new, easily pluggable module for CNNs.

1. Novel inductive bias

We acknowledge that the prior works that explore biological ideas focus on providing novel insights and, hence often fail to surpass original ML methods in raw performance [2-4]. Nonetheless, we have put ourselves into situations to provide not only novel insights but also practical improvement to the ML community. We have demonstrated consistent improvements in various architectures and tasks, underscoring the robustness and versatility of our approach. While our results are not SoTA, we believe our work has still meaningful contributions to the field by introducing novel inductive bias to the community.

2. Easily pluggable module

Historically, CNNs have evolved through the introduction of innovative modules. For example, a depthwise convolution module was originally proposed to improve computational efficiency not SoTA in MobileNet [5] and now plays a pivotal role in SoTA architectures like ConvNeXt and RepLKNet [6, 7]. Similarly, CoAtNet, a precursor to Astroformer, leveraged a hybrid module combining depthwise convolution and self-attention, enabling Astroformer to achieve SoTA solely through architectural refinements [8]. In this context, our proposed Lp-convolution module offers practical value as a pluggable component that is easily integrated into existing architectures, facilitating flexible and efficient deployment (See Appendix A.21).

We believe that exploring novel bio-inspired algorithms and providing new convolutional modules enrich the machine learning community by offering both theoretical insights and practical tools. By contributing an additional design choice that enhances robustness and versatility across architectures, our work supports innovation in machine learning model design.

References

[1] Dagli, Rishit. "Astroformer: More data might not be all you need for classification." arXiv preprint arXiv:2304.05350 (2023)
[2] Pogodin, Roman, et al. "Towards biologically plausible convolutional networks." NeurIPS (2021)
[3] Liu, Yuhan Helena, et al. "Biologically-plausible backpropagation through arbitrary timespans via local neuromodulators." NeurIPS (2022)
[4] Kao, Chia Hsiang, and Bharath Hariharan. "Counter-Current Learning: A Biologically Plausible Dual Network Approach for Deep Learning." NeurIPS (2024)
[5] Howard, Andrew G. "Mobilenets: Efficient convolutional neural networks for mobile vision applications." arXiv preprint arXiv:1704.04861 (2017)
[6] Liu, Zhuang, et al. "A convnet for the 2020s." CVPR (2022)
[7] Ding, Xiaohan, et al. "Scaling up your kernels to 31x31: Revisiting large kernel design in cnns." CVPR (2022)
[8] Dai, Zihang, et al. "Coatnet: Marrying convolution and attention for all data sizes." NeurIPS (2021)

Thank you for the detailed response. I remain unconvinced and will maintain my rating.

1. I do see that it's a novel inductive bias and easily pluggable – but to what end? What problem does it solve? Being biologically inspired is not an end in itself if it does not address an open problem.

2. Again, what problem does this inductive bias solve? We have strong backbones that can be used for transfer learning and we have better methods for these toy tasks. If your inductive bias solves some data-scarce problem (better than previous methods, including transfer learning from backbones that exist and can be downloaded), I suggest you present the evidence. Table 4 is one example for super tiny effects, so I don't consider it support for your claim that your method is useful.

3. I do not agree that they are robust across architectures. The differences are not statistically significant in 3 of 5 architectures. Percent improvements between RSA and accuracy are not comparable, as they are not on the same scale. I did not criticize the absolute magnitude of your RSA values; I am aware of them often being small, especially in the mouse when compared to task-trained CNNs.

Re 2) Again, what problem does this inductive bias solve? [...] I suggest you present the evidence. [...]

In addition to the Large Kernel Problem we’ve discussed in Re 1), we also tried to showcase the strengths of our method through its application to transfer learning. However, the effect size from the transfer learning results might feel underwhelming to you. Reviewer thinks if we are to claim our method is effective, we need to back it up with stronger evidence.

So, how about we take a look at our robustness experiments (Table 2)? The effect size there is much more significant. For example, in the case of the ConvNeXt network, the raw values show that our method almost doubles the robustness performance compared to the baseline across various types of corruption.

We haven’t fully investigated why Gaussian sparsity has such a dramatic impact on robustness yet. However, we believe this result lets us claim that our method is beneficial for model robustness along with the Large Kernel Problem.

Re 3) I do not agree that they are robust across architectures. The differences are not statistically significant in 3 of 5 architectures. Percent improvements between RSA and accuracy are not comparable, as they are not on the same scale. I did not criticize the absolute magnitude of your RSA values; I am aware of them often being small, especially in the mouse when compared to task-trained CNNs.

First off, thank you for clarifying that your critique isn’t about whether our experiments were conducted properly, but rather about whether the statistical significance level of our results is strong enough to support our claims. That’s a really helpful distinction, and we appreciate you pointing it out.

We observed a mostly decreasing trend as p_init increased, and when we said “robust,” we meant that the trends were generally consistent across different conditions. However, as you rightly pointed out, the significance levels stand out primarily in AlexNet and ConvNeXt.

Throughout previous revisions, we have also included the neural activity prediction of V1 (Table 5) using Lp-CNNs to support our claim. When we replaced the CNN in a CNN-GRU network (same model from previous studies) with our Lp-CNN, we found that Gaussian sparsity achieved the best performance. We think this result, combined with our RSA results, could support the idea that Gaussian sparsity allows CNNs to better align with the brain. We’d love to hear your thoughts on this—do you think it strengthens our argument, or is there something else we should consider?

Thank you again for taking the time to provide feedback. It means a lot to us. Again, it’s not just about getting higher scores—your constructive critiques are what truly help us improve. Please don’t hesitate to share more of your thoughts anytime. We deeply value them!

Thank you for taking the time to respond to our writing. It genuinely means a lot to us that you’ve come back to offer feedback. We believe helping us doesn’t just mean increasing our scores. We sincerely welcome your constructive criticism at any time—it has been an invaluable foundation for shaping our research direction. Following, we would like to clarify regarding the reviewer's additional feedback.

Re 1) I do see that it's a novel inductive bias and easily pluggable – but to what end? What problem does it solve? Being biologically inspired is not an end in itself if it does not address an open problem.

Yes, we completely agree with your point: “Being biologically inspired is not an end.” You pointed out that it’s unclear what open problem our method is addressing, and we think this might be because we didn’t effectively communicate our problem statement. In fact, we’ve proposed an open problem here: the “Large Kernel Problem.”

Let us explain. We revisit the large kernel problem as introduced in line 58. Large kernels in CNNs often fail to show consistent performance improvements as kernel sizes increase, even when additional parameters are allocated [1]. This raises an important question in machine learning: Can we expect performance gains by expanding kernel sizes horizontally, beyond the traditional vertical stacking of layers?

To clarify this further, please take another look at Table 1. When comparing the (Base) condition with the (Large) condition—where kernel sizes are increased—we see that performance generally declines across models, except for ResNet. This underscores our key point: if we’ve successfully communicated what is the large kernel problem here, the comparison that truly matters should be Large vs. Lp-Conv, not Base vs. Lp-Conv.

Here’s why our method stands out: it is explicitly designed to ensure that, regardless of the initial p-value, the Base model’s performance serves as a lower bound for the expected performance of large kernel CNN with Lp-Masks. This makes our approach far more stable than simply training large kernels arbitrarily, while still achieving significant performance gains.

How is this possible?

This stability arises because the enlarged kernels overlaid with Lp-Masks make the central parameters prioritized for learning as if initialized like Base model (see Figures 2c, d, e or Figure 3). Over time, the Lp-Mask dynamically changes its conformation in a task-dependent manner by expanding, contracting, narrowing, elongating, or rotating as needed. To validate this mechanism, we performed the Sudoku task (please see trained individual Lp-Masks in Appendix A.7—it’s worth checking out how they adapted their shapes during training!). For example, in Sudoku—a combination of vertical, horizontal, or square constraints but not diagonal goals—we found that no Lp-Mask formed a diagonal shape, underscoring its adaptability. This dynamic behavior is key to the performance gains we observed.

Did our method solve the Large Kernel Problem?

In a way, we believe it depends on how you interpret the results. Our approach is not just about achieving dramatic performance gains immediately but rather about providing a guaranteed lower bound on performance when enlarging kernels in CNNs.

To put this in perspective, think about how vertically extending a model—for instance, increasing ResNet-18 to ResNet-34 by doubling the number of layers—yields approximately a 1.25% performance gain. Similarly, our method achieves a stable 2.64% performance improvement on ResNet-18 by horizontally extending the model size through kernel enlargement. From our perspective, this represents a meaningful step toward addressing the Large Kernel Problem.

By stabilizing performance when increasing kernel sizes, our method offers a solid foundation for further exploration and refinement. We’re happy to discuss and answer more questions regarding this perspective—thank you for your thoughtful engagement!

References

[1] Peng, Chao, et al. "Large kernel matters--improve semantic segmentation by global convolutional network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

Thank you for your valuable feedback—it’s been incredibly helpful in improving our work. As today is the final day for reviews, please don’t hesitate to ask if you have any remaining questions. We’d be happy to clarify anything to ensure the work is as strong as possible.

Weakness 2) Missing ViT Baselines in Robustness and Sudoku Benchmarks

Thank you for pointing out the lack of ViT baselines for the Sudoku task. While we relied on a previously reported CNN model [1], we could not find ViT implementations specifically designed for this problem. Although Recurrent Transformer models, such as Yang et al. [2], achieve over 95% accuracy on textual Sudoku, no comparable ViT implementation exists for direct benchmarking.

To address this, we designed a custom ViT architecture with a final linear projection reshaped into[batch_size, 9, height, width], SudokuViT, tailored for Sudoku puzzles. Two variations were explored:

1. SudokuViT3x3: Processes 3 by 3 patches as sequences.
2. SudokuViT1x1: Treats each pixel as a token, effectively functioning as a standard Transformer.

Unfortunately, SudokuViT3x3 struggled to train effectively, achieving only 27% number accuracy. SudokuViT1x1 showed slight improvement, with 49% number accuracy, but still performed poorly on Sudoku-specific metrics such as row-column accuracy and box accuracy.

These results suggest that ViTs face significant challenges in capturing the grid-like structure and spatial relationships inherent to Sudoku puzzles, which CNNs handle effectively. However, we acknowledge that this performance could also stem from our inability to identify an effective training strategy or optimized model design for SudokuViT. Thus, it may not be appropriate to treat these results as definitive baselines for ViT performance on Sudoku.

For future work, we recommend referring to Yang et al. [2], where a Recurrent Transformer approach demonstrated strong performance on textual Sudoku, as a more suitable baseline for Transformer-based methods.

We appreciate the reviewer's suggestion, as it allowed us to explore the limitations and potential of ViT architectures for structured reasoning tasks like Sudoku. This exploration provides valuable insights into the architectural trade-offs between CNNs and Transformers in such domains.

References

[1] Oinar. “How to solve sudoku with convolutional neural networks (cnn).” GitHub link (2021)
[2] Yang et al. "Learning to solve constraint satisfaction problems with recurrent transformer." ICLR (2023)

Weakness 3) Redundant Section on RSM Similarity Comparison

Thank you for your valuable feedback on the RSM experiments. We recognize that the necessity of this section may not be immediately clear. However, one of the key goals of our study is to investigate whether integrating biologically observed inductive biases into CNNs not only enhances engineering performance but also improves alignment with brain representations.

The section 6 plays a critical role in demonstrating this connection. For example, biases inspired by the V1 (e.g., Gaussian sparsity, Appendix A.3) are shown to improve representational alignment with neural activity while simultaneously offering engineering benefits. This dual advantage underscores the value of biologically motivated constraints in informing model design.

We agree that further studies involving more datasets, brain regions, and CNN variations would strengthen our findings. As part of these efforts, we demonstrated neural activity prediction experiments (Table 5), which provide additional evidence of the alignment between CNN representations and biological data.

We hope this clarification justifies the inclusion of Section 6, emphasizing its role in illustrating the complementary relationship between neuroscience and AI.

We sincerely thank the reviewer for their thoughtful and encouraging feedback, which deeply motivates us. Below, we address the concerns raised and provide further clarifications.

Weakness 1) Limited Utility Demonstrated for Large Kernel CNNs

Thank you for your valuable feedback. We would like to address your concerns by highlighting the strengths of our research and providing our perspective on the issues raised.

1. Lp-Convolution for Large Kernel Problem

Firstly, we reiterate our discussion of the large kernel problem mentioned on line 58. While recent large kernel CNNs do not guarantee performance improvements when increasing the kernel size, even though additional parameters are secured (see Tables 1 and 4). This raises a critical machine learning question: Can we expect additional performance gains by moving beyond traditional vertical layer stacking and expanding the kernel size horizontally?

In Table 1 and 4, the performance gains when comparing our method with legacy models (Base vs. Lp-Conv) may appear modest. However, when applied to large kernel models (Large vs. Lp-Conv), we observe a significant increase regardless of the initial value of p. Specifically, in Table 3, applying Lp-Convolution to RepLKNet—a model that already utilizes large kernel sizes—resulted in a 1% performance improvement with almost no additional computational cost. We believe this constitutes a meaningful enhancement, demonstrating the practical benefits and reinforcing the potential of large kernel CNNs to improve both efficiency and performance.

2. Strong Robustness with Lp-Convolution

In Table 4, we demonstrate the, the Lp-Convolution model with p=2 outperforms the Base model in 57 corruption scenarios, whereas the Base model does not achieve a single win. This consistent superiority is not only reflected in the win counts but also in the absolute performance values, which we include in the raw performance table of ConvNeXt as follows.

When comparing these values, it becomes evident that our method exhibit exceptional robustness across various types of corruptions.

3. Comparison with ViT

Thank you for your valuable feedback. While our primary focus is on improving CNN architectures, we agree with the reviewer that including a comparison to ViTs can better highlight the utility and relevance of our proposed method. To address this, we conducted experiments with ViT base models on TinyImageNet, and the results are summarized below:

As shown in Table, Lp2-AlexNet achieves comparable performance to ViT-16x16 on TinyImageNet with significantly lower parameter counts and computational cost, demonstrating its efficiency. Thank you for raising this important point. We will include this comparison into our revised manuscript to provide a clearer perspective on the relationship between ViTs and CNNs.

Thank you for your valuable feedback—it’s been incredibly helpful in improving our work. As today is the final day for reviews, please don’t hesitate to ask if you have any remaining questions. We’d be happy to clarify anything to ensure the work is as strong as possible.