CROSS-CHANNEL ACTIVATION FUNCTION WITH PASS-THROUGH RATIO CONTROL

1. The experimental results only include the small-scale databases. The authors mentioned that the imagenet-1k results are included in Appendix E, but I cannot find them there. Moreover, I believe that Imagenet-1k results are important for this paper, which should be included in the main paper instead of appendix.

2. It is better to consider the time-cost of SPA. Is it similar with traditional activation functions?

• The paper claims that ReLU suffers information loss by eliminating negative features. However, the proposed method also eliminates features. Moreover, the ReLU masks element-wise features while the proposed method masks channel-wise features. It seems the proposed method would suffer more information loss. Therefore, I have doubts about the analysis regarding the shortcomings of ReLU and how the proposed method improve it.

• According to the experimental results, the improvement brought by SPA is marginal. An average accuracy and its standard deviation over multiple runs should be provided.

• Besides, suppose SPA does improve upon ReLU and GELU. The improvement brought by SPA seems not substantial compared to the increase in computational cost for a much more complicated activation function. I would appreciate a more comprehensive time complexity analysis of the proposed SPA.

Mistake in Mathematical Expression

One of the specific weaknesses in the paper is the definition of the set $S$ used in the SPA function. The paper states $S = \{x = [x_1, x_2, \ldots, x_C] \mid |x_1| + |x_2| + \cdots + |x_C| \leq \delta\}$ without explicitly requiring $x \geq 0$. This omission may compromise the theoretical foundation, as the non-negativity constraint is crucial for the simplex projection and the activation function's behavior. The authors should clarify this condition to avoid any misinterpretation.

Misleading Illustration

The three-dimensional illustration in Figure 1(b) appears to be hand-drawn, with the projection directions of various points and the axes appearing inconsistent and chaotic, which may mislead readers. High-quality and accurate visual representation is crucial for conveying mathematical concepts, and the quality of this figure does not meet this standard. The authors should consider revising this figure using mathematical 3D space plotting tools such as GeoGebra to ensure it accurately represents the SPA projection without misleading readers.

Theoretical Implications

While the paper provides a thorough experimental evaluation, it could benefit from a deeper theoretical analysis of the SPA function. Specifically, the paper could explore the theoretical implications of the SPA function on network convergence and generalization. A more in-depth theoretical discussion would strengthen the paper's contribution and provide a stronger foundation for the experimental results.

Generalization to Other Network Architectures

The paper focuses on the application of SPA to CNNs, but does not extensively explore its potential application to other types of neural network architectures, such as transformer models. Expanding the scope of the paper to include experiments or a discussion on the applicability of SPA to these architectures would enhance its significance and impact.

Discussion on Limitations

The paper could benefit from a more explicit discussion on the limitations of the SPA function. For example, the authors could discuss potential challenges in optimizing the δ parameter for deep networks or the computational overhead introduced by the SPA function. Acknowledging and addressing these limitations would provide a more balanced view of the SPA function's practical applicability.

In summary, the paper's weaknesses can be addressed by clarifying mathematical definitions, improving visual representations, expanding the theoretical analysis, exploring the applicability to other network architectures, and discussing the limitations of the proposed method. By addressing these points, the paper could provide a more comprehensive and robust contribution to the field of neural network activation functions.

1. The concept conveyed in the paper lacks significance. The author introduced two methodologies: cross-channel relation for activation and threshold v* to filter out irrelevant features.

1.1) The author posited that "These functions often process inputs separately, neglecting dependence between them, such as the spatial or cross-channel relation of the features. Spatial relation refers to the local connectivity and neighborhood structure of the features, while cross-channel relation refers to the correlation and diversity of the features across different channels. " In my opinion, the convolutional operation already calculates the cross-channel relation of the features. Therefore, introducing another cross-channel relation for activation function seems superfluous.

1.2) Regarding the threshold v*, feature normalization and bias serve a similar purpose. Consequently, the significance of the threshold appears diminished.

2. The proposed methodology has only been validated on toy datasets and tiny ImageNet. Larger-scale datasets are imperative. In my view, if the model is trained with an adequate number of dataset samples, the original cross-channel relation learned through convolution operation and the threshold will be well assimilated by the model.

3. The in-depth analysis explaining why deep models necessitate additional cross-channel relation and threshold parameters is absent.

4. The literature review is lacking. Several crucial and highly relevant works are absent.

References: [1] Dynamic Neural Response Tuning, ICLR 2024. [2] Exploring optimal adaptive activation functions for various task，IEEE BIBM 2020. [3] Exploring Optimal Adaptive Activation Functions for Various Tasks, 2020. [4] Deep sparse rectifier neural networks. JMLR 2011 [5] Density Modeling of Images using a Generalized Normalization Transformation, CoRR 2015. [6] ...