Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics

We sincerely thank the reviewer for their constructive feedback and questions. Below, we address the points raised in the review.

Weakness:
While we understand the reviewer’s point that performance improvements may appear small in some cases, we respectfully argue that even fractional improvements in accuracy can lead to substantial impact especially when amortized across many different deep learning applications. Activation functions like GELU and Swish also showed small improvements over ReLU, but they are now widely adopted, improving results in thousands of code bases. We specifically emphasize that, unlike many activations that show task or architecture-specific benefits, GoLU provides more consistent improvements across domains, including vision, language modeling, and generative modeling, as demonstrated in our experiments.

We would also like to emphasize a trend we observed in our Table 2 that the difference in top-1 accuracy between GoLU and GELU on ImageNet increases for larger models:

• RN18: +0.10
• RN34: +0.27
• RN50: +0.56
• WRN50-2: +0.65

In addition to its empirical performance, GoLU has a simple, closed-form definition and is theoretically well-motivated. Our provided CUDA kernel enables easy integration with no added latency, making GoLU a practical activation function that can be widely adopted by the community.

Question on Asymmetry Ablation:
We appreciate the reviewer’s suggestion to ablate the role of asymmetry. The specific example, ½ [GoLU(x) + GoLU(−x)] will symmetrize the full activation which is not desirable, however one can symmetrize the underlying distribution which will lead to ½ x [1+Gompertz(x) -Gompertz(−x)] (the derivation follows similar lines to our eq.19). This does not enjoy the properties of GoLU and is expected to lead to lower performance.

Instead, in Appendix B, we have followed a relatively similar idea, but have taken advantage of the insights gained in this work to explore whether we can improve an existing activation. Specifically, we took Mish, which is based on a left-leaning distribution, and flipped its distribution to obtain an activation with right-leaning asymmetry, which we refer to as Flipped Mish (FMish). We observed that this significantly improves performance compared to the original Mish activation and, surprisingly, makes FMish outperform symmetric activations like Swish and GELU, while it still does not surpass GoLU. This result aligns closely with what their slopes at the origin suggest: the slope of FMish (0.4) is lower than Mish (0.6) and Swish/GELU (both 0.5), but still slightly higher than that of GoLU (0.37).

Additionally, following the reviewer’s suggestion, we also conducted a more comprehensive ablation on the slope of GoLU. We introduced a parameter $\beta$ which controls the slope $\mathrm{GoLU}_{\beta}(x) = x e^{-\beta e^{-x}}$ and conducted an ablation study on this parameter for ResNet-34, ResNet-50, ViT-B/32, and ViT-B/16. The results, provided in the table below, indicate that for all these models, decreasing $\beta$ below 1 (which increases the slope) degrades performance, aligning with our expectations. Moreover, for ResNet-34, increasing the $\beta$ parameter to 1.5 or 2 (which decreases the slope) improves performance before degrading again at higher values. This shows that the default value ($\beta=1$) provides a balanced variance control in most cases, although some architectures may benefit from further reducing the slope.

Question on Interaction with Optimizers:
While we did use momentum-based optimizers in many of our vision experiments, we also evaluated GoLU with adaptive optimizers such as AdamW in multiple settings:

• TinyViT, ViT-B/32 and ViT-B/16 on ImageNet-1k
• GPT language models on TinyStories and OWT
• Denoising Diffusion Probabilistic Model on CelebA

In all of these cases, GoLU outperformed baseline activations, which suggest that GoLU’s improved generalization complements adaptive optimizers as well.

Additionally, as an example of a model trained with AdamW, we selected babyGPT trained on TinyStories and conducted the same loss landscape analysis as in Figures 5 and 9. We observed the same qualitative behavior, with GoLU exhibiting both the lowest mean loss and the lowest variance across the landscape:

We hope to have addressed the concerns raised by the reviewer. We respectfully ask that if you feel more positively about our paper, you kindly reconsider your rating accordingly. If not, please let us know if you have further questions or what can be further improved. We are happy to continue the discussion at any time before the end of the discussion period on August 6th. Thank you.

We sincerely thank the reviewer for their positive and constructive feedback. Below, we provide our response.

One of the insights presented in our work is the role of rightward asymmetry stemming from the underlying Gumbel distribution, which contributes to variance reduction and is associated with a smoother loss landscape. This mechanism appears to be an important (though not the only) factor behind the consistent improvements observed across a wide range of tasks.

To further investigate this property, in Appendix B, we applied this insight to Mish, which is based on a left-skewed distribution, and flipped its underlying distribution to construct Flipped Mish (FMish) with right-skewed asymmetry. Despite being a minimal modification, FMish significantly outperformed the original Mish, and also surpassed symmetric activations like Swish and GELU, though GoLU remained the top-performing activation. These results are consistent with our expectations based on the slope of each activation function: FMish has a slope of 0.4 at the origin, which is lower than that of Mish (0.6) and Swish/GELU (both 0.5), but slightly higher than GoLU (0.37).

Based on our findings, the slope near the origin appears to be a meaningful control parameter that could serve as a guiding principle for designing future activation functions (as done for FMish), whether through theoretical analysis, automated methods, or by motivating parameterized activations (please also refer to the table on Asymmetry Ablation in response to reviewer AMoC).

We hope these insights help illustrate the broader potential of our contribution. If there are any further questions or points that could help strengthen your view of the paper, we would be happy to continue the discussion at any time before the end of the discussion period on August 6th. Thank you.

I would thank the authors for the response. I would like to keep my score and keep a positive view on the work.

We sincerely thank the reviewer for their constructive feedback and questions. Below, we address the points raised in the review.

Q1 and W3 (theoretical insight and empirical evidence):
We would like to emphasize that, to the best of our knowledge, this is the first work to introduce asymmetry as a novel and relevant concept in the design of activation functions. This contribution is not merely an empirical observation. We provide a theoretical analysis that directly links rightward asymmetry to variance reduction in latent representations. Specifically, we show that the reduced slope around the origin, a result of the right-skewed asymmetry of the underlying Gumbel distribution, acts as a form of implicit regularization, which is desirable in deep learning.

Beyond the consistently strong performance of GoLU across a wide range of tasks, including vision, language modeling, and generative modeling, Appendix B provides additional empirical evidence highlighting the advantages of rightward asymmetry. We applied our insights to Mish, which is based on a left-skewed distribution, and flipped its underlying distribution to construct Flipped Mish (FMish) with right-skewed asymmetry.

Despite being a minimal modification, FMish significantly outperformed the original Mish, and also surpassed symmetric activations like Swish and GELU when evaluated on ResNet18, ResNet50, and ViT-B/32 trained on ImageNet-1k, though GoLU remained the top-performing activation. This trend aligns closely with our theoretical analysis regarding slope and variance control:

• Mish: slope = 0.6 at the origin
• Swish / GELU: slope = 0.5 at the origin
• FMish: slope = 0.4 at the origin
• GoLU: slope ≈ 0.37 at the origin

Q2 and W1 (Transfer gains with advanced training procedures):
While we understand the reviewer’s concern that some of the performance gains may appear small, we respectfully emphasize that even fractional improvements in accuracy can lead to substantial impact when amortized across many different deep learning applications. Activation functions like GELU and Swish also showed small improvements over ReLU, but they are now widely adopted, improving results in thousands of code bases. We specifically emphasize that, unlike many activations that show task or architecture-specific benefits, GoLU provides more consistent improvements across domains, including vision, language modeling, and generative modeling, as demonstrated in our experiments.

We would also like to emphasize a trend we observed in our Table 2 that the difference in top-1 accuracy between GoLU and GELU on ImageNet increases for larger models:

• RN18: +0.10
• RN34: +0.27
• RN50: +0.56
• WRN50-2: +0.65

Regarding the concern about baseline performance and training pipelines, we fully agree with the reviewer that modern training recipes can significantly improve baseline results. However, our goal in this paper is not to surpass SOTA, but rather to demonstrate that GoLU provides consistent benefits when substituted into established pipelines, without further tuning. For this reason, we standardized our training setup across activations (focusing primarily on original implementations when possible) to ensure a fair and controlled comparison.

That said, based on the reviewer’s suggestion, we trained ResNet-50 on ImageNet-1k using the A3 training recipe from [1]. We used the timm library and followed the hyperparameters listed in Table 2 of [1]. With three random seeds (0,1,2), the ReLU model achieved a mean top-1 validation accuracy of $\bf{78.17 \pm 0.039}$, consistent with [1]. We then simply retrained the model with GoLU, keeping everything else identical, and observed a mean top-1 validation accuracy of $\bf{79.15 \pm 0.024}$, which is a highly significant improvement over the default ReLU activation. This improvement is also consistently observed across the entire training curve.

Regarding the question on fine-tuning, while fine-tuning a ReLU-pretrained model using GoLU may yield improvements, we have not verified this, and this is not a claim made in our paper. It is conceivable that the weights of a model pretrained with ReLU are already aligned with that activation, and simply replacing it during fine-tuning may not reveal the full strength or effect of GoLU. Instead, we expect that future foundation models pretrained with GoLU will show improved performance, which will in turn benefit the resulting fine-tuned variants.

[1] Wightman, R., Touvron, H., & Jégou, H. (2021). Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476.

W2 (discussion on smooth loss landscape):
As noted in the manuscript, the smoother loss landscape observed with GoLU is an empirical observation (rather than a formal argument). While it aligns with properties like GoLU’s smaller slope and variance-reducing effect, it is meant to complement the main experimental results.

We agree that loss landscape visualizations must be interpreted cautiously. However, we would like to clarify that Figure 5 simply illustrates a smoother loss landscape along a 2D subspace spanned by random directions, in order to provide qualitative insight. While not definitive, this is consistent with GoLU’s observed generalization gains. We will clarify this in the final version to avoid potential over-interpretation.

We hope to have addressed the concerns raised by the reviewer. We respectfully ask that if you feel more positively about our paper, you kindly reconsider your rating accordingly. If not, please let us know if you have further questions or what can be further improved. We are happy to continue the discussion at any time before the end of the discussion period on August 6th. Thank you.

We sincerely thank the reviewer for their constructive feedback and questions. Below, we address the questions raised in the review.

1. We will update the captions of Figures 1 and 2 in the revised version of our paper to more clearly distinguish between the gated activations, their corresponding gate functions, and the distributions underlying those gate functions.
2. The three images in Figure 4 are randomly selected and are meant to visually showcase the variance reduction effect and support the theoretical argument. Including more samples would show qualitatively the same behavior and would therefore add limited value. We would also like to note that we observe such variance reduction effects across all intermediate layers of ResNet-50 trained on ImageNet-1k, not just in the final activation. We will comment on this in the revised version of the paper.
3. Motivated by the reviewer’s question, we selected babyGPT trained on TinyStories and conducted the same activation distribution analysis as in Figure 4. We observed that GoLU results in the lowest variance in the final layer’s activations: GoLU (Var: 0.042), GELU (Var: 0.066), ReLU (Var: 0.079), Swish (Var: 0.155). We further performed the same loss landscape analysis as in Figures 5 and 9, and again observed the same qualitative behavior, with GoLU exhibiting both the lowest mean loss and the lowest variance across the landscape. | Activation | Mean Loss | Loss Variance | |------------|-----------|--------------| | Swish | 1.6301 | 3.94e-05 | | ReLU | 1.6259 | 3.93e-05 | | GELU | 1.6133 | 3.93e-05 | | GoLU | 1.6079 | 3.91e-05 |

We hope to have addressed the concerns raised by the reviewer. We respectfully ask that if you feel more positively about our paper, you kindly reconsider your rating accordingly. If not, please let us know if you have further questions or what can be further improved. We are happy to continue the discussion at any time before the end of the discussion period on August 6th. Thank you.

We sincerely thank the reviewer for their positive and constructive feedback. Below, we provide our response.

Con 1:
We have indeed conducted an ImageNet equivalent of the experiment in Table 1 and observed a qualitatively similar variance reduction effect. However, we chose to include the CIFAR-10 results because, to the best of our knowledge, ImageNet images cannot be shared publicly due to license restrictions. In the revised version of the paper we will explicitly mention that the same effect is observed on ImageNet. Additionally, Figure 4 offers a similar analysis on randomly sampled ImageNet images but shows the full activation density distributions. We would also like to highlight that this variance reduction effect is not limited to the final activation, as shown in Figure 4, but is consistently observed across intermediate layers of ResNet-50 as well. We will add a note about this in the updated version.

Con 2 & Question:
Due to space constraints, we had to place several interesting results in the appendix. If the manuscript is accepted, we will have an additional page to move some of the more important results from the appendix, including some theoretical arguments, into the main body.

We hope that our responses have addressed the concerns raised by the reviewer. If there are any further questions or points that could help strengthen your view of the paper, we would be happy to continue the discussion at any time before the end of the discussion period on August 6th. Thank you.