Hadamard Representations: Augmenting Hyperbolic Tangents in RL

We thank reviewer HMtR for the rigorous review of our paper.

1. “The experiments are conducted on a very small fraction of only 8 hand-picked Atari games. While the performance of the proposed method is slightly higher than ReLU on average, this does not hold consistently even across these 8 games (as reported in the Appendix). Moreover, while these environments are notably sensitive to stochasticity, evaluation is carried out with only 5 random seeds I believe this to be very much insufficient for a paper with a simple architectural contribution that should be thoroughly validated. Thus, I am very much not convinced of the generality of these results, even across the full set of Atari environments, and would strongly encourage to expand this section. In case Atari is too demanding, I would suggest considering other more computationally-friendly benchmarks such as the 16 environments in the highly-parallelizable Procgen benchmark.”

We understand the reviewer’s point about the limited evaluation. Although we do believe that our paper provides insights into the dying neuron phenomenom and an activation function’s correlation with it, we have additionally ran a set of experiments on the full Atari suite. We have evaluated the Hadamard representation on a very recent vectorized version of DQN called Parallelizable Q-Network (PQN) [1]. Using PQN, we were able to run the (nearly) full atari suite of 51 games, and compared to tanh and ReLU.

2. “Even in the results of this paper, there does not seem to be a clear relationship between dead neurons/effective rank and performance, (e.g., ReLU outperforms Tanh while having a higher number of dead neurons). Thus, even analytically, I feel the paper in its current state leaves many unresolved questions and provides very little novel information.”

We agree with the reviewer’s point that there remains an analytical gap as to the strong ReLU performance as compared to Tanh. We have added a section in the main paper that goes more into depth on this subject. This section is highlighted in red and can be found on page 9. Specifically, we argue that, in contrast to the ReLU, dying hyperbolic tangents generate a bias on the next layer.

3. “I believe the claims about the probabilities of neurons 'dying' such as: "Taking a product of hyperbolic tangent activated neurons thus reduces the probability of neuron saturation from p to p^2." (ln242-243) are not correct, and potentially misleading, as they make independence assumptions that are not even reported by the paper's own results in Appendix C.3.”

You are correct. We will state these independence assumptions. We have slightly rewritten the corresponding explanation in the main paper as well as in Appendix C.3.

4. “In the MLP block of several modern LLMs such as Llama, the hidden representation is computed as (using this paper's notation) f(A_1x + B_1) * (A_1x + B_2). According to the paper's own analysis in Section 3, this simpler version should entirely prevent the dead neuron phenomenon with respect to the input activations. I wonder if the authors have considered and/or have this version.”

We were not aware of this version, as we had only seen these multiplicative interactions before in the Highway Network paper [3]. We have now added a reference to Gated Linear Unit [4] in our algorithm section, as we found that the function you proposed is derived from [4] (And our function is similar to it). Although we think this could also be a good activation, we do not believe that this version would completely prevent the dead neuron phenomenon. The f in f(A_1x + B_1) * (A_2x + B_2) represents the Swish function. The Swish also saturates into 0 for decreasing inputs, meaning that the final product will remain 0 if the Swish-activated representation saturates into 0.

5 (minor): To allow readers to visualize the problems with activations in hidden layers, we respectfully chose to keep figures 3 and 5 intact. In the experiment section, we already have quantitative dying neurons plots.

We hope to have taken away most of the reviewer's issues with the old manuscript. We kindly request that you review the updated manuscript in light of these clarifications and revisions, and update the score accordingly. If there are any more clarifications needed please do not hesitate to reply.

[1] Gallici et al. Simplifying Deep Temporal Difference Learning. Arxiv 2024

[2] Delfosse et al. Adaptive Rational Activations to Boost Deep Reinforcement Learning. ICLR 2024

[3] Srivastava et al. Highway Networks. 2015

[4] Dauphin et al. Language modeling with gated convolutional networks. PMLR 2017

Thanks for your response and for running some additional experiments.

However, I do not find my initial criticism to have been fully addressed:

1. In the caption of Figure 10, it reads that the Hadarmard representations (in contrast to the other results) are being applied at every layer of the network:

• I would really appreciate if the authors would clarify why the divergent implementation from their previous applications.
• Most importantly (please, correct me if I happen to be wrong in my understanding): doesn't this imply the number of parameters is doubled for each layer? This seems like a major limitation that is never discussed and that makes the PQN comparison unfair.

2. The new PQN results also lack clear details. The authors mention they do not evaluate Atari's hard exploration games, yet, I could not find what exactly these games are. Moreover, only the median results are reported over 5 seeds, without per-task results.

3. As the authors seemed to have agreed that the independence assumptions about neurons dying contradict the paper's own empirical results, I do not think adding a single sentence "Lastly, we make an independence assumption between two individual neurons." makes the paper's analysis any more valid.

Once again, I would like to state that I appreciate the research direction and its potential value. However, it seems to me that paper in its current form is still not sufficient, especially from an empirical perspective. Furthermore, much information and limitations seemed to not be clearly stated in the text (something which I found to be worsened in the current revision). For these reasons, I will not be modifying my score at this moment.

We thank reviewer Vagz for the review, and the positive note at the end. We will reply to all the weaknesses and questions the reviewer has noted.

1. "The authors claim that the dying neuron problem is not exclusive to ReLU but also extends to Tanh and provide a definition where the activation output . - - (Omitted for response length)"

You are right about the definition not working for ReLU. We have changed the definition and the explanation for it in line 137. However, what do you mean that intuitively, the neuron is not 'dead' if its output is not zero? We think of dying neurons as having no variation according to the input observation, meaning they are basically collapsed. We feel like a dead ReLU is a saturated ReLU, just like a dead Tanh is a saturated Tanh (Only into 1 end. so always only 1 or only -1). Please let us know if you have any other thoughts on this.

2. "The authors mentioned that the definition of a dead/saturating neuron applies when we test the neuron in all data in the buffer. Is this quantity actually being measured in the experiments? - “” “”."

Thanks for the observation, we indeed forgot to mention our approximation to this. We sample a batch from the buffer to calculate the dead neurons. We have added this explanation in Line 138 in the revised Manuscript.

3. "The empirical evaluation could be more substantial. - "" """

As this was an overlapping theme across reviewers, we have added experiments using a parallelizable version of DQN called PQN [1] on the (nearly) full Atari suite of 51 games. We report the median Human-normalized scores on page 9 of the new manuscript.

4. "More relevant activation function baselines are needed. - "" """

We have run additional experiments on our 8-environment DQN baseline, where we compared against the recent novel Rational [2] activation over 40M frames. We have updated figure 7a and Fig. 19 accordingly. As for the concatenated ReLU [3], this paper itself does not report any performance improvement over the non environment-resetting baseline (See Fig. 6 of [3]), which is why we have added the paper to related work but not compared against it. We hope this clarifies things.

Questions

1. "The authors mentioned in line 367 that they considered addition operation instead of Hadamard product and showed it performs poorly. There was no reason or postulate for why this is the case. Can the author provide a probability analysis similar to the one given on page 5?"

We believe that multiplicative representations also have different function approximation properties as compared to additions. As for the dying neuron analysis, this should give the same results as for a product of hyperbolic tangents, i.e: The final representation is only dead if both neurons are dead. However, due to poor performance of these additive representations, we have not looked further into depth on their exact functioning.

2. "*If the score is normalized with respect to the ReLU activation baseline score, why doesn’t the final ReLU baseline reach 1 at the end of training in Figure 7a and Figure 9a but slightly less than 1?"

If you look closely, the top of the uncertainty of the ReLU curve should hit 1. We normalize with respect to the highest and lowest score achieved by the ReLU per environment.

3. "* Why did the authors use HR Tanh only for the last layer?*"

Preliminary results showed a strong preference for CNN's towards the ReLU. However, for more clarity, we are now running additional experiments showing tanh and hadamard representations on the CNN's aswell. We hope to add them to the Appendix at the beginning of next week. As to the second part of your question, the architecture only has 1 hidden linear layer. Or do you mean the Q-values?

4. - Thanks for noticing, we changed this throughout the manuscript.

5. - Agreed. We have removed this paragraph and slightly rewritten this section.

6. "In some environments, such as Seaquest and SpaceInvaders, HR hurts performance compared to ReLU. Is there a reason for why"

Possibly due to a low-frequency/simplicity bias for these environments, but we still see substantial improvement over the tanh. However, we are hesitant to draw causal conclusions for performance differences per environment.

• Note that due to response length limitations, we have visually shortened some of your questions.

We hope to have alleviated the reviewer's concerns for the paper. In light of our response and the revised manuscript, we kindly ask the reviewer to reconsider the given score.

[1] Gallici et al. Simplifying Deep Temporal Difference Learning. Arxiv 2024

[2] Delfosse et al. Adaptive Rational Activations to Boost Deep Reinforcement Learning. ICLR 2024

[3] Abbas et al. Loss of Plasticiy in Continual Deep Reinforcement Learning. CoLLAs 2023

We thank reviewer 3nuW for the review of our paper.

1. “The lack of a diverse range of network architectures tested limits the generalizability of the findings. The paper should test HR on various networks and test HR on different layers. It would be better if HR were a universal activation function suitable for all layers, rather than only suitable for the last layer.”

In the Limitations section on line 533, we have explained that preliminary experiments showed a very strong preference for ReLU activations in the convolutional section of our network. As the networks used for most Pixel-based environments only have 1 hidden linear layer, this is the reason for our choice of this final hidden layer. Furthermore, we believe that this is the most important hidden layer as the Q-values or policies are usually linearly extracted from here. However, we agree with the reviewer that it would make sense to test the Hadamard representation on more layers. We are running additional experiments using a Hadamard representation on our convolutional hidden layers as well. We hope to add this to our Appendix at the beginning of next week.

2. “Absence of detailed comparisons with other novel activation functions that have emerged recently.”

We have conducted experiments on our original 8 Atari games comparing the Hadamard representation to the novel Rational [1] activation function. We have updated Figure 7a in the main paper comparing against this activation. Furthermore, we have updated figure 19 in Appendix D.2 to show the individual runs of the Rational activation.

3. “The paper did not conduct tests on diverse tasks, e.g. MuJoCo, DMControl. using only Atari does not demonstrate that this is a universal issue in reinforcement learning.”

Preliminary experiments have tested our hypothesis on a pixel maze environment, after which we experimented in the Atari domain. We respectfully feel that state-based continuous environments that do not support DQN are out of the scope of our paper. However, to make our evaluation more complete, we have done additional experiments using PQN [2] on the (nearly) full Atari suite of 51 games. The median Human-Normalized scores can be found in our updated manuscript in Figure 10.

4. "As mentioned in Line 279, the experiments are all performed on 8 Atari games. Testing on only 8 Atari games does not sufficiently demonstrate a stable improvement in performance."

See our response to 3.

5. "In the PREVENTING DYING NEURONS section in Line 191, the analysis is not related to reinforcement learning task at all. So, claiming that HR is an activation function particularly suitable for reinforcement learning feels somewhat forced."

We agree that this might not have been evident from the old manuscript. We now have added a section in our introduction which explicitly states that the dying neuron phenomenon is stronger in RL as compared to supervised learning. We leave further tests in supervised learning for future work. However, due to reviewer HMtR, we have noticed that modern LLM's also use (half-linear) multiplicative representations. We have cited to the paper using these architectures [3] in the algorithm section.

Questions 1. See our answer to Weakness 2.

2. In our opinion, the biggest challenges that arise might be finding the perfect combination of activation functions to use in a Hadamard representation. As seen in related work in LLM's [4], there are a lot of possible combinations. Furthermore, as our paper shows, the ReLU does not gain the same benefits as the hyperbolic tangent. As most algorithms use ReLU's in their network architecture, changing all these ReLU's to Hadamard hyperbolic tangents fundamentally changes the representation layers and might require different hyperparameter settings (learning rate, network width).

3. See our answer to Weakness 3.

We hope to have adressed most of the reviewer's concerns. We kindly ask the reviewer to review the updated manuscript in light of our revisions, and reconsider the assigned score. If there are any more questions, please do not hestitate to send another message.

[1] Delfosse et al. Adaptive Rational Activations to Boost Deep Reinforcement Learning. ICLR 2024

[2] Gallici et al. Simplifying Deep Temporal Difference Learning. Arxiv 2024

[3] Dauphin et al. Language modeling with gated convolutional networks. PMLR 2017

[4] Noam Shazeer. GLU Variants Improve Transformer. 2020

We thank reviewer S1DP for the review of our paper. We will reply to both Weakness proposed by the reviewer.

1. “The paper has to convincingly show that Hadamard representations provide a benefit over a well-tuned base system with twice as many hidden units.”

To accommodate for the reviewers’ valid concerns, we have tested the 1024 latent dimensional setup for three different learning rates, and added these to appendix C.2 in our revised manuscript. Over three different learning rates (1e-5, 5e-5, 1e-4) with a 1024 dimensional latent state, it shows that performance increases when decreasing the learning rate, but eventually becomes worse again. Our Hadamard representation still proves to be stronger.

2. “Missing baseline. The paper does not compare Hadamard representations to resetting baseline like Redo [1] and continual backprop[2]. Why not just reset the units that have saturated in the tanh network?”

We understand the reviewers concern of missing baselines. However, a deeper dive into Redo [1, Fig. 10] shows that Redo does not really improve DQN performance in the normal setting, but only when you change the algorithm by increasing the gradient steps per frame from 0.25 to 1. Further work in [3, Fig. 12] also shows that Redo (resetting) can even harm performance. For this reason, although Redo remains interesting related work, we did not implement their algorithm as a baseline but use their work for fundamental insights.

As for [2], we thank the author for this addition which we will add to our related work section. However, as this paper was published 7 days before the ICLR 2025 submission deadline, we were unfortunately not aware of this paper and did not compare with it. To accommodate the reviewer and compare against a suiting baseline, we have run experiments comparing our Hadamard Representation to a novel ICLR 2024 learnable Rational Activation [4] and have added this comparison in our main paper.

We kindly request that you review the updated manuscript in light of these clarifications and revisions and reconsider your score.

[1] Sokar et al. The Dormant Neuron Phenomenon in Deep Reinforcement Learning. ICML 2023

[2] Dohare et al. Loss of plasticity in deep continual learning. Nature 2024

[3] Delfosse et al. Adaptive Rational Activations to Boost Deep Reinforcement Learning. ICLR 2024

As per comment of Reviewers 3nuW and Vagz, we have run additional experiments on the 51 Atari suite for PQN, using both full Hadamard representations (CNN + MLP) as well as only MLP Hadamard representations. We have updated Fig.10 on page 9 with the Median Human-Normalized scores. We show a significant improvement over the novel PQN baseline using the full Hadamard representation over 51 games.

In addition, also as per reviewer requests, we are running the Crelu baseline on the 51 Atari environments, and plan on adding it to the PQN results in Fig.10 tomorrow before the deadline!