Hadamax Encoding: Elevating Performance in Model-Free Atari

We thank Reviewer kiFq for the extensive review, and acknowledging the practical impact of our encoder design. As the Reviewer’s stated weaknesses are similar to the questions, we will stick to addressing the questions 1 by 1.

• We have done tests integrating Hadamax and the baseline with a mixture-of-expert encoder head, specifically the SoftMoE head [1]. Using a SoftMoE head, we noticed that the performance dropped. We were hesitant to report this, as we could have made an implementation error, which would not have been fair to the SoftMoE authors. Furthermore, the authors of SoftMoE and RL network pruning [2] are working with, to the best of our knowledge, a different Atari domain with sticky-actions, and mostly experiment on a subset of the 57-game suite. It is therefore not possible to do objective comparisons using scores from their papers. In order for us to compare to the official results in the Rainbow and PQN paper, we follow the 57-game, 200M frame implementation on Atari without sticky actions, and report the median Human-normalized scores. Having said this, we do see strong potential in future work integrating e.g. SoftMoE with Hadamax, as Hadamax is focused on the convolutional part of the encoder, while the Mixture of Expert models are focused on the bottleneck part of the encoder.
• We have calculated the parameter allocation between the encoder and the policy/value head:

• We agree that a Hadamax implementation on continuous control benchmarks would offer more insights. However, pixel-based continuous control is, like Atari-57, very compute intensive, as it requires a lot of samples to converge as opposed to state-based continuous control. Even though we concur that testing the capabilities in fine-grained spatial reason would be very interesting, we respectfully leave the integration of the Hadamax encoder in the pixel-based continuous control domain for future work. We did however run an additional evaluation experiment on the actor-critic architecture PPO in Vizdoom (Deathmatch), where we replaced the baseline encoder with Hadamax. While not continuous control, VizDoom also demands spatial reasoning in a dynamic FPS environment. Hadamax also achieves strong performance benefits over the baseline encoder in VizDoom, as shown in the following table:

• In Appendix E.1 we have an ablation where we ‘naively’ scale Hadamax networks in depth, which shows that simply scaling in depth yields decreasing returns for Hadamax (and for the baseline, but that is not shown here). Regarding the residual connections, IMPALA's architecture separates channel increases and downsampling into initial convolutions and pooling layers within the Conv sequences, allowing the residual blocks to operate on constant channels and spatial sizes for identity skips without needing projections. Our Hadamax encoder does not have this, creating channel mismatches that prevent direct residuals. We therefore did not experiment with residual connections. If you would prefer, we are willing to add an ablation to the camera-ready showing (projected) residual connections on the Hadamax encoder.
• Sorry for the confusion, but do you mean, taking the full Hadamax encoder, but substituting the max-pooling with strided convolutional layers, thereby increasing the encoder’s convolutional depth? If that is the case, we are also able to add this ablation to the camera-ready version.
• We have tested the inference times on the Pytorch models:

And also the GPU memory of the Pytorch models:

These tests were done on an Nvidia 3080 Laptop GPU. As expected, Hadamax-PQN uses more GPU memory than Rainbow. However, we kindly emphasize that these numbers are still only a fraction of the available GPU memory on any recent GPU. Furthermore, as also stated in the original PQN paper, PQN saves 26GB of replay-buffer RAM compared to Rainbow.

We thank you for your detailed review, and will implement the additional clarifications / ablations in the main paper and appendices of the camera-ready version, as they will certainly strengthen the paper.

[1] Mixtures of experts unlock parameter scaling for deep rl, Obando-Ceron et al. 2024.

[2] In value-based deep reinforcement learning, a pruned network is a good network, Obando-Ceron et al, 2024.

We sincerely thank you for your supportive review. Your appreciation for the writing quality, insightful components (e.g., max pooling's overlooked role), thorough ablations, and performance gains is encouraging. We will respond to your points made.

• We agree with your point of looking beyond performance analysis for certain components. As a result, we have done additional experiments analyzing two feature metrics for the isolated Hadamard and Maxpooling components. In addition to the effective rank, we have now also analyzed convolutional channel cosine similarity, as done in [1]. A low cosine similarity indicates that the channels are extracting dissimilar or uncorrelated features from the input data, which is desirable and suggests diversity among the channels. The results are interesting. Our paper already shows increased effective rank (Fig. 8) in the deeper convolutional feature space (layer 2 & 3) as compared to the baseline. However, investigating both Hadamard and Maxpooling separately shows that Maxpooling increases effective rank as compared to the baseline, but also slightly increases channel cosine similarity. Meanwhile, the Hadamard component also slightly increases effective rank but massively decreases channel cosine similarity by up to 90% as compared to the baseline. This gives further insights into why the combination of Maxpooling and Hadamard could work so well, as Hadamard strongly compensates against the increase in channel cosine similarity caused by Maxpooling. We will add these results to the main paper, along with a detailed explanation. A table summarizing the results is seen below:

• We have analyzed the forward pass computation time for both Hadamax encoder, baseline encoder, and the Rainbow encoder. The inference times are as follows:

This shows that due to setting the convolutional stride to 1, and adding maxpooling elements as well as Hadamard interactions, the inference time of the encoder will increase. Hadamax inference time is slightly higher than Impala inference time. We will be sure to add these inference times to the main paper in the camera-ready version.

• We are grateful for the reviewer acknowledging the scope of the Atari-57 benchmark. We have however ran an additional evaluation on a PPO baseline in the pixel-based Vizdoom environment: Deathmatch. This encoder was similar to the Atari encoder, which allows us to directly implement the Hadamax encoder here. The results are as follows:

The results show a significant increase over the baseline encoder, and will be added to the camera-ready version to improve the environment diversity.

• We have not run any tests in standard computer vision benchmarks. We do think that the Hadamax architecture, or maybe a revised, more complex version thereof, can also improve performance in standard computer vision benchmarks. However, this is always hard to predict, because if we look at the other way around, the newest computer vision encoders generally do not transfer well to pixel-based RL. This is a part of the reason that a new state-of-the-art algorithm such as PQN still uses the 'nature' DQN encoder from 2015.

Thank you again for your insights. We will implement the more detailed convolutional feature analysis, the VizDoom results and the inference times in our manuscript, in order to strengthen the camera-ready version of the paper.

[1] Manifold Regularized Dynamic Network Pruning, Tang et al, 2021.

Thank you for your detailed review and for highlighting the strengths: the simplicity, reproducibility, clarity, and impressive performance from architectural changes. We would like to respond to the weaknesses and questions raised.

• We understand that the ALE 200M benchmark might seem like a limited evaluation. However, we would like to kindly reiterate that we have performed experiments on all of the 57 games for 200M frames in the Atari suite, similar to the original DQN, DDQN, C51, Rainbow papers etc. This still remains the largest, most diverse and challenging testbed in visual (discrete) RL. However, in order to increase environment diversity and take your concerns seriously, we have ran additional preliminary experiments on another pixel-based environment “VizDoom: Deathmatch”, using the actor-critic PPO algorithm:

Also here, we show a remarkably strong increase over the baseline by replacing the encoder. We will add the results to our updated manuscript, extending our testbed beyond Atari.

• The Reviewer has a point in that there is no thorough technical analysis or proof as a precursor to the encoder design. The Hadamax design has been the product of a generalist look on all of the recent work in non-algorithmic RL, in order to strongly improve the widely used, older ‘nature’ encoder that has been used in pixel-based RL since 2015. To the best of our knowledge, the only alternative encoder used in literature was the Impala encoder. We therefore believe that we bring novelty in the way that encoders are structured, also showing that deeper encoders such as Impala are not necessarily needed for the state-of-the-art performance gains that Hadamax can bring. Adding to this, we have now done additional experiments further analyzing the effect of the isolated Hadamard and the Maxpooling operators on the convolutional feature space. Next to the effective rank already discussed in the paper, we have analyzed the feature space channel cosine similarity, as done in [1]. A low cosine similarity will indicate that convolutional channels are extracting uncorrelated features from the input data, which is positive shows a feature diversity. Maxpooling has shown to increase effective rank but also slightly increases the channel cosine similarity. Meanwhile, the addition of the Hadamard component shows a strong decrease in channel cosine similarity by up to 90%. This gives further insights into why combining Maxpool and Hadamard works so well, as the Hadamard interaction compensates against the increase in channel cosine similarity caused by Maxpooling. We will make sure to add these results to the main paper, strengthening our reasoning behind the proposed architecture. A table summarizing the results is seen below:

• In our paper (fig. 7) we show that Hadamax is also effective on the distributional C51 architecture. Next to this, we have now shown Hadamax to also be working on the PPO actor-critic architecture on VizDoom. We therefore believe that this approach can be applied to different domains and model architectures, as long as there is convolutional downsampling involved. Besides this, we believe that the practical effectiveness of the Hadamax encoder might also allow researchers working in e.g. Image Recognition to try and implement these architectural changes in their research on encoder synthesis.

Nonetheless, we thank you for the review on our work. We would however kindly ask to reconsider the given score, as we have now added strong results on an additional hard pixel-based environment (Vizdoom) in the paper, as well as more insights into the effect of Maxpooling and Hadamard on the convolutional features. Furthermore, we would like to emphasize that we believe we have made a very strong academic contribution in encoder synthesis in pixel-based RL. Most recent algorithms are still employing the ‘nature’ DQN encoder from 2015- we believe this indicates that successful encoder synthesis for Reinforcement Learning is very hard, due to the unstable nature of optimization as compared to Supervised Learning. To the best of our knowledge, Hadamax-PQN is now a state-of-the-art RL algorithm on Atari when looking at wall-clock time and performance.

[1] Manifold Regularized Dynamic Network Pruning, Tang et al, 2021.

Thank you for your positive review—we really appreciate your high rating and enthusiasm for the simplicity, reproducibility, and strong empirical performance of Hadamax. We'll address your stated weaknesses and questions below:

Weaknesses:

• We agree with the reviewer that implementations across different image-based domains would present more opportunities in terms of the academic contribution and transfer of knowledge. However, as much as we would like to have included, for example, the full pixel-based MuJoCo domain, the compute requirements are a limiting factor. The full 57 game Atari suite for 200M frames (as is used in the original DQN, DDQN, Rainbow, C51 papers) is already a very demanding benchmark.

Having said this, we do see that environment diversity is a recurring theme amongst the reviewers and wanted to address some of these concerns. We have therefore conducted preliminary experiments using the actor-critic based PPO, on a pixel-based VizDoom (Deathmatch) environment for 40M frames:

Note that, as in the PQN framework, we only substituted the original encoder with a Hadamax encoder. Although Vizdoom is usually trained for longer, there is already a clear performance boost over the baseline encoder, and we will add the results, as well as the code, in the camera-ready version.

• As for your comments on the poster/presentation, if our work gets accepted, we will be sure to try and include as many use-cases as possible.

• About the emphasis on vectorized RL: We see your point and apologize if this muddied the waters. We'll slightly revise the structure of the manuscript for a clearer storyline in the camera-ready version.

• We have already included the precise code of the encoder in Appendix B. Or do you mean that readers could have benefited from it in a more visual way? If it’s the latter, we are more than willing to create a more in-depth diagram for the camera-ready version.

Questions:

• The biggest limitation of Hadamax is that it increases the parameter size of the encoder and adds additional training time, in comparison to a less complex baseline encoder. Furthermore, the Hadamax encoder, in this form, has been designed for pixel-based environments only where a CNN feature extractor is followed by a bottleneck (Linear layers).
• We think that, because pixel-based inputs (especially in Atari) are high-dimensional, the encoder synthesis differs from that of non-pixel domains. We assume Hadamax works so well here because it improves the feature extraction process. In non-pixel-based domains, the hidden layers are commonly larger than the original input spaces, so in that case we see the MLP encoder as more of an ‘optimization vessel’ rather than a true (compressing) encoder. From what we’ve seen, these MLP encoders on state-based inputs generally have different design requirements.

We thank you once again for your detailed review, and praise of our work. We will implement your suggestions as well as the VizDoom results in the main paper, in order to strengthen the camera-ready version.