Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning

We thank the reviewer for their feedback! We are glad that the reviewer found the method simple and effective and that the empirical investigation is thorough and strong.

Q1: It is currently unclear how 'dormant' neurons are determined.

A: We adopt the definition from the original paper that identifies dormant neurons in deep reinforcement learning [1]. The dormancy is measured based on a neuron's activation threshold. We will add the precise formulation to the revised manuscript.

Q2: You mention following the work of others and aiming for 90% sparsity. The effect of increased sparsity on model changes and performance is a crucial aspect that needs further elaboration. Can you provide more details on how sparsity was implemented (e.g., pruning method, frequency). Elaborate on the observed effects of varying sparsity levels (even if only 90% was the target) on both the model's architecture (e.g., changes in connectivity, effective parameters) and its performance metrics.

A: In the case of static sparsity, a random network topology with the specified sparsity level is initialized at the beginning of training and kept fixed throughout. In the case of pruning, we followed the same strategy of prior work that studied pruning in RL [2,3]. The network starts with a dense network and low magnitude parameters are gradually pruned throughout training according to a polynomial schedule. We will provide more experimental details of these analyses in the revised version.

Following the reviewer’s suggestion, we have conducted extra experiments varying the sparsity levels. We report the IQM with 95% confidence interval at 40M steps aggregated over the 20 studied games. Consistent with our previous findings, sparsifying only $\psi$ leads to higher or similar performance to sparsifying the whole network.

Q3: The scaling by 4 factor seems arbitrary without further context, though there is likely a good reason for it. Is it related to a specific architectural choice, computational constraint, or a baseline from previous work?

A: Our choice of a scaling factor of 4 for most of the analysis is based on the observation found in prior works that 4x provides substantial performance improvements, whereas further scaling offers only marginal benefits [2,4]. Nevertheless we provide our main results for four different scales (Figure 6). We will extend our discussion in Section 3.2 on the justification for this choice.

Q4: Figure 10 & Performance Claims. You conclude that "the performance of the 2 scaled models is nearly identical." However, observation of the IQRs (presumed to be the shaded regions) shows non-overlapping distributions across many frame numbers. If the scaling of this figure were different, these deviations might appear more significant. While the overall conclusion regarding the bottleneck might be correct, it's important to acknowledge and comment on these visible deviations in the IQRs. If the differences are not considered significant, please explain why.

A: We agree with the reviewer that the use of “nearly identical” is not precise. We will revise the text to state that the methods achieve 'comparable performance' for greater clarity.

Q5: Overlapping distributions for GAP and SoftMoE are shown [boxplots]. It's questionable whether a notched boxplot or a Wilcoxon rank-sum test would show statistically significant differences in their medians. Given this visual overlap, can you provide a more robust argument for why GAP is considered "higher performing than all other methods." If statistical tests were performed, please give the results. If not, consider performing them or refining the claim.

A: We will clarify that with respect to SoftMoE, GAP achieves similar performance but is both more efficient and much simpler.

[1] Sokar, Ghada, et al. "The dormant neuron phenomenon in deep reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

[2] Ceron, Johan Samir Obando, Aaron Courville, and Pablo Samuel Castro. "In value-based deep reinforcement learning, a pruned network is a good network." International Conference on Machine Learning. PMLR, 2024.

[3] Graesser, Laura, et al. "The state of sparse training in deep reinforcement learning." International Conference on Machine Learning. PMLR, 2022.

[4] Ceron, Johan Samir Obando, et al. "Mixtures of Experts Unlock Parameter Scaling for Deep RL." International Conference on Machine Learning. PMLR, 2024.

We thank the reviewer for their feedback! We are glad that the reviewer found our empirical validation to be extensive and the approach highly effective, computationally efficient, implementation-friendly, and requiring minimal architectural changes. Below, we provide detailed responses to each comment, incorporating the results from all requested experiments.

W1: why the absence of a well-structured bottleneck—and the use of flattened encoder outputs—leads to degraded plasticity or impaired learning dynamics…. a more principled explanation would substantially strengthen the work’s impact and conceptual clarity.

A: The flattening operation transforms the encoder's output into a large 1D vector, potentially losing some of the spatial structure captured by the encoder. This flattened vector, combined with the scaled width of the next layer, creates a high-density connection, which makes it difficult to make effective use of the features' spatial relationships, and may result in overfitting which often causes a loss of plasticity [1]. These effects can result in the model attending to unimportant features and regions in the input, as demonstrated in Figure 3. To understand this further, we demonstrated in Figure 4 that preserving the spatial structure by learning higher level features before forwarding them to the dense layers mitigates this issue.

W2, Q4: “Despite broad benchmarking, the experimental domains remain confined to pixel-based benchmarks with discrete action spaces (e.g., ALE and Procgen), with no evaluation on continuous control domains (e.g., DeepMind Control Suite, Adroit), limiting the scope of generalizability.” “how do the authors view its applicability in environments that require fine-grained spatial reasoning or partial observability?”

A: Following the reviewer’s suggestion, we expand our evaluation to the continuous control domain. We evaluate SAC on DeepMind Control tasks. We ran each experiment with 5 seeds. We follow the CNN architecture presented in [1], which consists of four convolutional layers, followed by critic and actor embeddings of a size of 50 neurons. We scale the critic and actor embedding by 8x and 16x. GAP consistently yields significant performance gains over its corresponding baseline.

We thank the reviewer for pressing us on this point, as the addition of these extra results help reinforce the generality of our findings.

W3: The absence of parameter count ….it remains unclear whether the observed performance gains result from genuine improvements in scaling capacity or merely from more efficient parameter utilization. Clarifying this distinction—and including performance comparisons at matched parameter counts—would offer a more rigorous understanding of whether GAP enables true architectural scaling or simply more effective use of fewer parameters.

A: We reported the parameter count in Figure 5, right. Note that the parameter count of GAP matches the parameter count of SoftMoE-1 (Please see Figure 1 for illustration and Section 2 for more details). We also tested a baseline with this same parameter count. As demonstrated in Figure 5, parameter-matched baseline performs worse than other methods, confirming the performance improvements are attributable to our architectural design, not just parameter count.

Q1: Have the authors evaluated the effectiveness of GAP in data-augmented actor-critic settings such as DrQ or DrQ-v2?

A: Following the reviewer's suggestion, we ran additional experiments using the DrQ (top) and DrQ-v2 (bottom) algorithms on the Atari100K benchmark with a 4x scaled network using 5 seeds. Consistent with previous findings, the results show that GAP provides performance gains in both cases.

We once again thank the reviewer for suggesting this extra set of experiments.

Q2: Figure 4 suggests that scaling both $\phi$ and $\psi$ improves performance, but scaling only $\psi$ hurts performance. Could the authors include results for a $\phi$-only scaling variant to further substantiate the attribution of degradation of only scaling $\psi$? This ablation would help isolate the relative contributions of encoder and head scaling and offer stronger support for the claim that the performance degradation resides primarily in $\psi$

A: As suggested by the reviewer, we conducted an experiment with a deeper $\phi$ while maintaining the original, unscaled size for the $\psi$. We report the average return at 100M steps. As shown below, providing high-level features to $\psi$ enhances performance in both its unscaled and scaled versions, with the most significant gains observed in the scaled version.

Q3: The paper focuses heavily on scaling the fully connected head , while prior work in computer vision typically emphasizes scaling the encoder . Could the authors elaborate on the motivation behind this choice? While it is understood that, in typical vision-based RL architectures, contains the majority of the parameters, it remains unclear why encoder scaling—which is the standard approach in computer vision tasks—is deprioritized. From a representational learning perspective, enhancing the encoder capacity would seem more intuitively aligned with performance improvements.

A: Our findings in Figure 2, right, demonstrate that scaling the dense layers alone leads to a performance decline comparable to that of scaling both the encoder and dense layers simultaneously. This observation, along with the success of recent methods like SoftMoE—which achieved optimal performance by applying mixtures to the dense layers—motivates our focus in this paper on scaling $\psi$. Nevertheless, we conducted an additional experiment scaling only the width of the encoder layers by 4x. Due to the high computational costs and the limited rebuttal period, we report the IQM with 95% confidence interval over 20 games at 40M steps. Consistent with our previous findings, scaling the encoder decreases the performance of the model, whereas incorporating GAP into the network leads to a significant improvement.

[1] Nikishin, Evgenii, et al. "The primacy bias in deep reinforcement learning." International conference on machine learning. PMLR, 2022.

[2] Yarats, Denis, et al. "Improving sample efficiency in model-free reinforcement learning from images." Proceedings of the aaai conference on artificial intelligence. Vol. 35. No. 12. 2021.

We thank the reviewer for their feedback! We are glad that the reviewer found our experiments comprehensive and that the method is both effective and readily adaptable.

W1: The width-depth scaling claims is unsupported. The authors claim that GAP unlocks width and depth scaling, but this is never supported with an experiment. I except to see two figures both having performance on the y-axis vs width and depth on the x-axis. The current experiment only shows that the performance using GAP is better than the baseline in wide, deep, and wide&deep variations. No claims about scaling can be made using this experiment. If the authors want to keep the claim, then they need to support it with the proper evidence.

A: We have conducted extra experiments with varying depth on Rainbow using the ResNet architecture. Due to the high computational costs and the limited rebuttal period, we report the IQM with 95% confidence score over 20 games at 40M steps. As demonstrated below GAP consistently significantly improves performance of deeper networks. Results across varied width are presented in Figure 6.

W2: The authors use the word scaling in a different way from its common use which makes things a bit complicated. The argument of the scaling is almost always missing each sentence. For example, one can argue a method X allows (performance) scaling with the width of the encoder layers then show a figure with the width on the x-axis and performance on the x-axis. This supports the claim clearly. On the other hand, the paper use the word scaling loosely which makes it lose its meaning altogether. The only place where such experiment is presented in Figure 6 (first two) where the performance barely scales with width and doesn't scale. with replay ratio.

A: In most of our experiments we focus on scaling the width of the dense layers, following previous work [1,2]. In the leftmost plot of Figure 6, GAP consistently outperforms the corresponding scaled baseline. Moreover, the scaled network with GAP outperforms the unscaled baseline. In the second from left plot, a constant scale of 4 is used across varied replay ratios in the x-axis; we show that GAP helps enable obtaining sample efficient agents, as shown by the improved performance over the baseline with the same replay ratio value. We will add a precise definition of scaling and further clarify these results in the revised version of the paper.

W3: The paper lacks providing any reason for this bottleneck phenomenon. It remains unclear why the bottleneck can severely affect performance. I think the paper should at least give an explanation or even a hypothesis for why this is the case.

A: We believe that the flattening operation contributes to forming this bottleneck. It transforms the encoder's output into a large 1D vector which, combined with the scaled width of the next layer, creates a high-density connection. This high density makes it difficult to make effective use of the features' spatial relationships, and may result in overfitting which often causes a loss of plasticity [3]. These effects can result in the model attending to unimportant features and regions in the input, as demonstrated in Figure 3. To understand this further, we show in Figure 4 that preserving the spatial structure by learning higher level features before forwarding them to the dense layers mitigates this issue. We will extend our discussion on this point in the revised version.

Q1: n Figure 6, why do we see a performance drop when we scale the width by a multiple of 2 and increase with a multiple of 4?

A: GAP x2 outperforms or matches GAP x1 in 19 out of 20 games. The slight reduction in the aggregated score is due to a single outlier game, breakout, where we observed high variance across our five runs. We will include detailed per-game results in the revised manuscript to make this clear.

I think the paper provides a good contribution, but I think the authors should work on their claims to avoid overstating the results. I think the paper addresses an important problem and there is an audience for it. I highly encourage the authors to improve their writing and moderate their claims to reflect what is given in the experiments.

A: We thank the reviewer for their constructive suggestion and their acknowledgment of the problem's importance and the potential interest our work holds for the community. We agree that our use of the term scaling could be clearer. In the revised manuscript, we will explicitly define it, and incorporate the new experiments presented in this response that further confirm our findings.

Minor rephrasing

Thank you for providing us with suggestions for rewording, we will incorporate that in the revised manuscript.

[1] Ceron, Johan Samir Obando, et al. "Mixtures of Experts Unlock Parameter Scaling for Deep RL." International Conference on Machine Learning. PMLR, 2024.

[2] Sokar, Ghada, et al. "Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL." The Thirteenth International Conference on Learning Representations.

[3] Nikishin, Evgenii, et al. "The primacy bias in deep reinforcement learning." International conference on machine learning. PMLR, 2022.

W1 and W2: I thank the authors for providing the depth-scaling experiment. This increases my confidence in the claims in the paper. However, one remaining concern is the wording. As you can see in the table provided, the performance of GAP from Deeper-2 to Deeper-3 actually decreases, so it still cannot allow for performance scaling with depth or width (e.g., performance drops when increasing depth or width). In the paper, you should clearly mention this limitation and always say "better scaling than baseline" instead of just "better scaling".

I thank the authors for their rebuttal. I think my concerns are addressed, given that the authors promised to make the necessary changes in the revised version. I will change my rating to reflect that.

We thank the reviewer for their feedback, and we appreciate their recognition of the method's strong performance, simplicity, and impact. Below we provide responses to all comments.

W1: The main hypothesis of the paper says low-dimensional and well structured bottleneck allows scaling. Does that mean that a 2x2xd bottleneck doesn't allow scaling? Also, what does well-structured really mean?

A: What we mean is organizing the encoder's output in a way that facilitates feature learning in the dense layers; maintaining the features' spatial relationships and learning more generalizable representation. Methods such as sparsity, tokenization, or GAP create a more effective structure than simply flattening the output and densely connecting all neurons. Note that it is not just a matter of parameter count. As we demonstrate in Figure 5 (right), an equally-parameterized baseline still underperforms.

W2: The paper is technically void. There's not a single new or interesting equation (let alone theorem) in the paper.

A: While our work doesn't introduce new equations given its empirical nature, we respectfully disagree that this implies a lack of technical depth. The paper provides rigorous analysis, a novel and effective architectural design, and extensive validation of its efficiency across diverse settings and metrics. Further, we demonstrate that one can reduce the complexity of prior approaches (such as MoEs and tokenization) without sacrificing accuracy. We strongly believe this type of evidence-backed insight is a valid contribution to the field, and particularly in light of its simplicity.

W3: The point of the paper could have been conveyed in one page.

A: The paper's value extends beyond our final finding of the architectural modification that boosts the agent’s performance. Indeed, a core part of our work is the extensive set of analyses which provide the evidence for our final claim. Without these, the paper would reduce to an anecdotal observation, rather than useful insights backed by data and analyses.

Q1: Fig. 2 (Right) which game is this?

A: It is aggregated results over the same set of 20 games used in all our experiments. We will clarify this in the caption.