We deeply appreciate your thorough and insightful review, particularly your recognition of the paper's novelty, practicality, and rigorous validation. Thank you very much for finding our work extremely valuable and novel while the effectiveness of our method is validated across multiple scales from theory to high-dimensional large-scale experiments. Your feedback provides clear and sincere insights on how our work offers a significant, practical, and theoretically sound advancement for the deep RL community.

1. "To clarify how well the theory holds experimentally, could you plot how long $D(s) - 2 \delta -\eta >0$ continues from the early stages of learning?"

Regarding how well the theory holds experimentally, Figure 3 in Section 5 reports temporal difference throughout the entire training, i.e. both early-stage training and late stage training. The results reported in Figure 3 demonstrate that CoAct TD learning achieves higher temporal difference throughout training compared to canonical methods, which has been predicted by the theoretical results in Theorem 3.4 and Theorem 3.6.

2. "Looking at Figure 2, the learning curve for $\epsilon$-greedy appears not to have converged. When iterations are increased further, how do the final convergence scores compare between $\epsilon$-greedy and CoAct TD Learning? While I understand that the proposed method accelerates convergence in early learning, I believe the convergence values are also important in practice."

The cumulative return reached by CoAct TD at iteration 84 is the optimal value for the Chain MDP. At your suggestion we ran the experiment for longer and found that $\epsilon$-greedy converged to this same optimal value at iteration 176.

3. "As an application area for reinforcement learning, LLM post-training is gaining attention. Post-training starts from a state where the LLM has completed pre-training, but would this method still be effective for improving learning efficiency in such cases? If effective, I believe this would further enhance the value of this research."

Thank you very much for this intriguing and insightful question. Yes, indeed. Because the pre-training of an LLM does not involve the similar structure of reinforcement learning as the post-training. Currently, reinforcement learning is used either to enhance the mathematical and scientific reasoning capabilities of LLMs or in alignment as in RLHF. In both of these cases, post-training the large language model is the start of the reinforcement learning training in a newly defined task, and hence yes indeed we can expect CoAct TD to be effective in improving learning efficiency in both of these cases.

We sincerely thank you for your insightful and in-depth review. We are truly glad you found our work impactful and straightforward, and that you noted our paper is well-written and easy to follow. We are especially glad to hear that you consider our core idea both novel and intriguing, with the potential for significant impact on the field.

1. “(I) Assumptions: Line 128-130: …It is unclear whether this remains true as training progresses. Is there empirical or theoretical evidence to support this claim beyond the initial phase? (II)Line 131-133: The statement that "In particular, in high-dimensional environments in a significant portion ... has little correlation with the immediate rewards." also needs supporting evidence or citation. (III) Theorem 3.4 depends on assumptions that " $Q_\theta(s,a)$ is $\eta$-uninformed and $\delta$-smooth." Can you provide proof or empirical evidence that these assumptions hold, especially beyond the very start of training? It appears these conditions may only be valid at initialization and not throughout training.”

All these questions consolidate the Definitions introduced in Section 3 and its relation to Theorem 3.4 and Theorem 3.6. Yes, there is indeed evidence to support this. In particular, Figure 3 in Section 5 reports the temporal difference throughout the entire training, i.e. both early-stage training and late-stage training. The results reported in Figure 3 demonstrate that CoAct TD learning achieves higher temporal difference throughout training compared to canonical methods as has been predicted by the theoretical results in Theorem 3.4 and Theorem 3.6.

2. ”Experimental results”

We highly appreciate your insightful suggestions regarding in which places the content and the Figures should appear and the organizational presentation of the experimental results to provide a better reading experience. These suggestions are now indeed reflected for the camera ready version of our paper.

3. ”Additional Baseline: It would strengthen the empirical evaluation to include a comparison with a purely greedy baseline, where $\epsilon$ -greedy is not used (i.e., the agent always selects the action with the maximum Q-value).”

At your suggestion we ran the experiment with a purely greedy baseline. This algorithm completely fails to learn, never achieving a reward above 0.2. In contrast as shown in Figure 2, $\epsilon$-greedy achieves reward 8 after 100 iterations, and CoAct TD achieves reward 11. We can also add this purely greedy baseline to the results as well.

We are truly glad to hear that you found our work novel and easy to use, and that our analysis offers a detailed overview of the agent's performance. We are also sincerely glad you highlighted that our paper provides both theoretical analysis and empirical evaluation, supporting the method's intuition and effectiveness. We highly appreciate your positive and valuable feedback.

1. "It would be great if the authors could explain the intuition of learning for actions with the lowest value estimation. What the agent needs to learn in the end is the action with the highest value, for deriving a good policy. Explainning the intuition more might be helpful for readers to understand the methodology."

This insightful question is highly related to understanding how higher temporal difference results in effective learning, and the core counterintuition section of our paper that starts in Line 173. In particular, our theoretical analysis shows that taking the lowest value action in fact yields a higher temporal difference compared to canonical, and temporal difference update is essentially the update that leads to learning an optimal policy, i.e. the action with the highest value. Hence, our method shows that by taking action $a_{\textrm{min}}$ in fact we can increase temporal difference, and hence accelerate learning the highest value action.

2. ”The explanation remains unclear regarding how a larger TD error contributes to more effective learning.”

Thank you for this insightful question. It is well known in the reinforcement learning literature that larger temporal difference will result in effective learning [1]. Although we also cite this work [1] in page 3 we can also further add more explanation regarding larger TD and effective learning.

[1] Andrew W Moore and Christopher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning, 1993.

3. "Could the authors please clarify more about how the algorithm is evaluated? In empirical evaluation, it is not clear whether the reported performance reflects the agent’s performance during online interactions or is based on a a fixed policy rollout, i.e., whether the agent still follows $a^{\textrm{min}}\_{Q\_\theta} (s)$ as the left side of Algorithm 1 indicates. Given that online actions affect the environment, I think it is important to specify this point. If following $a^{\textrm{min}}\_{Q\_\theta} (s)$ is part of the learning process, then reporting the return during interactions would be more informative than only evaluating the final policy, as it captures the consequence of taking actions that are considered to be poor choices."

The reported performance reflects the agent’s performance when $a^{\textrm{min}}\_{Q\_\theta} (s)$ is taken.

4. "The application of the method may be limited by the properties or constraints of the environment. Taking $a^{\mathrm{min}}\_{Q\_\theta} (s)$ may not always be applicable in practice. In online settings, every chosen action affects the environment. Since the action value estimation reflects how good an action is, taking the action with a low estimation introduces concerns about whether it causes irreversible damage to the environment, especially in real-world applications like robotics and autonomous driving. In such domains, collecting data following $a^{\textrm{min}}\_{Q\_\theta} (s)$ could be unacceptable due to safety or cost concerns."

We would like to emphasize that this question is again highly related to the core counterintuition part of our paper starting in line 173. The core counterintuition introduced in our paper is the realization of the critical difference between $a^{\textrm{min}}\_{Q\_\theta} (s)$ and $a^{\textrm{min}}\_{Q^\*} (s)$. In particular, from the perspective of $Q^\*$ in fact the $a^{\textrm{min}}\_{Q\_\theta} (s)$ is not the true minimum of the optimal state-action value function $a^{\textrm{min}}\_{Q\_\theta} (s) \neq \textrm{argmin}\_{a \in A} Q^\*(s,a)$. Thus, the damage that can be done to the environment is relevant to the true rewards obtained by the agent and hence $Q^*$ but not $Q_\theta$. This is indeed very subtle and requires a bit of counterintuitive thinking. We can also provide more textual explanation on this between Line 173 and 187.

Thank you very much for finding the core idea of our paper novel and well-written while addressing a critical problem with a strong logical structure that flows from theory to practice with rigorous validation of the theory in both motivating examples and large-scale standard benchmarks. We highly appreciate your valuable and insightful review.

1. "The theory for CoAct TD relies on the Q-function being uninformed, a condition primarily met during early training. What is the method's behavior in late-stage training when this assumption no longer holds? A brief analysis demonstrating that the agent does not get stuck in known undesirable states would increase my confidence in the method's robustness."

Thank you for this thoughtful question. Figure 3 reports temporal difference throughout the entire training, i.e. both early-stage training and late stage training. The results reported in Figure 3 demonstrates that CoAct TD learning achieves higher temporal difference throughout training compared to canonical methods as has been predicted by the theoretical results in Theorem 3.4 and Theorem 3.6. As we investigate the results reported in Figure 3 together with the results in Figure 1 and Figure 5 we see that the agents throughout the entire benchmark robustly overperform compared to the canonical methods.

2. "The paper's claim of being a "fundamental improvement" is broad, yet experiments are confined to DQN-family algorithms. Could the authors briefly discuss how the argmin Q principle could be extended to continuous control algorithms (e.g., SAC, TD3)?"

Thank you for this insightful question. The policy update in SAC is performed via taking a gradient step with respect to the loss $J(\phi) = D_{KL}(\pi_{\phi}(\cdot | s_t) || \exp(Q_{\theta}(s_t,\cdot)/Z_{\theta}(s_t))$. Here we can take the gradient of the cost function $\hat{J}(\phi)$ with $-Q_{\theta}$ to obtain $\pi_{\phi’}$ with probability $\epsilon$ to obtain the analogue of the argmin $Q$ principle.

3. "The paper links higher TD error to better exploration. Can the authors provide any direct evidence of improved exploration quality, such as higher state visitation entropy compared to baselines?"

Many thanks for this intriguing question. The rather well-known link between higher temporal difference and better exploration was investigated in quite early work starting from [1], and the main connecting evidence between higher TD error, exploration quality and faster learning in our paper is the illustration of the significantly improved performance of our method in the ALE benchmark.

[1] Andrew W Moore and Christopher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning, 1993.

We highly appreciate again that our efforts to produce a rigorous analysis were recognized by you.