Thank you for your thorough review and constructive feedback on our paper. To address your questions and concerns, we have provided detailed responses below.

Question 1

PPO practitioners have found much larger batch sizes of 100k samples to work much better than smaller batch sizes

=> Thank you for pointing out this work that uses very large batch sizes. However, the effectiveness of batch size remains somewhat controversial—likely depending on the environment—as other studies [1, 2] argue that smaller batch sizes can be more beneficial. The success of the work you mentioned may be due to its specific setting, where over 10,000 agents can generate a massive number of data points. In general, we believe that, given typical computational and cost constraints, updating an LLM with a batch size of 100k is often not feasible in standard setups.

Question 2

Is the improvement from simply ignoring reward-model mistakes or from genuinely mitigating advantage confusion?

=> Rather than focusing on reducing the error rate of a trained reward model (which is certainly one way to improve performance), our method directly targets the mitigation of advantage confusion. Even when there is no reward model error, advantage confusion can still arise due to techniques like advantage normalization (Figure 2). We observe that SPPO still shows improvements in the noise-free setting (Section 5.4).

Question 3

Comparing the effect of bigger vs. smaller batch sizes on advantage confusion

=> Thank you for suggesting this interesting experiment. If the paper is accepted, we will include results evaluating our method across a range of batch sizes. However, we believe the effect is quite task-dependent. Based on our current understanding, Atari games tend to perform better with smaller batch sizes [1, 2], while MuJoCo tasks may benefit from larger batch sizes [3].

Weakness 1

The method adds extra hyperparameters $(\alpha, \beta, Z)$, which can be a drawback for ease of adoption

=> While the symmetric RL loss introduces three hyperparameters, $\beta$ and $Z$ can be treated as a single term. To demonstrate ease of adoption, we conducted a hyperparameter sensitivity analysis for $\beta$ while keeping $\alpha$ and $Z$ fixed—particularly in the PPO setting (see Table 11 in the Appendix). We observed consistent performance improvements across a wide range of $\beta$ values, suggesting that the method is easy to apply in practice.

Weakness 2

While the paper shows that advantage sign flips frequently occur, the discussion is largely heuristic rather than formal.

=> We provide empirical evidence of sign flip ratios in Table 2, evaluated across multiple random seeds for a diverse set of environments (5 for Atari games and 30 for the MuJoCo benchmark). We believe that advantage sign changes are obviously expected after advantage normalization, but we will include results from additional environments to further support this observation.

Weakness 3

For the RLHF tasks, all evaluation rests on an imperfect reward model and GPT-4-based comparison (no direct human eval). Though typical in modern LLM research, more thorough human-based or multi-metric evaluations could be enlightening.

=> We used GPT-4 as the evaluator and measured win rates based on two comparison examples, following the AlpacaEval [4]. Our experiments involve relatively smaller models and simpler tasks, where GPT-4’s evaluations have been shown to align closely with human judgments.

Thank you for suggesting REDQ, DroQ and CrossQ. We will make sure to cite them in the our paper.

[1] The Phenomenon of Policy Churn, Neurips 2022
[2] Small batch deep reinforcement learning, NeurIPS 2024
[3] Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation, ICLR 2022
[4] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators, COLM 2024

Thank you for your thorough review and valuable feedback on our paper. We hope the response below addresses your concerns.

Weakness 1

Insufficient Comparison with Other Robustness Methods: While the paper focuses on the symmetric RL loss, it lacks detailed comparisons with other established methods for robust RL, such as Double Q-learning or SAC, which could provide a clearer context for its benefits.

=> Rather than expanding to additional types of reinforcement learning algorithms, we focused on those whose working mechanisms align with the symmetric loss formulation in noisy classification settings. Instead, we aimed to demonstrate the effectiveness of the symmetric RL loss across a diverse range of problem domains—including discrete action spaces, continuous action spaces, and NLP tasks—highlighting its applicability even to large language models.

Weakness 2

Limited Theoretical Justification: The paper would benefit from a more detailed theoretical explanation of why the symmetric RL loss is particularly effective in addressing the noise in reward prediction, especially when compared to existing techniques like Generalized Advantage Estimation (GAE).

=> In all our experiments, we used Generalized Advantage Estimation (GAE), and we will explicitly clarify this in the experimental section. Since GAE depends on the reward, noise in the reward can lead to confusion in advantage estimation. Our symmetric RL loss acts as an effective accelerator to mitigate this issue. We provide a detailed analysis of how this acceleration benefits the learning process in Section 4.3 and Appendix A.1 to B.2.

Weakness 3

Hyperparameter Sensitivity: The sensitivity of the performance to hyperparameters such as $\alpha$ and $\beta$ is not sufficiently explored.

=> We provide additional hyperparameter sensitivity analysis in Table 11 of the Appendix for SPPO, both with and without reward noise, showing consistent improvements across a range of values. Note that $Z$ and $\beta$ can be treated as a single hyperparameter (see the first paragraph of Section 4.1 for details).

Weakness 4

Scalability and Computational Cost: More extensive analysis is needed to understand how these hyperparameters affect training and performance stability.

=> The symmetric RL loss is scalable, even for large language models. When selecting the next action (token) via softmax, the model also produces probabilities for non-selected tokens. Computing gradients with respect to these non-selected probabilities introduces some additional overhead, but this cost is roughly equivalent to adding a single MLP layer (we will include this explanation in the paper). Given the overall size and structure of LLMs, this additional cost does not significantly affect training speed. In fact, SPPO showed faster training than PPO for some seeds—i.e., within the overhead of adding one MLP, GPU memory allocation conditions had a greater effect. We briefly address training speed in the last paragraph of Section 5.5.

Thank you for your thorough review and valuable feedback on our paper. We try to handle your questions and provide additional details below.

Please let us know if our responses resolve your questions and concern. If so, we would greatly appreciate your consideration in updating your score. We’re also happy to continue the discussion if you have further questions. Our answers are as follows:

Question 1

I don't understand why adding the reverse advantage loss could help relieve the problem of estimation error resulting from noisy human feedback and multiple sources of scaled models.

=> In the Gradient Analysis section, we describe how the reverse RL loss helps the learning procedure in the presence of noise, acting as an accelerator. We believe you would agree that when noise is present, the policy lacks certainty about which action to take. Since the reverse RL loss gradient has its maximum magnitude at 50% (a parabolic function) and its direction aligns with the original RL loss, it helps the policy move away from this ambiguous state. Please refer to further details in Section 4.2 and Sections A.1 to A.4 in the Appendix. We provide a comprehensive derivation and analysis for all cases.

Question2

Why bother to use symmetric CE loss other than directly train on large amounts of data sets of diverse tasks?

=> Increasing the data amount is one potential solution, but Reinforcement Learning (RL), especially with large models, has burdensome such as generating actions and getting the corresponding rewards. For the rewards, we would need a highly engineered reward function, human, or AI evaluators (which can also have noise). Therefore, achieving better performance using the same amount of data is obviously advantageous. The symmetric RL loss (or symmetric CE loss) facilitates this and is easy to implement. Since action (or token) sampling is done through the softmax, the other probabilities can be obtained naturally for symmetric losses.

Question 3

What is the role of $Z$ in equation 7?

=> $Z$ was introduced in previous work [1] on noisy classification datasets to address the issue of $\log 0 = -\infty$, which cannot be used when updating a neural network. To resolve this numerical problem, $-\infty$ is replaced with a negative value $Z$, which has been shown to still satisfy the conditions of a robust loss function. In our case, we also use $Z$ to handle negative advantages (see Section 4.1 for more details)

Question 4

Why SPPO still performs well in noisy-free setting?

=> Compared to A2C, PPO has off-policy update parts and typically uses smaller batch sizes (e.g., 64), whereas A2C processes all data points at once (i.e., no batch). Additionally, PPO applies the advantage normalization by default, which can introduce sign changes in the advantages. While PPO is more sample-efficient than A2C, these characteristics introduce sources of noise—even in noise-free settings. Our symmetric RL loss helps mitigate these noisy factors, which can lead to confusion in advantage estimation, thereby improving performance even in clean environments (see Section 5.4 for details).

Question 5

What is the benefits of advantage normalization with small batch sizes?

=> While some studies [2, 3] suggest that smaller batch sizes can be more beneficial than larger ones, our main point is not directly about batch size. In practice, we commonly use batch sizes such as 32, 64, or 512 for PPO, whereas A2C processes the entire dataset without batching.However, after applying advantage normalization, the signs of the advantages can flip depending on how the batch is sampled. For example, a given sample $(s,a)$ might end up with a positive or negative advantage solely based on the composition of the batch. This introduces confusion, which can be interpreted as a form of noise. In such cases, the symmetric loss can help mitigate the impact of that noise.

That said, this doesn’t mean advantage normalization is harmful. Although it can cause sign changes in advantages, it also helps limit the influence of extremely large advantage values, thereby stabilizing the policy update process (Please refer to Section 5.4).

In summary, the symmetric RL loss retains the benefits of advantage normalization while further alleviating the instability caused by sign changes.

Weakness 1

Baselines are too few for comparison in Table 2.

=> We further provide results for PPO in a noise-free setting in Table 12, including an additional environment. Additionally, we report A2C results in Tables 8 and 9 to support further comparison.

[1] Symmetric Cross Entropy for Robust Learning with Noisy Labels, ICCV 2019
[2] The Phenomenon of Policy Churn, Neurips 2022
[3] Small batch deep reinforcement learning, NeurIPS 2024

Thank you for your thorough review and valuable feedback on our paper. We hope the response below addresses your concerns.

Weakness 1

The approach enumerates all possible actions in the loss, which can be computationally feasible for small/medium discrete action spaces (e.g., Atari), but becomes very large and inefficient for language modeling.

=> In large language models, the next token (action) is sampled via softmax, which already provides probabilities for all other (non-chosen) tokens—so there's no need for additional computation to obtain them. Incorporating the probabilities of other tokens during backpropagation introduces some additional computation, but it is roughly equivalent to adding a single MLP layer. Given the overall size and structure of LLMs, this does not significantly impact training speed. In fact, SPPO (the symmetric RL for PPO) showed faster training than PPO for some seeds—i.e., within the overhead of adding one MLP, GPU memory allocation conditions had a greater effect. We briefly address training speed in the last paragraph of Section 5.5.

Weakness 2

More recent studies on LLM alignment typically use newer models (e.g., Llama 2 / 3, Qwen 2.5 or beyond) and richer feedback datasets (e.g., Ultrafeedback, Chatbot Arena open data, Nectar etc.).

=> These methods were motivated from the perspective of noisy rewards in standard deep RL tasks. We show that they perform well across a range of RL tasks and limited experiments in RLHF. It is difficult to cover every setting---we think that the promising nature of this approach is established by our experiments. We promise that if the paper is accepted we will add Qwen2.5 on the TLDR task to the results table.