Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales

However, the experiment results are not convincing.

First, the PPO baseline results in Atari seem to underperform the known results at openrlbenchmark. See the table below, which shows the author's PPO performance significantly underperform the reported performance. I am using the results at 5M episodes for a fair compairson because it seems the authors used 5M episodes in Figure 3, which is missing a legend.

The RLHF experiment design also seems problematic. The GPT4 win rate of a 6B model with SPPO in TL;DR summarization is only 52.50%, whereas comparable work using RLOO gets 77.9% win rate (https://arxiv.org/pdf/2402.14740) or PPO 67%. Some of these discrepancies might come from the training codebase / dataset. For example, the CarperAI/openai_summarize_comparisons has double spacing in its preference dataset between TL;DR: and the actual response. Some of the weird quirks the authors suggested with the policy model / RM might come from small but relevant details like this.

>>> from datasets import load_dataset
>>> ds = load_dataset("CarperAI/openai_summarize_comparisons", split="train")
>>> print(ds[0]["chosen"])
TL;DR:  Snooped, found something, should I admit what I found so we can have a more honest conversation about it with less denial on her part?
>>> ds
Dataset({
    features: ['prompt', 'chosen', 'rejected'],
    num_rows: 92534
})
>>> print(ds[0]["chosen"][5])
:
>>> print(ds[0]["chosen"][6])
 
>>> print(ds[0]["chosen"][7])
 
>>> print(ds[0]["chosen"][8])
S
>>> 

Overall, despite having interesting theory and analysis, I am concerned by the quality of the experiment design and results.

1. The introduction should provide more context on the current research directions and challenges in enhancing robustness within the field. Without this background, readers may struggle to understand the specific difficulties and the broader context of the problem.
2. The method section lacks clarity, making it difficult for readers to follow the transition from the standard RL loss described in the PRELIMINARIES section to the modifications introduced in the APPROACH section. Additionally, the definition of $Z$ is vague, only stating that Equation 5 defines $Z$, which complicates understanding.
3. The experimental section could benefit from comparisons with other methods aimed at enhancing robustness in RL. Such comparisons would provide a more comprehensive evaluation of the proposed method's performance and highlight its advantages and disadvantages relative to other robustness-enhancing techniques.

1. The paper needs to be better polished and organized, especially in the Experiment section. For instance:

   • Figure 1 is placed on page 5 but is first cited on page 2.
   • Figure 2 is in the Approach section on page 6, but its first citation is in the Experiment section on page 10.
   • The caption of Table 2 is inconsistent with its content.
   • "Figure 4.1" is mentioned in line 466, but its reference is unclear.
   • The phrase "the sum of rewards as defined in 1" in line 167 is ambiguous; the specific reference needs clarification.
   • Notations in Section 5.2 are confusing. The first and second paragraphs introduce DA2C and DSPPO, but the third uses SA2C and SPPO.

2. Section 5.4 appears to contribute minimally to the paper. The observed benefits of SPPO over SA2C are likely due to the advantages of PPO over A2C, rather than the symmetric loss proposed in this paper.

• This paper seems to have the seeds of a great idea. Some deeper investigation and perhaps an attempt at generalizing the proposed approach beyond two specific loss functions might be useful.
• I could see the parallels between the RCE loss and the proposed reverse A2C and reverse PPO losses. But a first principles derivation of the loss could perhaps be more convincing and lead to a more general loss. As it stands, the loss seems slightly shoe-horned, and I'm not convinced it is the correct drop in for a reverse cross-entropy loss. I see the advantage value $A(x, k)$ as the equivalent of $q(k|x)$. That cal also be connected to the noisy labels and the error in advantage estimation. But the proposed loss seems to consider the action taken by the agent to be $q(k|x)$. Could the authors explain the justification for this choice? One suggestion to instead use the advantages is to map the advantages to a simplex and turn them into probabilities. The RCE loss can then be directly adapted. Could that perhaps be a better approach?
• Perhaps this is a repeat of the previous point, but in equation 8 the clipping used in PPO is not present. Also since the PPO loss does not follow the CE loss structure of having a log p, the presence of Z in the reverse seems incongruous.
• Instead of more specific weaknesses for the work presented in the paper, I have questions for why certain approaches are not considered, or suggestions to strengthen the paper. I include these in the next section. The rest of this section I will point out minor typos and edits.
• On line 53, perhaps cite [1] as an example of ensembles being useful for better value prediction
• On like 54, perhaps cite [2] or some other paper for an example of normalization helping.
• Lines 68-70 seem like a generalization. A2C does not do advantage normalization
• The paper calls its solution the "symmetric RL loss" but the loss pertains to policy gradient, or even more specifically to actor-critic, methods. Perhaps call it the symmetric policy gradient loss?
• Line 177-178. The advantage function is slightly misconstrued here. The advantage function estimates how much better it would be to deterministically take action $a$ instead of following the current policy $\pi$.
• Line 178: " In the approach section" is an awkward phrasing. Consider "in the next section"
• Line 189: " ... also consider incorporates ..." is wrong grammatically. Perhaps drop "consider"
• Line 213 claims: A highly engineered reward function is required to eliminate errors, .... A reference to back this claim up would be helpful.
• Line 215: "Has model errors", can be better expressed as "has estimation errors" or "approximation errors".
• Line 248: Perhaps the reference is meant for Equations 7 and 9, instead of 4 and 9?
• Line 399: Better to cite Bellemare et al. [3], for the arcade learning environment.
• Line 403 posits that the Atari environments only give rewards of 0 or 1. That does not seem to be correct. Is the paper focusing on a particular setup where the rewards are clipped?
• Line 457: Extra "is": Note that the open-source GPT-J model "is" often outputs empty summarizations for most evaluation data
• The table in Appendix 14 does not clarify whether the "performance increase" is perplexity or reward.
• Line 721, citation for Wang et al. can instead be for the published paper [4]
• Last few lines of page 15: it is helpful to review if you specify what operations you did at each step here.

References: [1] Fujimoto, S., Hoof, H. and Meger, D., 2018, July. Addressing function approximation error in actor-critic methods. In International conference on machine learning (pp. 1587-1596). PMLR.

[2] Yue, Y., Lu, R., Kang, B., Song, S. and Huang, G., 2024. Understanding, predicting and better resolving Q-value divergence in offline-RL. Advances in Neural Information Processing Systems, 36.

[3] Bellemare, M.G., Naddaf, Y., Veness, J. and Bowling, M., 2013. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47, pp.253-279.

[4] Wang, Y., Ma, X., Chen, Z., Luo, Y., Yi, J. and Bailey, J., 2019. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 322-330).