Beyond Verifiable Rewards: Scaling Reinforcement Learning in Language Models to Unverifiable Data

Thank you very much for your time and efforts in reviewing our manuscript. We will make sure to incorporate your feedback into our later revisions.

Thank you very much for your constructive feedback and very positive view towards our paper!

Evaluation of unverifiable tasks relies on proxy metrics only. For Numina-Proof (Sec. 7 & App. C.1), the authors switch to a test-set negative-log-likelihood surrogate. While JEPO shows lower NLL than RL, the paper never validates that lower NLL correlates with human-judged proof quality, leaving the headline claim (“improves reasoning on unverifiable data”) empirically under-supported. A small human study or at least an LLM-as-judge sanity check would have strengthened the evidence.

Thank you for raising the suggestion. Indeed we agree that the result can be strengthened further by correlating NLL with human-judged proof quality. We did not yet have the capacity to carry out the human study by the time of the publication, so tried to resort to test set likelihood as a proxy. With negative results of using automated proof judge [1], we feel that LLM-as-judge for the proof data might not deliver very reliable eval results either.

We did have some manual qualitative assessments of the trained model: while the baseline outputs a proof directly, JEPO trained model outputs a high-level proof strategy followed by the detailed proof. This might corroborate with why the test set NLL is better, due to the learned CoT strategy before the detailed proof.

Despite the limitations, we would still like to argue that the comparison is apple-to-apple, with a reasonable baseline.

[1] Petrov et al, 2025, Proof or Bluff: Evaluating LLMs on 2025 USA Math Olympiad

Loose theoretical guarantees – While the Jensen lower bound is tractable, it is strictly looser than the full ELBO; the paper provides limited analysis of how the gap affects convergence or final policy optimality, leaving theoretical performance bounds unclear.

We acknowledge the theoretical looseness with the bound. With theory, we can seek to close the bound by increasing $n$, but fundamentally this is what we could do with such a methodological design.

We have extensively assessed the empirical impact of such a theoretical loose bound in experiments. Fig 1 showed that despite with a lower bound objective, we can optimize the target reward function; Fig 2 showed that increasing $n$ to $n=4$ correlates with the reward objective better than $n=1$, which corroborates with the performance ablations of Figure 2 across different $n$s.

The high-level message is that while the lower bound objective might be a loose objective, its looseness can be mitigated largely in practice by choosing a reasonable value of $n$. In our ablations, $n=4$ seems to suffice for the math related domains. And we leave to practitioners to understand the impact of this hyper-parameter in their specific settings.

Reward-free versus reward-rich trade-off – In verifiable settings JEPO lags behind standard RL during training and relies on KL regularization to maintain stability, suggesting efficiency drawbacks when high-quality rewards are available; the paper could discuss when one should prefer JEPO in practice.

Indeed JEPO relies on KL regularization slightly more than RLVR in our ablations, but we feel that’s not a strong constraint. More broadly, we find that on domains where data is short-form and the reward is quite accurate, RLVR retains its advantage compared to JEPO (as we have shown in Fig 1). That said, JEPO is still quite competitive in this regime.

Yet in semi-verifiable or non-verifiable domains, JEPO has an advantage due to its more flexible learning objective (without a reliance on good reward). In practical situations where semi-verifiable or non-verifiable data are abundant, JEPO might be worth a try for practitioners.

Thank you very much for your time and efforts in reviewing our manuscript. We will make sure to incorporate your feedback into our later revisions.

Thank you very much for your constructive feedback and very positive view towards our paper!

The main experiments are based on MATH and Numina only, which only assesses in-distribution problems. This is important as we do not always have in-distribution data for training.

We acknowledge this is a limitation of this work, that the training and testing sets are mostly in distribution. That said, we think this setting offers a more narrow yet more controlled investigation into performance gains, and maybe rule away slightly confounding factors that impact how out of distribution generalization works.

We would hope that investigations into out of distribution generalization can be pursued on its own right in another follow up work, or in industry labs where such considerations are more practically meaningful. Nevertheless, we hope this work offers some limited insights.

In the Numina benchmark, roughly 40% of training examples cannot be scored by SymPy. As the authors note, the baseline RL agent therefore optimizes only a subset of the data, and JEPO is evaluated under conditions where a large fraction of examples have no ground-truth verification. This leaves open whether the method scales when the majority of data are truly unverifiable, or when tasks fall outside the narrow domain of mathematical proofs.

Thank you for the comments on this. We have indeed manually checked that among the non-proof Numina dataset, there is still a significant portion (about 40%) of the examples where the answers are in between short-form (e.g., a single number of a single expression) and very long-form (e.g., a full proof). These answers are usually a bit elaborate and cannot be checked by the Sympy scorer we implemented for the experiments.

In Sec 7, when training on the semi-verifiable Numina dataset, we actually excluded the proof Numina sub-set, and only trained on the non-proof section of the dataset. If we understand your comment correctly, this would mean that our approach can scale to cases where a good portion of the data is unverifiable (40%), though not necessarily “majority of the data”. We think it is a meaningful ablation to understand whether JEPO works in the presence of a majority of unverifiable data. That said, in practice, data collection works jointly with algorithmic design, and if the in-the-wild Numina dataset contains 40% unverifiable data, it might be less interesting for a case where most data is unverifiable. Would the reviewer has in mind some dataset like this?

In the core MATH benchmark the authors state "We do not compare with other baselines developed in prior work." This leaves open the possibility that apparent gains might come from experimental setup rather than the proposed JEPO algorithm.

Thank you for pointing this out. Maybe more precisely, what we meant is that: (1) We compare with a RL algorithmic baseline in a way that minimizes the discrepancy between experimental setups; (2) There are other RL algorithmic variants, with changes complementary to our JEPO algorithm.

By (1), we practically mean that the only difference between JEPO vs. RL baseline, is the loss function. They share other settings such as hyper-parameters, training-generation loop, leave-one-out control variates construct etc. This rules out confounding factors as the reviewer worries about as much as possible, leading us to conclude more confidently that the improvements or trade-offs are due to loss functions.

It is also important to note that since we do on-policy training, our algorithm recovers state-of-the-art algorithms like GRPO (when GRPO is on-policy, its importance ratios).

By (2), we mean that there are changes such as different ways of doing group aggregation (e.g., difference between Dr. GRPO vs. GRPO), which are complementary to the designs of our algorithm. That is, in principle, such algorithmic changes can be combined with our approach, for which we find an ablation offers less insight.

For MATH evaluation, it might be more ideal to consider MATH-500, which are less likely to be contaminated.

Thank you for the suggestion on this. We can try to carry out additional evaluation on this if the reviewer finds it useful.

SymPy is known to have many parsing issues specifically for MATH [2], so the reward may not be entirely accurate either.

Thank you very much for the suggestion. We have some additional fixes inside the “Sympy” parser that actually removes some of the false positives and negatives suggested in the paper you shared, but still there might be some inaccuracies remaining. We will make sure to discuss this point in the revision.

The authors use Llama 3 70B as the LLM-as-a-Judge for evaluation. The authors also mention the drawback of using LLM-as-a-Judge for domain-specific and long-form data. While Llama 3 70B is a strong model, it may be a less competitive option compared to some of the leading reasoning models available today, especially on this particular task of verifying proofs.

We agree with the reviewer’s comment on this. We used LLM-as-judge mainly to assess semi-long-form data, so data in Numina which is not as long-form as proof, but is more elaborate than a single mathematical expression that can be verified by Sympy. We find LLM-as-judge to work reasonably well in this case, even with Llama 70B, though indeed more powerful models can potentially provide more accurate assessments.

To provide some calibration results using LLM-as-judge for the Llama 70B model: with manual checks, we find that Sympy correlates with human 80% while LLM-as-judge can improve it to 90%. But potentially more competitive models can offer even higher correlation with some tweaks or tuning, which we leave to further investigations.

We want to emphasize that we did not use LLM-as-judge to verify proof in the paper. The main reason is due to the negative result shared in [1], where the authors shared discrepancies between LLM-as-judge from frontier models and human annotators. Though the Numina proof dataset might be distributionally quite different from the Math Olympiad as studied in [1], we feel less confident in using LLM-as-judge score for proof verifications.

[1] Petrov et al, 2025, Proof or Bluff: Evaluating LLMs on 2025 USA Math Olympiad

For the semi- and unverifiable splits, success is measured by a composite score that falls back on majority voting from Llama 3 70B when automatic SymPy checks fail. Because of the above reasoning, it makes it hard to know whether gains are real reasoning improvements or artifacts of that judge's potentially deviated preferences.

We acknowledge that LLM-as-judge might potentially have its deviated preferences, e.g., to certain formatting. In our early investigation of LLM-as-judge, we do find that the judge model can alleviate more false positives than the false positives it introduces compared to Sympy checker. As evidence of this, we have found that training against LLM-as-judge signals helps improve evaluation performance to a level slightly better than training against just limited Sympy signals.

Our early manual check of LLM-as-judge found that it is quite accurate for assessing medium-form data in the semi-verifiable dataset (note that they are generally not as long as full proof, but rather elaborate short-form answers). We can provide a more accurate evaluation of LLM-as-judge against human annotations in the revision, in case the reviewer finds it useful.

In the introduction (line 28), the authors define data as unverifiable if the ground truth answer to the problem cannot be verified with reasonable automated effort. If I understand correctly, this definition still assumes the availability of the answer. Is that correct? If so, do we also care about the quality and the uniqueness of the answer?

That is correct, we still assume access to a ground truth answer.

Regarding quality: we think it is definitely important to guarantee that the answers are of high quality. Indeed, this will directly feed into how well the model learns from the training set and impacts its ability to generalize. We feel this argument applies for both regular RLVR and JEPO.

Regarding uniqueness: we believe this is quite a subtle question and whose impact might warrant more careful investigations. In RLVR, the verifier or reward function judges if an answer is equivalent to the ground truth answer, and so it kind of assumes that the ground truth might be non-unique (otherwise we’d use a string match reward). It might be interesting to investigate further how the diversity or uniqueness of the answers, impact RLVR and JEPO in more practical scenarios.

In Section 5, the authors indicate the difficulty of using LLM-as-a-Judge as a way to tackle unverifiable data. Do the authors have concrete examples of this, even when the (unverfiable) ground truth answer is available?

We meant to indicate that LLM-as-judge can be used for assessing the equivalence of model generated answer and an available ground truth answer, hence replacing the role of a Sympy checker in more unverifiable domains. That is, our discussion always assumes access to a ground truth reward.

In case a ground truth reward is not available, it feels more akin to the conventional RLHF settings where a reward model trained from human preference or a judge model directly assesses the quality of responses. Along this line, [1] is an example where the LLM is directly prompted as a judge to deliver an evaluation score (without access to ground truth).

[1] Lee et al, 2024, RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Thank you very much for your time and efforts in reviewing our manuscript. We will make sure to incorporate your feedback into our later revisions.

Line 70… Line 160 reads a bit awkwardly...

Thank you very much for the extremely detailed feedback on typos and grammar improvements, we will make sure to revise in our next manuscript.

Contrary to the checklist point 7, I don't see any error bars..

Indeed, lightly shaded dots are results from multiple runs with the same configuration. In our own experience, we have found in some cases, it is a bit more informative to display the full distribution over different runs rather than standard errors or confidence intervals. We’d be happy to provide additional information such as error bars and confidence intervals if the reviewer finds them useful.

I would suggest moving the SFT comparison from the Appendix to the main paper...

We are definitely happy to move the SFT ablation discussion and more details into the main paper as the reviewer suggests. The current formatting in the paper is primarily due to space limitations, but if we are allowed an extra page in camera ready, additional results can be presented in the main paper.

Line 203 - Things like learning rate should be tuned (at least using a coarse search) for each method.

It is true that hyper-parameters like learning rate can be ablated in this case. In our early investigation, we have found the evaluation performance to be reasonably robust against learning rate variations, though the training curve gets impacted slightly (e.g., within a stable interval of learning rate, increasing the learning rate leads to faster growth in training reward across the board). We can provide additional information in the revision.

Besides learning rate, we have indeed included other hyper-parameters for comparison, which we find to be central to RLVR methods in general, such as the number of samples per prompt $n$ and the regularization coefficient $\beta$.

Lines 206-208 - I don't think this is good reasoning for not comparing against other work... What RL algorithm is used in the paper?

We agree that some technical details can make a material difference in practice. Maybe a more precise way to state the rationale here, is that some algorithmic variations are orthogonal to our algorithmic proposals in this work, and can be readily combined with our method in a complementary way. For such cases, we argue it might offer less value to carry out a comparison.

The algorithm we compare against is a simple version of RLOO (Reinforce with leave-one-out [1]. We argue it is the backbone for most existing RL algorithms such as GRPO [2] and its variants. Indeed, when on-policy (which is the experimental setup we adopt), GRPO and its variants will fall back to RLOO. RLOO also shares the most algorithmic structure with our method (e.g., leave-one-out control variate), which we find a comparison might be the most insightful.

[1] Ahmadian et al, 2024, Back to Basics: Revisiting REINFORCE...

[2] Shao et al, Pushing the limits of mathematical reasoning...

Lines 227-230 - on Figure 1, the RL frontier is clearly to the left of the JEPO frontier, except for very high KL divs..

We acknowledge that RL traces out a better KL-reward frontier in this case. But we have emphasized a few points in the paper: (1) It is the training plot where RL is better, and on evaluation plot (mid plot), JEPO is as competitive; (2) RL is at an advantage here because it has access to the reward function to plot against too, unlike JEPO, which does not assume access to the reward.

The key message is that Figure 1 shows a case where RLVR should work the best (i.e., short-form data with a good verifiable reward). Section 7-8 shows the results on semi-verifiable and unverifiable domains, where JEPO methods shine. This is a practical trade-off we seek to highlight in the paper.

Figure 2 - it's interesting and good to verify but perhaps unsurprising that RL ...

Thank you for finding the results interesting. Indeed, the results are meant to display a potential misalignment between likelihood based approach and Sympy based approach. It is indeed a limitation from the space constraints that causes our sub-optimal arrangement of texts and figures in the paper, we will make improvements in the revision.

Lines 260-261... I think it's pretty important to have some quantification of the false positive and false negative rates across RL answers and JEPO answers on some subset of data.

We acknowledge that LLM-as-judge might have its limitations. In our early investigation of LLM-as-judge, we did find that the judge model can alleviate more false positives than the false positives it introduces. As evidence of this, we have found that training against LLM-as-judge signals helps improve evaluation performance to a level slightly better than training against just limited Sympy signals.

With some manual check, we find that Sympy correlates with human 80% of the time while judge correlates with 90%. We find that the judge’s improvement on Sympy’s false negative overwhelms its potential loss on false positive, resulting in an accuracy that’s higher than just using judge. Hence we use a combined score for eval. We are happy to provide more detailed LLM-as-judge calibration results in revision.

Lines 335-338 - I have a simpler alternate explanation...

We agree with the reviewer’s interpretation, this is indeed the conclusion that we seek to find more empirical evidence for, since it is not obvious by default.

originality of the paper I'm saying is fair

Despite the strong connections to prior work such as ELBO, VAE, IWAE, we believe that there are many technical contributions on top which distinguish our work: (1) JEPO is the first implementation of such ideas “as they are developed mathematically”. Though prior work borrows high level ideas, the implementations turn out to be very much similar to RLVR, see discussion in 170-175; (2) JEPO has unique technical contributions specific to LLM, instrumental to actually making the system “work”, e.g., how to resolve certain technical constraints specific to LLM. Indeed, it is not trivial to apply VAE ideas to LLM out of the box.

While JEPO is not groundbreaking in its mathematical foundation (which is the same as VAE), we argue that lots of important work in space is not either. Rather, we feel technical solidity should take more priority over novelty in the space of ideas.

Line 87 - regarding the motivation, is this really what we want? ... but I'd be interested in seeing more analysis around whether this is a problem.

We believe JEPO does not resolve the fundamental CoT correctness issue arising in conventional outcome based RLVR scenarios. Indeed as the reviewer suggested, we cannot do better unless a process supervision signal is available.

Our speculation is that by leveraging potentially more unverifiable and semi-verifiable data, JEPO can benefit from a larger effective training set, compared to RLVR with a more rigorous reward function. In doing so, the model might overfit less to the final ground truth answer, and hence reducing the effect of “bad CoTs leading up to correct answer”. We do not have a systemic evaluation of this, rather we only evaluate the test set performance as a proxy metric.

is itself a lower bound right? Seems worth discussing and investigating more. Also, how does this compare to the unbiased term in Eq. 4?

Note that $g_2^{(n)}$ is itself not a lower bound, but rather a partial gradient to the $n$-sample lower bound (see Eqn 7).

Eqn 4 shows $g_2$ for a single sample lower bound, which can be seen as a special case to the $n$-sample lower bound with $n=1$.

Comparing the two: these are gradients to different lower bounds. In general, the bound is tighter as $n$ increases (see discussion near line 131), but the variance of the gradient might increase as well, causing a trade-off in principle. Figure 2 shows the empirical comparison that indicates that increasing $n$ in general seems to be helpful, though there is also the practical constraint of the algorithm being more expensive in $n$ due to generations. We will highlight more such discussions in the revision.

I only took a brief look at VR-CLI but it seems to me more different from your work than your writing suggests?

We leaned on the more conservative side and discussed some similarities between VR-CLI and our approach. To be fair, VR-CLI is on a high-level similar to using likelihood as a reward, which bears intuitively resemblance to the first part of the gradient $g_1, g_1^{(n)}$ in this case. Yet, many technical details might differ: VR-CLI is not theoretically grounded to improve the likelihood objective, as we have discussed in this work with the lower bound formulation, which requires transformation of individual log likelihood.

Lines 334-335 - where in Section 6 do you show the different

This was a discussion of early investigations we have: we found that when $\beta_\text{sup}$ is large, it tends to produce worse results with short-form answer dataset like MATH. We speculate this is because the force of SFT loss is too large with large values of $\beta_\text{sup}$, and causes the training to overfit more (after all, it’s simple to predict just a few tokens worth of answers, compared to long-form answers). We will provide more detailed results in the revision.

There is much more the authors could discuss on limitations.

We are happy to discuss the limitations as the reviewer pointed out, given more space in the camera ready revisions.

The authors claim that "Our work is algorithmic and theoretical in nature and does not produce immediate societal impacts"

We agree that if the algorithmic improvements reach certain points, it can generate sizable social impact. We are happy to provide more discussions on this in the checklist and in the revision.

Thanks for your response!

Quotes below abbreviated for character limit.

Indeed, lightly shaded dots are results from multiple runs with the same configuration. [...] We’d be happy to provide additional information such as error bars and confidence intervals if the reviewer finds them useful.

Yes, please provide error bars/confidence intervals, with at least 3 seeds (ideally 5 or 10).

It is true that hyper-parameters like learning rate can be ablated in this case. In our early investigation, we have found the evaluation performance to be reasonably robust against learning rate variations, though the training curve gets impacted slightly (e.g., within a stable interval of learning rate, increasing the learning rate leads to faster growth in training reward across the board). We can provide additional information in the revision.

Besides learning rate, we have indeed included other hyper-parameters for comparison, which we find to be central to RLVR methods in general, such as the number of samples per prompt $n$ and the regularization coefficient $\beta$.

Yes, please provide additional information, including which hyperparameters you compared and tuned, which were most important, what ranges you roughly considered, how you tuned, etc.

We agree that some technical details can make a material difference in practice. Maybe a more precise way to state the rationale here, is that some algorithmic variations are orthogonal to our algorithmic proposals in this work, and can be readily combined with our method in a complementary way. For such cases, we argue it might offer less value to carry out a comparison.

The algorithm we compare against is a simple version of RLOO (Reinforce with leave-one-out [1]. [...]

Thanks for the clarification. The paper could be reworded to better reflect this.

We acknowledge that RL traces out a better KL-reward frontier in this case. But we have emphasized a few points in the paper: (1) It is the training plot where RL is better, and on evaluation plot (mid plot), JEPO is as competitive; (2) RL is at an advantage here because it has access to the reward function to plot against too, unlike JEPO, which does not assume access to the reward.

Sorry, I still strongly disagree with JEPO being competitive even on the evaluation plot. I reiterate my original comment which was meant for both the train and evaluation plots: "the RL frontier is clearly to the left of the JEPO frontier, except for very high KL divs. For very high KL divs, you might be running into the limits of model capacity, which may explain why performance is similar. In any case, the point of building these plots and frontiers is to look at the whole frontier, and overall the RL frontier looks much better."

The key message is that Figure 1 shows a case where RLVR should work the best (i.e., short-form data with a good verifiable reward). Section 7-8 shows the results on semi-verifiable and unverifiable domains, where JEPO methods shine. This is a practical trade-off we seek to highlight in the paper.

Maybe (I'm not entirely sure about this) it might even be better to just cut out Figure 1 entirely (or leave it to the Appendix, as a sort of ablation/study just for understanding purposes), and instead do more experiments in the settings where JEPO shine, to further illustrate the usefulness of JEPO over existing RL methods. That is, you generally don't convince others by showing a setting where your method is expected to be disadvantaged, and then demonstrating that you are disadvantaged (even if less so than one might initially expect). Your efforts are probably better spent focusing on demonstrating where your method is advantaged (and justifying why those situations where your method is advantageous are of practical relevance/importance).

[...] We are happy to provide more detailed LLM-as-judge calibration results in revision.

That would be great.

We agree with the reviewer’s interpretation, this is indeed the conclusion that we seek to find more empirical evidence for, since it is not obvious by default.

If you agree, you should edit the paper to reflect this.

Despite the strong connections to prior work such as ELBO, VAE, IWAE, [...] Indeed, it is not trivial to apply VAE ideas to LLM out of the box.

While JEPO is not groundbreaking in its mathematical foundation (which is the same as VAE), we argue that lots of important work in space is not either. Rather, we feel technical solidity should take more priority over novelty in the space of ideas.

Yes, that's why I'm saying it's fair, not bad.

[...] We will highlight more such discussions in the revision.

Great, thanks for the additional clarification.

[...] We will provide more detailed results in the revision.

Great, thanks.

Thank you very much for your time and efforts in reviewing our manuscript. We will make sure to incorporate your feedback into our later revisions.

Thank you very much for your constructive feedback and very positive view towards our paper!

Entropy dynamics: In JEPO, how does the entropy behave during training? In standard verifiable RL training, entropy typically decreases and then stabilizes. Does JEPO follow a similar pattern, or is the behavior different? Would adding an explicit entropy loss to JEPO be beneficial?

In JEPO, entropy dynamics behaves rather similarly as regular RL training. This might be because JEPO ultimately looks to find CoTs which maximize the marginal likelihood of the ground truth answer, in a similar spirit as regular RL that finds CoTs that make the ground truth answer generation more likely.

We speculate that adding entropy to JEPO would potentially have a similar effect as adding entropy to regular RL. It might help avoid premature convergence in some cases but tuning the entropy coefficient itself involves some hyper-parameter tuning to make things stable.

Answer granularity: In the verifiable reward setting, is the correct answer just the final boxed answer (e.g., an integral value in AIME problems), or does it include the full solution reasoning block? How do you extract from the generation? Do you simply parse the content inside <think> tokens?

Thank you for raising this point! We have actually discussed the specifics of this important technical detail in Appendix B. We will make it more clear in the revision.

During SFT we have taught the model to carry out reasoning, followed by the final answer is, followed by the ground truth answer (e.g., an integer value). In other words, the overall string output from the model should be

Reasoning + the final answer is + answer

We extract the string before the marker the final answer is as the reasoning CoT $c_i$, and the string after as the final answer $a$. If the model does not comply with this formatting, we penalize the model output with a constant penalty. We find that with this, the model can quickly learn to format the full response correctly.

Another idea is indeed as the reviewer suggests, we can use a special thinking token as a delimiter, rather than a marker string. This will potentially make the formatting more stable and the parsing more convenient.

Robustness to incorrect labels: How robust is JEPO to incorrect ground-truth labels ? For example, in [1], it was shown that RLVR is relatively robust to such label noise. Have you observed similar robustness in JEPO?

We have not thoroughly investigated the robustness of JEPO performance to ground truth labels. Though based on the conceptual connection between JEPO and regular RLVR, we would expect some robustness to transfer given the evidence in [1]. It would be a valuable follow up ablation to carry out, to better understand whether JEPO scales better in the presence of data noise.