VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data

We thank the reviewer for your meaningful feedback and recognizing that (i) our paper presents strong empirical results and robust experiments, (ii) it addresses an important gap in PRM research, (iii) it provides thorough comparison between math PRMs and VersaPRM, and (iv) it facilitates open science with open-sourcing.

The synthetic data generation process produces high-quality multi-domain reasoning labels. This claim is not fully supported, as the precision is around 70-75%, lower than 80% claimed from OpenAI.

Our synthetic labeling achieves ~90% of the accuracy of human-labeled PRM800K (the target benchmark), despite requiring no manual effort. The slightly lower precision (70-75%) reflects the cost-effectiveness trade-off: OpenAI’s 80% required expensive human annotation, while our method is fully automated.

The impact of synthetic data noise is not fully explored. Although manual evaluation suggests 75% accuracy, additional breakdowns on how mislabels affect model performance would be beneficial. Auto-Labeling Reliability The auto-labeling method for reasoning steps, while cost-effective, introduces potential noise. The manual evaluation (75% accuracy) suggests that a portion of the training data may contain incorrect annotations, which could impact PRM reliability.

We did not attempt to further decrease the noise in the training dataset, as we found 75% label accuracy sufficient for VersaPRM to generalize robustly across domains and inference methods. From a preliminary analysis, we observe that many mislabels stem from a failure to identify faulty reasoning due to incorrect factual information.

Further noise reduction (e.g., via stronger labeling models) is promising but deferred to future work.

The choice of backbone LLMs (Llama vs. Qwen) is not fully explored—for example, the effect of using even larger models (e.g., GPT-4 or DeepSeek-R1) remains an open question. Impact of larger-scale LLMs on PRM performance (e.g., GPT-4 vs. Llama-3).

In Section 6.4, we provide initial results with DeepSeek-R1 (700B parameters) in the law domain. In addition, we are currently testing this approach in the biology domain and will include these results. Broader exploration of large models for PRM initialization/auto-labeling remains future work.

More examples of mislabeled CoTs to analyze failure cases.

We provide examples CoTs mislabeled by math PRM but not by VersaPRM in this link. We will include more diverse examples of errors that VersaPRM also makes.

There are few typo and words repeated. Please check carefully.

Thank you for noting this—we will thoroughly proofread the manuscript.

Final note: Thank you again for the comments and we appreciate the thorough review!

We thank the reviewer for the thoughtful feedback. We've addressed your concerns below.

Multiple test-time computation methods (MV, WMV, BoN, Beam Search, MCTS) but computational costs across these methods are not controlled

We clarify the computational costs of test-time methods: WMV and BoN scale with the number of generated CoT solutions $N$. Beam Search’s number of beams $B$ is equivalent to $N$, and MCTS scales with the branching factor times the number of iterations. Figure 6 compares MCTS and Beam Search in terms of computational cost for an equivalent number of generated CoT solutions.

However, our goal is not to compare inference-time methods against each other, but to show that VersaPRM consistently outperforms math PRMs across all these methods.

30 examples is insufficient to establish confidence in labeling quality

We expanded our sample to 60 questions. The revised analysis shows that 78% of responses marked correct by the autolabeler were accurate (95% CI: 0.68–0.89), and 70% of those marked incorrect were accurate (95% CI: 0.62–0.78). Based on this, we estimate 74% of CoT responses in the training set are correctly labeled. We will label more examples for the revision.

Using Llama-70B directly as an evaluator at test time compared to VersaPRM

Thank you for your valuable suggestion. We provide an additional experiment comparing VersaPRM against Llama-70B directly as a judge on MMLU-CoT-Eval. VersaPRM outperforms Llama-70B across all values of $N$.

Recall the training data for VersaPRM is labeled by Llama-70B with access to the correct answers during prompting, ensuring high-quality step-wise labels (Section 5.3). At inference, however, Llama-70B lacks prior knowledge of the correct answer, limiting its judging effectiveness as a judge. Thus, the gains stem from our PRM methodology, not merely Llama-70B’s knowledge.

The experiments do not adequately assess the cost-effectiveness of the approach; Using a 70B model to generate training data is expensive

We used Llama-70B via AWS Bedrock batch inference to label ~85K CoTs in our training data, costing ~$50. This is far cheaper than manual labeling or stepwise rollouts. We will clarify this in the final paper.

Lack discussion of several important related areas

Thank you for suggesting potential discussion on important related areas. We will add in detailed discussion of these areas in the final paper.

Limited discussion of potential biases inherited from the generator and labeler LLMs

Thank you for raising this issue. Since both generator and labeler LLMs are Llama models, VersaPRM may perform better on Llama-based inference. A detailed study of the bias would be important, especially for alignment. We will add this discussion in the revision.

The reason behind moderate improvements in some domains (e.g., History) compared to other domains

Manual inspection of questions in domains with moderate improvements (e.g., history and health) reveals that these primarily test factual recall over reasoning. We posit that aggregation methods like BoN and WMV are less effective in these domains, as success depends largely on the LLM’s factual knowledge–not reasoning quality–and whether the PRM can recognize the correct fact. Therefore, we suspect PRMs struggle in these domains due to weaker pretrained knowledge of health/history.

No ablation study about the synthetic data generation pipeline.

Section 5.3 presents an ablation on auto-labeling prompts (Lines 275–282). We also tested removing ground-truth answers, which dropped agreement rates from 83% to 60% for CoT originally autolabeled as correct and 70% to 40% for those autolabeled as incorrect. We are conducting further end-to-end PRM evaluations using ablated training data and will update results accordingly.

A more detailed error analysis showing specific examples where VersaPRM succeeds but math PRMs fail; Visualizing what constitutes "good" versus "bad" reasoning steps across different domains

We provide mislabeled CoTs from math PRMs that VersaPRM correctly labels here. We will include examples of VersaPRM errors.

VersaPRM for iterative refinement of reasoning (rather than just reranking)

Section 6.3 tests MCTS and Beam Search, which achieve strong results with significantly fewer computations.

Open-ended generation tasks extension

We include an experiment on open-ended law questions.

Final Note: We appreciate the in-depth feedback and hope our comments and additional experiments address the questions. If our responses have addressed your concerns, please kindly consider raising the score.

We thank the reviewer for the feedback, and for acknowledging that (i) our method fills a gap in the existing literature on PRMs, (ii) we provide comprehensive evaluation criteria for our model, and (iii) our open-sourced code facilitates future exploration of this field.

Although the automatic labeling method is used, its accuracy still has room for improvement (about 75%), and there may be some incorrectly labeled data, which may interfere with model training and affect the further improvement of model performance.

While the automatic labeling method introduces some noise (~75% accuracy), our PRM remains robust to moderate noise levels. This is evident from VersaPRM’s strong performance gains over previous math PRMs, despite being trained on noisy data. In this paper, we used Llama-70B due to budget constraints and as it was sufficient to create a model that outperforms previous math PRMs. That said, to further increase labeling accuracy, we can employ stronger models, such as GPT-4o, or reasoning models such as DeepSeek-R1 or OpenAI o1. This would give us higher accuracy labeling. Given time constraints for this response, we will aim to include an improved model with more accurately labeled data in the final version of the paper.

Although a variety of evaluation methods are used, these methods are mainly based on existing reasoning strategies and data sets, and may not fully cover all possible reasoning scenarios and fields. For some more complex or challenging reasoning tasks, the performance of the model may need further verification.

We have evaluated VersaPRM across diverse test-time methods. We acknowledge the importance of broader validation and will discuss and explore additional reasoning scenarios in the final version.

Although a comparative analysis with a variety of open source mathematical PRMs was conducted, the comparative analysis was mainly based on existing models and datasets. For some more advanced models or methods that have not yet appeared, the comparative advantages of the model may need to be re-evaluated. Exploring Mathematical Extrapolation of Large Language Models with Synthetic Data (Li et al., Findings 2024)

We will cite “Exploring Mathematical Extrapolation of Large Language Models with Synthetic Data” in the revised version. This paper introduces a novel arithmetical puzzle task and demonstrates how fine-tuning on large-scale synthetic examples enables precise multi-step mathematical reasoning. The fine-tuned model not only solves the puzzles but also generalizes to harder, out-of-distribution problems involving larger numbers and the composing components of the arithmetical puzzle problem. While this approach is effective for math, we note its applicability to non-math domains remains unclear, as defining comparable structured tasks in those areas presents challenges.

Final note: Thank you again for the comments. If you have any remaining questions, please do not hesitate to let us know.

We thank the reviewer for the thoughtful feedback on the paper, recognizing that (i) its claims are well supported and (ii) this work is the first attempt to broaden PRMs beyond the math domain. We will incorporate your suggested revisions into our final paper.

The comparisons the authors carry out in their main experiments are valid. However, to truly demonstrate that VersaPRM is “domain-general” I would have preferred to see some evaluation beyond just MMLU-Pro, especially since the training traces are gathered from it.

While we do not currently have results for VersaPRM on datasets beyond MMLU-Pro, we have conducted additional hold-one-out evaluations, which we will include in the final paper. Specifically, we exclude one domain category (law, biology, or CS) from training and assess whether the resulting PRM generalizes to the held-out domain.

Results are available here. As shown, the WMV generalization performance of VersaPRM trained with one domain held out is comparable to that of the full model. This suggests that its generalization ability is not merely due to broader coverage of the training data, but rather indicates genuine domain-general reasoning capabilities.

I am not aware of any essential references that the authors missed (with the exception of work currently in submission). "To CoT or Not to CoT" (Sprague et al. '24) could help contextualize the authors' finding that math PRMs are ineffective outside math domains - and the authors' demonstration that WMV with VersaPRM outperforms MV on non-math domains is extra surprising (positively) in light of the conclusion from that work that CoT primarily supports math performance and not other domains.

We appreciate this reference and will include it in Section 2. The following sentence that will be added:

“According to the work of Sprague et al. (2024), most of the reported advantages using CoT stem from math or math-related tasks.”

I'm glad to see folks working to broaden the applicability of CoT and test-time search beyond math, these are encouraging results. The MCTS performance under VersaPRM is especially cool - looks like performance doesn't saturate with increasing budget nearly as quickly as it does with the other setups.

Thank you for this positive feedback. We will emphasize it in the final version.

On line 306 there is a missing space between "PRM" and "via".

Thanks for catching this–we will fix it.

Final Note: We appreciate the reviewer’s insights and suggestions and are encouraged by the positive assessment of our work on broadening test-time scaling methods to domains beyond math.