Thank you for your thoughtful and constructive review - we're glad you enjoyed the narrative clarity, novelty of applying particle filtering to inference-time scaling, and the strength of our experimental results and analysis.

 

Conditions Under Which Addional Exploration is Helpful: It’s true that in easier problem settings - where the language model has a very high probability of answering correctly - extensive exploration may not be necessary, and early pruning can save compute. That said, reward models are inherently uncertain: they’re not verifiers, but transformer models trained to assign scores, and scoring partial solutions is particularly noisy and difficult. Because of this built-in uncertainty, for most problems, and especially for non-trivial tasks, the stochasticity in particle filtering is beneficial - it helps retain potentially correct paths that might otherwise be prematurely discarded.

 

---

MCTS Comparison Thank you for this thoughtful comment. While both PF and MCTS seek to balance exploration and exploitation, they differ fundamentally in design and suitability for inference-time scaling of LLMs.

• Design:

  • MCTS builds a tree and backpropagates scores from full rollouts or value estimates.

  • PF maintains a flat particle population, updates weights step-by-step, and resamples stochastically.

• Value Function Sensitivity:

  • MCTS can use value estimates to avoid full rollouts, but errors in these estimates backpropagate and bias future decisions.

  • PF avoids this by using only local stepwise scores in a stochastic manner - more robust to reward noise.

• Parallelism & Scalability:

  • MCTS requires sequential updates and backtracking, limiting parallelization.

  • PF updates all particles in parallel, aligning better with modern inference engines like vLLM.

• Empirical Comparison:

  • We ran MCTS (from Zhang et al. 2024) vs. PF on AIME 2024 with LLaMA3.1-8B and budget 32.

  • PF: 16.6% accuracy, MCTS: 6.6%—PF outperforms MCTS under equal compute.

We will include this empirical comparison and discussion in the final version.

References:

Zhang, D., Huang, X., Zhou, D., Li, Y., & Ouyang, W. (2024). Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B. https://arxiv.org/abs/2406.07394

 

---

Thank you again for your thoughtful review. We hope these clarifications and updates address your concerns and further strengthen your confidence in the work.

Thank you for your detailed review and for highlighting the importance of our problem setting, the soundness of our approach, and the strength and breadth of our experimental evaluation - we appreciate your careful assessment. We address additional comments below.

 

Deterministic Search vs. Particle Filtering: Thank you for the thoughtful comment. While it's true that particle filtering can also discard low-scoring trajectories, the key distinction is that it does so stochastically rather than deterministically. This means there is always a non-zero probability of retaining low-scoring (but potentially correct) paths, unlike deterministic search which prunes them irrevocably. As a result, particle filtering offers greater robustness to early reward noise, especially under imperfect PRMs, by maintaining exploration over diverse hypotheses.

An example of this behavior is illustrated in Figure 2 of our paper, which shows a real example of the full PF method: the first step of the trajectory that ultimately leads to the correct final answer receives the lowest initial score. Yet, because PF retains a non-zero chance of continuing such paths, this trajectory survives and ultimately yields the correct solution.

 

---

MCTS: While both Particle Filtering (PF) and Monte Carlo Tree Search (MCTS) aim to balance exploration and exploitation, they differ fundamentally in design and suitability for inference-time scaling in LLMs. To directly compare the two, we implemented the MCTS baseline from Zhang et al. and evaluated it against PF using the same base model (Llama 3.1 8B Instruct) and compute budget (32 rollouts). On the AIME 2024 benchmark, PF achieved 16.6% accuracy, significantly outperforming MCTS, which achieved only 6.6%. We will include this empirical comparison and accompanying discussion in the revised version.

References:
Zhang, D., et al. (2024). Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B.

 

---

Confidence Intervals: Thank you for this important suggestion — we completely agree. To address this, we ran BoN, Beam Search, and Particle Filtering across 3 different seeds using 3 different generator models on the AIME 2024 benchmark. Below, we report mean accuracy along with standard deviation (±), which we will include in the revised version to better reflect result variability and statistical significance.

 

---

Additional Benchmarks: Thank you for the observation — we agree that many questions in FinanceBench and NumGLUE involve significant numerical reasoning, and that this likely contributes to the strong performance of our math-trained PRM. While these tasks also include elements such as reading comprehension and scientific context, we acknowledge that the overlap with mathematical reasoning is substantial. We will revise the discussion to better reflect this and avoid overstating the generality of the PRM.

 

---

PRM vs. ORMs: We appreciate the question. While "PRM as ORM" scores entire partial trajectories, it is still fundamentally different from using an outcome reward model (ORM). PRMs are explicitly trained to evaluate partial solutions step-by-step, whereas ORMs are trained to judge only final outputs. Since particle filtering requires scoring and resampling based on intermediate generations, PRMs remain the appropriate choice—even when used in aggregation modes resembling full-sequence evaluation. We will clarify this distinction in the revision. Notably, as we show in Figure 5b, other methods for using PRM scores, including the Last score, also perform very well.

 

---

TSMC: Thank you for this thoughtful and detailed clarification—we appreciate the opportunity to improve the precision of our comparisons to TSMC. You are correct that our original statements could be clarified further.

• Our original statements referred specifically to the formal setup introduced in Feng et al. [8], which trains an optimal value function via Contrastive Twist Learning. This approach relies on both a trained generator and supervision from a ground-truth verifier, as detailed in Sections 3.4, 4, and Appendix C of [8]. You are correct that TSMC doesn't require a ground-truth verifier at inference time—the verifier is only used during training to shape the twist function.

• This highlights a key difference in our premises: while TSMC (e.g., [8], [33]) focuses on learning theoretically optimal twist functions, our work assumes that PRMs are inherently imperfect and noisy. We ask a different question—how to best scale LLM inference given an off-the-shelf PRM—with a method that requires no training or tuning, offering greater robustness in practice.

• You're absolutely right that Zhao et al. [33] released their code. Our comment specifically referred to Feng et al. [8], which we focused on since it applies TSMC to math. To our knowledge, their implementation and models are not publicly available—we’ll clarify this in the final version.

• Regarding empirical comparisons, we acknowledge that [8] uses a trained value function and argues that off-the-shelf PRMs are weaker. Our goal was to highlight that even under this disadvantage—using a fixed off-the-shelf PRM (Qwen2.5-Math-PRM-7B)—our method outperforms [8], achieving 75.4% on MATH500 with 128 samples, compared to 60.8% in [8] using 240 samples, a trained generator, and a trained value function.

We’ll revise the final version to clearly reflect that [8] is focused on learning optimal PRMs, while our work addresses how to conduct inference-time search when the PRM is fixed and noisy.

 

---

Runtime Exploration: Thank you for the thoughtful comment. We ran wall-clock time comparisons on a 50-question subset of MATH500 using Qwen2.5-Math-1.5B-Instruct and vLLM as the inference engine:

• BoN (budget=32): 211.69 sec
• BoN (budget=64): 392.14 sec
• Beam Search (budget=32): 549.48 sec
• Particle Filtering (budget=32): 401.63 sec

We find that PF at budget 32 takes comparable time to BoN at budget 64, while achieving higher accuracy (84.6 vs. 82.8), indicating a favorable compute–accuracy tradeoff.

Regarding FLOPs: in theory, PF, BoN, and Beam Search have similar cost under perfect KV-cache reuse (where past token KV-caches are fully reused in decoding). However, in practice, step-wise inference leads to low KV cache hit rates because inference engines like vLLM aren't optimized for token-by-token generation. This causes duplicated compute and increased latency. Importantly, these differences are driven more by current engineering constraints than by the underlying algorithm.

(As further evidence of how much these engineering implementation details can dominate: running DVTS, a better version of beam search, took days.)

We will include these latency results and a discussion of the practical vs. theoretical tradeoffs in the revised version.

 

---

Greedy Lines in Figure 4c: The small discrepancy in Figure 4c arises because greedy decoding uses temperature 0, whereas the multi-sample methods—including the 1-sample case at $2⁰$—use temperature 0.8. Thus, the "1 sample" point reflects a stochastic sample that may occasionally outperform deterministic greedy decoding, depending on the model. In the 1.5B case (Fig. 4c), this occurred; for the 7B model (Fig. 4d), greedy slightly outperformed the sample.

That said, the differences are minute—on the order of 0.005 to 0.01 in accuracy—and well within the expected variance from sampling.

 

---

Performances of Different PRMs:

• Thank you for highlighting this insightful finding.
• EurusPRM-Stage2 improves at higher budgets likely due to greater robustness to noise with more particles:
  • At low budgets, PF relies heavily on accurate early-step scoring, and EurusPRM may be less reliable at early steps due to noisier or less calibrated partial scores.
  • With more particles, PF can better tolerate this noise—allowing good trajectories to survive and improve final accuracy.
• Qwen2.5-Math-PRM-7B stagnates early possibly because of limited answer diversity:
  • Most viable completions are already surfaced with a few samples. Additional samples contribute diminishing returns.
• We note that this ablation used only 100 MATH500 questions (vs. 500 in main results), so variability is higher, especially at small budgets.
• We’ll clarify these points in the revision.

 

---

Social Impact: Thank you for raising this important point. We will revise our broader impact statement to note that while our method improves inference robustness, it could amplify harm if misused in sensitive applications.

 

---

We will be sure to fix all formatting related concerns in the camera ready version of this work.

We appreciate your thoughtful feedback. We hope that our clarifications on the novelty of our approach and the strength of our experimental results have further reinforced your confidence in the significance of this work.

Thanks for your response! The additional discussion is helpful, and I appreciate your additional results, which increase my confidence in your findings and conclusions.

Quotes abbreviated below for character limit.

PRM vs. ORMs

Yes, ORMs are only trained to judge final outputs, but they can also provide scores over partial sequences. One might expect the PRM to be better on partial sequences since that's what it's explicitly trained for, but I think it would be good to have empirical validation of this. This is probably getting a bit off topic from the point of the paper though so we can stop discussing this.

TSMC section

Overall I think much more, and much clearer, discussion needs to be had with regards to this relationship. I agree with the other reviewer APTW who also highlights the connection to particle based methods as a weakness that needs to be better addressed and clarified.

Our original statements referred specifically to the formal setup introduced in Feng et al. [8], [...]

My suggestion would be to dedicate an entire paragraph, possibly even an entire subsection, to clarifying the relationship with TSMC. If you are referring to a specific work, then avoid lumping in other works in the same bucket. But I wouldn't recommend referring too much to a specific work either, without also comparing to other closely related works (I mean specifically from the conceptual/theoretical standpoint here).

This highlights a key difference in our premises: while TSMC (e.g., [8], [33]) focuses on learning theoretically optimal twist functions, our work assumes that PRMs are inherently imperfect and noisy. We ask a different question—how to best scale LLM inference given an off-the-shelf PRM—with a method that requires no training or tuning, offering greater robustness in practice.

The first question you ask is the same one that [8] focuses on - how to best scale LLM inference given an off-the-shelf PRM. If the key difference is in avoiding training/tuning, this is the part that should be highlighted. And what do you mean by "greater robustness" here?

You're absolutely right that Zhao et al. [33] released their code. Our comment specifically referred to Feng et al. [8], which we focused on since it applies TSMC to math. To our knowledge, their implementation and models are not publicly available—we’ll clarify this in the final version.

As I mentioned above, it's best not to conflate different works and lump all TSMC works in the same bucket. You should compare with TSMC works broadly; I encourage general comparison, but I discourage making comments that come across as general comparison but are actually based on a specific work.

Regarding empirical comparisons, we acknowledge that [8] uses a trained value function and argues that off-the-shelf PRMs are weaker. Our goal was to highlight that even under this disadvantage—using a fixed off-the-shelf PRM (Qwen2.5-Math-PRM-7B)—our method outperforms [8], achieving 75.4% on MATH500 with 128 samples, compared to 60.8% in [8] using 240 samples, a trained generator, and a trained value function.

This wasn't what I was trying to get at with my original comment. My point was that you're using different off-the-shelf PRMs, and that your off-the-shelf PRM is likely quite a bit better than the off-the-shelf PRM that [8] uses, which I think makes the comparison unfair, and advantaged in favor of your work.

We’ll revise the final version to clearly reflect that [8] is focused on learning optimal PRMs, while our work addresses how to conduct inference-time search when the PRM is fixed and noisy.

As far as I'm aware, [8] is not about learning optimal PRMs. Twist functions are not the same as PRMs. You could try to use them interchangeably, but they're trying to learn different things; optimal twists are defined in relation to the target distribution which also generally contains the base model as part of it. Twist functions are akin to value functions in soft RL (see [33] for the theoretical details of this connection), which are not the same as PRMs. [8] discusses a connection to PRMs in 3.3, noting some similarities and other key differences.

I get the feeling the authors have not really understood the connection between their work and existing TSMC works, both from the original paper and from the follow-up comments. I would appreciate if the authors could really dig into this and better understand and discuss. I would be happy to discuss this with the authors in more detail, but would like to see the authors do a better, much more detailed attempt first.

Runtime Exploration

Thanks for these results. They are great, and I think should go in the main paper (with confidence intervals on accuracy).

That said, the differences are minute—on the order of 0.005 to 0.01 in accuracy—and well within the expected variance from sampling.

Confidence intervals for this plot (based on sampling variance) would be great to address this.

Thank you for your thoughtful response. We completely agree that a deeper dive into the relationships between this paper and related SMC work is very necessary.

 

Here, we elaborate on works our reviewers have mentioned:

[18] - Syntactic And Semantic Control Of Large language Models Via Sequential Monte Carlo

[8] - Step-by-step Reasoning For Math Problems Via Twisted Sequential Monte Carlo

[33] - Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo

 

We’d like to begin by acknowledging that particle-based Monte Carlo methods are a foundational tool with deep roots in probabilistic inference, and that many recent papers—including ours, [18], [8], and [33]—build on this shared framework. No single work lays exclusive claim to these tools; rather, each adapts them to its own application domain and practical constraints. The differences across works often stem from how the SMC procedure is instantiated for a particular task.

 

[18] - This work explores token-level SMC within probabilistic programs, targeting constrained generation. While our work shares some high-level conceptual parallels, its goals are quite different. We focus on step-level particle filtering for inference-time scaling, emphasizing robustness to noisy, imperfect reward models—especially when applying LLMs to diverse reasoning problems.

[33] - This work introduces the Twisted SMC (TSMC) framework, showing how to learn intermediate twist functions to approximate target distributions that are defined solely via terminal rewards. This paper presents a general and theoretically rich foundation, proposing the contrastive twist learning (CTL) approach to train twist functions in language modeling scenarios. Its contributions lie largely in the development of this theoretical framework.

[8] - This follow-up to [33] applies the TSMC framework to multi-step symbolic reasoning in math problem-solving. [8] adapts TSMC to the math domain by:

• Training a separate proposal model (generator) to sample high-quality candidate trajectories;
• Using a ground-truth verifier to score correctness of full solutions;
• Learning a twist function via CTL that approximates expected future reward, trained on those samples.

They show that TSMC outperforms beam search and other decoding strategies in this setting. Since [8] operates in the same domain as our paper—mathematical reasoning—we focus our detailed comparison on this work going forward, referring to it as “TSMC.”

---

 

Our work and TSMC both leverage sequential resampling and particle-based sampling, but differ meaningfully in motivation, assumptions, and implementation. While both use particle methods, Twisted SMC and standard particle filtering are distinct algorithms, and we summarize the core differences below:

 

1. Different Motivations and Assumptions

TSMC seeks to improve sampling efficiency by learning an optimal twist function—a modification of the target distribution that ideally reduces the variance of importance weights. To learn this twist, [8] and [33] train a value function using ground-truth supervision. (They also show that using this value function outperforms off-the-shelf, frozen PRMs.) This is in strict contrast to our work that assumes a fixed, imperfect process reward model (PRM) and explores how to use it most effectively at inference time—without training any additional components. This distinction is important: we frame inference-time scaling as posterior inference under noisy supervision, not as amortized variational inference over a reweighted posterior.

 

2. Value Function vs PRM

TSMC aims to train an optimal twist function, which is formally a value function proportional to the expected final reward conditioned on a partial trajectory. While both value functions and PRMs can provide step-level or token-level signals, they are different theoretical objects, serving different purposes and trained under different objectives. The value function used in TSMC is specifically trained to approximate expected future reward, which enables it to reweight trajectory distributions toward those that lead to correct final outputs. In contrast, our work uses fixed PRMs that do not always capture expected future correctness and our method does not rely on supervision or make assumptions about their optimality (PRMs output correctness of a single step). Instead, we treat PRMs as potentially noisy and imperfect, and the particle filtering method is explicitly designed to be robust in such settings.

 

3. Training-Free vs Learned Components (continued in second response)

 

4. Practical Use Cases (continued in second response)

 

Where [8] and our Particle Filtering work meet: (continued in second response)

Thank you for your thoughtful and constructive review—we’re grateful for your recognition of the paper’s theoretical grounding, strong empirical results, and robust ablations across hyperparameters.

---

Runtime Exploration: Thank you for the thoughtful comment. We ran wall-clock time comparisons on a 50-question subset of MATH500 using Qwen2.5-Math-1.5B-Instruct and vLLM as the inference engine:

• BoN (budget=32): 211.69 sec

• BoN (budget=64): 392.14 sec

• Beam Search (budget=32): 549.48 sec

• Particle Filtering (budget=32): 401.63 sec

We find that PF at budget 32 takes comparable time to BoN at budget 64, while achieving higher accuracy (84.6 vs. 82.8), indicating a favorable compute–accuracy tradeoff.

Regarding FLOPs: in theory, PF, BoN, and Beam Search should have similar cost under perfect KV-cache reuse (where past token KV-caches are fully reused in decoding). However, in practice, step-wise inference leads to low KV cache hit rates because inference engines like vLLM are not optimized for token-by-token generation. This results in duplicated compute and increased latency. Importantly, these differences are driven more by current engineering constraints than by the underlying algorithm.

(As further evidence of how much these engineering implementation details can dominate: running DVTS, a better version of beam search, took days.)

We will include these latency results and a discussion of the practical vs. theoretical tradeoffs in the revised version.

Related Work: Thank you for pointing this out and for highlighting the relevant prior work. We’ll revise the conclusion to better reflect this and adjust our claims accordingly.

Stopping Criterion: Thank you for the question. For all methods, including particle filtering and the baselines, we used a consistent stopping criterion: each generation runs until the $< EOS >$ token is produced or the model hits a predefined maximum length. This ensures that completions are full solutions and not prematurely truncated. We will clarify this detail in the revised version.

---

Thank you again for your thoughtful comments. We hope the above information further solidifies your support of this work.

Thank you for your thoughtful review. We address your comments below.

 

Comparison to Other Works: Thank you for the thoughtful comment.

• We agree that particle-based Monte Carlo methods are a shared foundation across multiple works and note that particle filtering is a decades-old concept with long-standing use in probabilistic inference. As such, none of these papers - including ours or [18] - are unique in applying such techniques to language models.

 

• Despite their shared theoretical inspirations, the goals of the two works are quite distinct.

  • [18] focuses on token-level SMC within probabilistic programs for constrained generation.

  • Our work develops step-level particle filtering for inference-time scaling, designed to be robust to noisy and imperfect reward functions when scaling LLM inference across diverse reasoning tasks.

 

• Because [18] addresses a different setting and is not directly applicable to our evaluation domains (e.g., MATH, AIME), we did not include it as a baseline. We instead compare to and outperform TSMC for Math [8], which also uses SMC for LLM reasoning and is closer in scope to our work. We exceed [8]'s performance by 14.6 points using half the budget on the same dataset (MATH500) and model (DeepSeekMath7B), and we do so without needing fine-tuning or ground-truth verifiers.

 

• We also implemented the MCTS baseline from Zhang et al. and evaluated it against PF using the same base model (Llama 3.1 8B Instruct) and compute budget (32 rollouts). On the AIME 2024 benchmark, PF achieved 16.6% accuracy, significantly outperforming MCTS, which achieved only 6.6%. We will include this empirical comparison and accompanying discussion in the revised version.

 

---

Updating $w_t$ multiplicatively

Thank you for raising this point — you're absolutely right that the choice to multiply weights across steps deserves clarification.

• We use a multiplicative update so that each particle’s weight reflects the quality of the entire partial trajectory, not just the most recent step. This aligns with standard particle filtering practice, where the total weight captures cumulative likelihood.

• That said, we acknowledge that multiplicative updates could potentially compound noise, so we explore alternative reward aggregation schemes (e.g., using only the final step score, minimum score, etc.) as ablations in Appendix A and Section 4.5 (see Lines 131–135) and show that these offer different trade-offs between stability and expressivity.

We’ll make this connection more explicit in the main text.

 

---

Other Details

• $w_t$ is initialized to $1.0$:

  • Yes, each particle’s initial weight w_t is initialized to 1.0. We will make this explicit in the revised algorithm description.

• Proportional rather than equality in $w_t \propto w_{t-1} \hat{r}$:

  • We use a proportionality symbol rather than equality to reflect that the weights are unnormalized at each step. The normalization is performed later during the resampling step via a softmax over all particle weights. We will clarify this in the text to avoid confusion.

• Sampling is without replacement:

  • Thank you for raising this point. To clarify, our resampling is performed with replacement, consistent with standard practice in particle filtering. This means that a particle at step $t$ can be selected multiple times and continued by multiple particles at step $t+1$. An example of this behavior is illustrated in our main method Figure 2 of our paper. We will make this explicit in the revision.

• Line 88 Bernoulli Distribution:

  • Thank you for the careful reading - you're absolutely right that the emission distribution as written is formally incorrect. It should be written as $p(o_t \mid c, x_{\leq t})$, since the emission probability depends explicitly on the newly sampled token $x_t$.
  • In our model, the reward function $r(c, x_t)$ directly determines the likelihood of observing $o_t$, and thus the correct conditioning set includes $x_t$. We’ll revise the notation to reflect this.
  • This kind of dependency is common in state space models, where although emissions are conditioned on past latent states, they also rely on the current state sampled at each time step.

• Subscript Formatting:

  • Thanks for catching this. We will fix both issues in the final version.

 

References: Zhang, D., et al. (2024). Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B.

---

Thank you again for your thoughtful review. We sincerely hope these clarifications and updates address your concerns and strengthen your confidence in the work.