Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization

Thank you very much for your thorough review and valuable feedback! We greatly appreciate your insightful suggestions and positive assessment!

The authors demonstrate (specifically in Figure 2 Column 3) that the voting budget can adaptively change w.r.t. the difficulty level of the problem. Apart from the token budget, I am wondering if the authors could also offer accuracy on different difficulty levels. Now there is only overall accuracy in Table 3.

Absolutely. We break down the overall accuracies reported in Table 3 by difficulty level in the table above (numbers in parentheses indicate the number of problems solved). Some observations are:

• With larger budgets, we observe improved performance on the harder levels (3 and 4), as more voting budget becomes available.

• It is surprising that level 5 performance remains the same across all budgets. We conjecture that this is due to the limitations of the 8B model, which may only solve a subset of the hardest problems.

The design choices of the dataset construction in Section 4 (Appendix B) need more justification. For example, why at most "8 trials" and early stop if an answer appears "3 times"? Would the design choices have a significant impact on the final performance (sensitivity)?

In general, these choices do impact the performance of sequential voting (SV) itself. These choices were made to ensure that SV performs comparably to majority voting (MV), a baseline that is more familiar to the community.

• Reason of chocies:

  • In our observations, SV without early stopping tends to underperform compared to parallel MV. With the current setup, SV achieves performance similar to MV (Figures 2a/2b), which allows for a meaningful comparison between our ASV-IuB-$q$ models and MV.

  • We set the trial cap at 8 to avoid unbounded responses. This limit, combined with early stopping, empirically matched MV performance, so we kept it.

Since our key insights lie in the RL component, we found that aligning SV with parallel MV is a reasonable choice, as it allows for: (i) a sufficient reward margin for optimization, and (ii) a fair comparison to a community-standard baseline (MV).

Once again, we gratefully thank the reviewer for the insightful suggestions and positive evaluation! We hope our clarifications have adequately addressed your questions. We sincerely appreciate your support and will carefully revise the manuscript to further improve its clarity!

We sincerely appreciate your time in assessing our paper and your thoughtful feedback.

We’d like to first clarify a possible misunderstanding regarding the role of sequential voting (SV): it is not intended as a significant contribution of this work.

Role of SV: It is a simple, constructive alternative to o1/R1-style long CoT, with extended length and reasonable performance gains over conventional CoT.

• Why an alternative: As o1 is proprietary and R1 was released on Jan 20, ~10 days before the submission deadline.

• Why SV:

  • Simple: A minimal construction that allows us to focus on our RL contribution without the distraction of engineering long responses;
  • Characteristics: (i) gains over standard CoT; (ii) extended length.
    • SV resembles longCoT in these characteristics (though its gains over CoT are moderate), enabling us to demonstrate the effectiveness of our RL algorithm.

• What do we expect from SV?:

  • Superior accuracy?: No. As shown in Fig 2, SV performs comparably to parallel majority voting (MV). We expect advanced reasoning methods to outperform MV, and hence SV.

  • Practical value?: No. The main insights we convey do not lie in SV. After all, it is a dummy construction used to highlight our RL algorithm, which carries the core insights of this work.

We understand the concern that SV may not fully capture how o1/R1 works fundementally. Due to the unavailability of R1, we construct SV that at least resembles longCoT in terms of the aforementioned characteristics.

Given the considerations above, we believe our construction is a fair and reasonable choice for illustrative purposes.

To reiterate, our core contributions are:

• (i) casting adaptive reasoning as constrained inference;
• (ii) designing a constrained RL algorithm — which merits particular emphasis — that is simple and grounded.

In addition, the final accuracy stems from two orthogonal axes: (i) the reasoning axis; and (ii) the RL axis.

The derivation of our algorithm and the ''upstream'' metric (constraint satisfaction) are theoretically agnostic to the choice of long-response.

We intentionally kept the reasoning axis simple and did not attempt to boost performance through it, so as to isolate and highlight the RL component.

These points collectively suggest that a reasonable construction is sufficient for emphasizing our RL contribution.

We hope these discussions clarify our contributions. We believe our RL contribution—unanimously considered as sound and novel by other reviewers—merits recognition.

We quote reviewer toYu:

not only sheds new lights on adaptive reasoning but also provides a valuable algorithmic contribution to constrained RL as a whole.

For specific comments:

1. R1/O1 type models: We hope the above clarifies our choice of SV, given the unavailability of such models.

2. Trials as metric:

   1. • This choice was mainly motivated by Table 3. Since the self-correction works don't report token counts, we use the number of trials/turns as a proxy.

      • Besides, SV-SFT performs similarly to MV in trials (Figures 2a/2b); and scales comparably in tokens (Figure 5b, Appendix). Hence, trials serve as a reasonable proxy.

   2. Practical value considering inference time: This is an inherent limitation of any long-form reasoning method like longCoT, not a consequence of our RL algorithm. Substituting SV with O1/R1-style responses would result in similar inference-time overhead.

      After all, SV is a dummy alternative to longCoT, and was not intended to offer practical value beyond its illustrative purpose.

3. Table 3: As noted in Sec. 5.1, Table 3 is not intended to suggest our method outperforms self-correction. These are two orthogonal research directions: constrained inference and self-correction.

   Its purpose is simply to illustrate that constrained inference can achieve comparable performance to a well-established line of work. For this purpose, we believe transcribing their results is reasonable—it is convenient and avoids potential discrepancies from re-implementation.

   A more informative comparison is Fig. 2, where we evaluate against an efficiency boundary interpolating between two extremes of non-adaptive (homogeneous) cases. See our response to reviewer JCKH for details.

Again, we sincerely appreciate your suggestions and your assessment that our claims are supported, our theoretical and experimental designs have no issues, and our framework is rigorous.

We will carefully revise the manuscript to further improve clarity based on your suggestions, and we hope our responses have adequately addressed your questions.

Thank you for your insightful feedback!

We extend some discussion on accuracy to clarify our evaluation design, which also helps explain our choice of baseline later on. While important, accuracy depends on two orthogonal axes: (i) the RL axis; and (ii) the reasoning axis.

• RL axis: We're glad the reviewer agrees the adaptive allocation is ''theoretically expected'' and ''verified'';
• Reasoning axis: (A)SV was intended as a simple alternative to longCoT with reasonable accuracy, as R1 was unavailable until 10 days before the submission deadline. Empirically, (A)SV performs comparably to SC (Fig 2), enabling:
  • a sufficient margin for optimization;
  • a fair comparison w/ SC, a baseline more familiar to the community.

(SC is used here to match the reviewer’s terminology, though we use MV in paper.)

One key observation is: (i) SV is an all-voting case (can be seen as $q_+ = 1$); (ii) ours are adaptive with voting capped by $q_+$. This suggests, roughly speaking, that ours should be theoretically upper bounded by SV.

Since SV performs comparably to SC, we consider SC a reasonable baseline—based on the reasons above, not chosen ad hoc.

While further engineering on the reasoning axis (e.g. better alternative than SV) could improve overall accuracy, we refrained from over-optimizing it, as it's not the core insight of this work and could distract from the RL contributions.

• Baseline: Continuing from the discussion above

  • Interpolation baseline: We compare ours against a hypothetical efficiency boundary (HEB) (gray line Fig 2), defined by two non-adaptive extremes:

    • $q_+ = 0$: All short responses;
    • $q_+ = 1$: All long responses.

    A model above the HEB can be interpreted as having improved efficiency.

  • Why SC: SC was chosen because it closely aligns with the HEB (Fig 2). We mention SC in the abstract, as readers are likely more familiar with it.

  • [1, 2]: While [1, 2] are more recent, our goal is to evaluate against the HEB, which reflects non-adaptive cases. As we refrained from over-optimizing the reasoning axis, the upper bound (SV ≈ SC) is relatively low—but sufficient for testing our RL part—and is likely outperformed by [1, 2]. Thus, we don’t find a direct comparison with them entirely fair/necessary.

  We hope it clarifies our baseline choice and will revise accordingly, including additional discussion of [1, 2].

• Dataset: MATH500: (i) it has difficulty metadata; (ii) LLaMA 3.1 8B Inst. has moderate accuracy. All numbers hereafter refer to this model.

  Apart from metadata: Easier ones (GSM8K, SVAMP, ASDIV w/ pass@1 of ~85% [3,4,5]) have small reward margins, as most queries are solved in one attempt; For harder ones like AIME (2/30 solved [6]), efficiency makes little sense as the model rarely succeeds.

  We hope our grounded derivation and constraint satisfaction curves offer more insight and confidence than accuracy numbers alone.

• Model: We don't particularly find 8B ''small'', especially given the cost of online training. For reference, [7]—an online self-correction—uses 7B models. While we didn't explore other families, we're optimistic about the method's generalizability, thanks to its (i) model-agnostic derivation; (2) simple update (Eq. 5).

• 2x efficiency: Apologies for the confusion. It comes from Tab 3 and Fig 2. At $q_+ = 50$, ours has 55.4% (a 4.14% gain) w/ 2.16x trials (Tab 3), while the HEB (aligned with SC, if not better) in Fig 2 requires ~4.5x trials for similar accuracy—implying ~2x efficiency.

• Table 3: We didn't re-run; the numbers were taken from the cited papers to avoid implementation discrepancies.

• Online training is inherently expensive: For 70B, under our setup, we estimate that a single epoch of generation alone could take ~2,600 H100 hours at 300 output tokens/sec per node. In practice, training throughput is significantly lower due to memory overhead from model optimization—and this estimate excludes other potential costs. These limited us from exploring larger models and harder datasets (which likely require larger training sets).

We quote reviewer toYu:

not only sheds new lights on adaptive reasoning but also provides a valuable algorithmic contribution to constrained RL as a whole.

We are not aware of RL-for-LLM works that explicitly impose linear constraints to control response distributions.

We believe our RL contribution merits stronger recognition—especially given your positive evaluation of it.

Again, we appreciate your valuable suggestions! We will revise accordingly and we hope our responses addressed your questions.

[3] arxiv.org/abs/2407.21783

[4] 2410.06638

[5] 2502.12134

[6] 2410.01560

[7] 2407.18219

Thank you very much for your thorough review and valuable feedback! We greatly appreciate your insightful suggestions and positive assessment!

We address these three comments collectively, given their relevance to one another.

However, I'm still hesitant to accept that responses constructed this way are actually long/short-context responses -- the long ones look like a mere aggregation of short responses.

One major confusion is whether prompts generated via SV sufficiently represent the "unnecessarily tedious long reasoning trace" that the paper sets out to improve. This is rather important as SV seems to be the only experiment scenario.

On the paper structure, maybe it's a better idea to include more explanation on prompt generation (the proposed SV, ASV), not only to clarify the experiment setting more, but to also convince readers that SV is adequate in representing long response situations that the paper initially tries to improve.

We understand the concern that SV may not fully capture how o1/R1 works at a fundamental level. This construction was chosen due to the unavailability of R1, which was released on Jan 20—roughly 10 days before the submission deadline.

Reproducing o1/R1-style longCoT, or anything more similar, was beyond the scope of our focus on constrained LLM inference. Moreover, it was unclear to the community how to reproduce such methods until the R1 technical report [1] was released on Jan 22.

The SV construction resembles two key characteristics of longCoT: (i) improved accuracy, and (ii) extended response length. It therefore serves an illustrative purpose to highlight our core RL contribution.

Additionally, our paper (i) casts the problem in terms of distribution constraints, rather than directly optimizing over the long-response group, and (ii) derives an RL method that is theoretically agnostic to the specific type of long response. These efforts together further support the use of a constructed alternative.

Given these considerations—and the fact that o1/R1-style longCoT was not available at the time—we hope you find the use of this construction reasonable.

We completely agree that devoting more space to explaining the design decisions and clarifying the role of SV would significantly improve the paper's clarity. Thank you for pointing this out—we will revise the manuscript accordingly!

[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. https://arxiv.org/abs/2501.12948

revealing effectiveness of the proposed Sequential Voting and IBPO optimization paired with Adaptive Sequential Voting, although improvements seem a little marginal.

One potential weakness is that many of the baseline results are not run but transcribed from others' works. This harms the solidity of the experiment comparisons.

We believe these are relatively minor points, if our understanding is correct. As such, we've kept our response concise to avoid unnecessary distraction. (Further details can be found in our response to Point 3 for Reviewer hh45.)

After all, our work on constrained LLM inference and the self-correction literature are essentially two orthogonal directions. Transcribing their results is both convenient and helps avoid potential implementation discrepancies.

Once again, we sincerely appreciate the reviewer’s insightful feedback and encouraging assessment! We hope our clarifications sufficiently address your questions and clarify our choice of construction. We deeply appreciate your support and will carefully revise the manuscript to further improve its clarity!