Self-Steering Language Models

Thank you very much for your positive review and recommendation for acceptance! We’re happy to hear that you found the experimental design choices reasonable to show the effectiveness of DisCIPL and that the empirical results seem promising.

We appreciate the helpful feedback — in response to your suggestions, we’ve added experiments scaling up New Follower Models as well as a detailed Inference Cost Comparison.

Please see responses to the points you raised below:

How would the method scale to larger LMs, e.g. >= 7B?

We’ve added a range of new Follower models to our evaluations. These include scaling up Followers from the Llama-3.x family (to 3B and 8B, respectively), which we find improves DisCIPL’s performance. Please see the section on New Follower Models for more details.

How much slower, if any, is the method compared to o1?

DisCIPL inference takes from a few seconds to several minutes, depending on the exact inference program, which is similar to o1’s inference speeds on our tasks. Nevertheless, in terms of other key metrics of inference cost, we find that DisCIPL programs are 40.1% shorter than o1 reasoning traces (in tokens) and are generated by a smaller model, meaning that DisCIPL is also less costly. (The dollar cost of running the inference program with a small Follower turns out to be negligible.) Please see our new Inference Cost Comparison for more details.

SMC requires sampling token-by-token; does this introduce overhead?

Thanks for this good question; there is some overhead introduced by SMC, but the reason this arises requires a bit of unpacking. First, note that standard autoregressive generation also involves sampling token-by-token from the LLM. However, in a standard (synchronous) decoding setting, each new token is a continuation of the same prompt prefix. In contrast, in our (asynchronous) decoding setting, we have multiple parallel processes issuing queries to the LLM, each of which might have a different prompt prefix. In a naive implementation, loading the different contexts and associated activations in and out of GPU memory introduces significant overhead. Fortunately, some really great tools have been developed recently to improve efficiency of serving LLMs in exactly this async setting. In our implementation, we use vLLM as a backend, which implements features like KV caching and continuous batching that significantly improve inference efficiency. In practice, we find that vLLM results in a 2-5x speedup over the standard HuggingFace Transformers backend, making it possible to run SMC with many particles efficiently on a single A100/H100 GPU.

It might be interesting explore generalization to math reasoning tasks (e.g., Game of 24)

We definitely agree and this direction (including Game of 24) is something that we had previously discussed among the authors. In brief, we agree that extending to math reasoning could help to make the case for the generality of the method. However, current approaches to math reasoning make certain assumptions (e.g., use of pre-trained reward models for tasks like MATH500 and AIME) that make it so that there is a single, “one-size-fits-all” inference strategy (i.e., calling the reward model periodically during generation, or just calling it once at the end to score completions). This is in contrast to the domains we look at, where there is meaningful variation in the inference programs on a task-by-task basis and the Planner really has to generalize (as opposed to just calling an off-the-shelf verifier). For a more in-depth discussion of some of these considerations, please see our response in Author responses to key issues.

Thanks again for the time spent in reviewing our submission and for your recommendation of acceptance — it means a lot to us! Please feel free to let us know if there’s anything further that you think we might be able to clarify for you or the other reviewers in the remainder of the rebuttal period.

Thanks for the thoughtful review! We’ve added a number of additional experiments to the paper based on your suggestions, including experiments with New Follower Models as well as a new Inference Cost Analysis. Please see responses below.

Only a weak follower model (llama3.2-1b) has been tested. Many other models could be considered at no extra cost.

Thanks for your helpful and specific suggestions. We’ve added a range of new Follower models to our evaluations. These include scaling up Followers from the Llama-3.x family (to 3B and 8B, respectively) as well as new architectures, including your suggestion to use the Qwen3 architecture. Please see the section on New Follower Models for more details.

As you predicted, we found some evidence that Qwen-3-1.7B is stronger than Llama-3.2-1B, both when evaluated by itself and also as part of DisCIPL.

Regarding the other architectures you suggested:

• Gemma 3. We also looked at extending these evaluations to include Gemma-3-1B as Follower. We ran into a small snag with vLLM compatibility (specifically pertaining to Gemma-3’s sliding window attention) that will require more time to address than we currently have due to COLM rebuttals. Nevertheless, we are excited about the possibility of including this architecture in our paper. Thanks again for the pointer to this model.
• Gemini Flash / GPT-4.1 / GPT 4.1-nano. Thanks for these suggestions. One challenge we face with API models is that our method requires access to the Follower model’s logprobs in order to compute importance weight updates (this is described in S3.1 of the paper under “General Framework”). Additionally, to speed up inference, we use vLLM as a backend for KV-caching, which also requires the Follower to be run locally. That said, note that in principle DisCIPL-RS could be run without logprobs, and thus could be instantiated with GPT 4.1-nano or other API models as Followers. In our initial experiments, we found DisCIPL-RS to be weaker than DisCIPL-SMC, but some findings from our New Follower Models experiments suggest that for stronger Follower models, DisCIPL-RS can be competitive.

The inference cost comparison is not clear.

We appreciate the feedback; this was also something that other reviewers noted. Please see our new Inference Cost Comparison section. We have refined our analysis to provide a clear apples-to-apples dollar cost comparison using current API pricing — we hope this helps to address your questions.

Constrained generation tasks are artificial; showing more convincing use-cases seems important.

Thanks for this feedback. We definitely believe our method could also be extended to other domains (e.g., math reasoning) and that this could strengthen the narrative. Given the limited time in the rebuttal period, we’ve chosen to focus for now on addressing empirical questions raised by reviewers that can be productively addressed within the scope of the existing domains. Please see our responses in Author responses to key issues for more discussion on this question.

Thank you again for your helpful suggestions regarding the Follower model experiments and inference costs comparisons! Please don’t hesitate to let us know if you have any follow-up questions that arise from looking at these new results. We’re grateful for your time and consideration.

Except for more convincing benchmarks, all my concerns and questions were answered. I will raise my score accordingly and would have raised even more given more benchmarks. Maybe something like SWE bench could be considered. I would like to point out that I am generally not convinced by arguments claiming that other papers use the same standard of evaluation quality. This goes against progress and only sets your work as part of theses lower standard of benchmarking instead of raising it to a convincing level of scientific evidence. That being said I appreciate the extensive set of new experiments that the authors brought to the rebuttal.

Dear Reviewer,

As we near the end of the discussion period, we wanted to make sure you had a chance to take a look at the rebuttal we posted earlier this week.

We ran several new experiments that specifically address your comments. We hope you will consider these in your final evaluation, and we’d be grateful for any further feedback or follow-up questions you might have.

Thank you again for your time and consideration.

Sincerely,

The Authors

Thanks for your insightful review! In response to your suggestions, we’ve added several new experiments, including additional Planner Baselines: CoT and PoT as well as a new Inference Cost Analysis.

The setup closely resembles Program-of-Thought (PoT) style methods; can you add a baseline?

Thank you for the suggestion! Your point is well taken that if we assume access to a strong Planner model (e.g., GPT-4o) then there are other ways of using that model besides the existing Planner-only baselines, which perform direct generation.

In response to your feedback, we explored two different approaches that utilize the Planner in more computationally-intensive ways:

• Planner Chain-of-Thought (CoT)
• Planner Program-of-Thought (PoT)

We provide a detailed writeup of these experiments in a separate post: Planner Baselines: CoT and PoT.

Regarding PoT specifically, one of our co-authors spent quite a bit of time doing prompt engineering to try to get PoT to work for our domains; you can read the writeup from this exploration here.

Overall, we found that it’s quite difficult to generate vanilla Python programs to solve the kinds of tasks in our evaluation: Whereas the original PoT paper was designed for numerical reasoning tasks, our work is focused on text generation. (This is why our programs are designed to utilize Follower LMs, which are good at generating text). For this reason, and due to time limitations, we ended up not running full experiments with PoT; nevertheless, if after reading you have suggestions for overcoming these hurdles, we would be happy to continue to pursue this direction.

If such a strong model is already available, it's unclear why weaker models are used in subsequent stages.

Good question — we show that there are multiple reasons why we would want to offload computation to Follower LMs.

• Performance: We find that the Planner alone actually doesn't perform that well on many tasks (see Planner-only baselines, including the new CoT baseline above). For instance, on COLLIE sentence tasks, GPT-4o only gets 24% Pass@1, compared to 81-87% for DisCIPL. This highlights a key point of this paper: We can use a relatively strong (but still unreliable) LM as a Planner and far surpass its performance with our method.
• Scalability: We can efficiently run inference with many Followers, whereas scaling up the Planner LM is orders of magnitude more expensive. Specifically, the marginal cost of doubling inference-time compute in DisCIPL w/ Llama-3.2-1B is 100x cheaper than GPT-4o and 10000x cheaper than o1. Even when we factor in the cost of generating inference programs with a GPT-4o Planner, we find that our approach is still less expensive than o1. (Please see our section on Inference Cost Analysis for more details.)

The benchmark is already solved by o1.

Our primary argument here is about efficiency, not absolute performance. Please see the section, “What is the role of o1 in the experiments?” in Author responses to key issues.

The approach may not generalize well to other types of planning problems.

While we’re not sure what problems you might have in mind, we believe our approach could generalize to many kinds of planning problems that have a natural step-by-step decomposition. In particular, we think the ideas from DisCIPL might apply to agentic tasks like web navigation; please see our response to Reviewer DTJV where we explored this idea.

SMC seems to be not necessary here. Can we use async probabilistic programming with standard parallel repeating generation?

Yes! This condition is already included in our paper as DisCIPL-RS (see Lines 176-179 for a description and Table 1 / Figs. 3, 5, 7, 8 for results). Note that this variant of our method uses the same probabilistic programs to perform standard parallel repeating generation (aka rejection sampling) as you described. We find that this approach consistently underperforms SMC by a margin of 9%-70% (absolute percentage points) across domains.

We hope these responses answered your questions. Thanks for your time and effort in engaging with our submission. Please don’t hesitate to follow up if there is anything further we might clarify. Thank you for your consideration!

Dear Reviewer,

As we near the end of the discussion period, we wanted to make sure you had a chance to take a look at the rebuttal we posted earlier this week.

We ran several new experiments that specifically address your comments. We hope you will consider these in your final evaluation, and we’d be grateful for any further feedback or follow-up questions you might have.

Thank you again for your time and consideration.

Sincerely,

The Authors

Dear Reviewer 3m39,

As the discussion period concludes today, we wanted to follow up to see if you’ve had a chance to review the updates on the thread. We greatly appreciate your detailed review and suggestions, which have guided several new additions, including:

• Planner Baselines: CoT and PoT

• Inference Cost Analysis

We're grateful for the constructive engagement from all reviewers—two have updated their scores, and all three have now recommended acceptance.

If you have time today, we’d be grateful to hear whether these updates address your concerns. We're also happy to provide any further clarification or answer any remaining questions that you may have.

Sincerely,

The Self-Steering Authors

Thank you for this very insightful review. In response to your feedback, we’ve added a number of new experiments, including New Follower Experiments, New Planner-only Baselines, and a new Inference Cost Analysis.

Additional experiments involving stronger Planners and Followers could clarify the method’s scalability.

Thank you very much for the suggestion. We agree this is an interesting direction, particularly the question of how performance scales with larger Follower LMs. In response to your suggestion, we ran several New Follower Experiments.

Briefly, we find that DisCIPL scales with Follower size (+0.09 Pass@1 with 8B vs. 1B Llama) and benefits from stronger small LMs like Qwen3-1.7B (+0.07 Pass@1 over 1B Llama). We also observe some other interesting scaling trends regarding the validity/coherency tradeoff.

Regarding the Planner, we also ran some new experiments with stronger variants of the GPT-4o Planner-only baselines (see here).

We also agree with your comment that scaling down the Planner within the context of DisCIPL would be an interesting direction to explore (though it seems that not all the reviewers would be in agreement on this direction). In particular, while we didn’t have time in the current rebuttal push to run experiments with weaker Planners, we think it would be interesting if there were some intermediate-sized LLM (e.g., one of the DeepSeek-R1 distills) that were capable of acting as both Planner and Follower. We leave this for future work!

A detailed breakdown of compute cost would help practitioners judge real‑world trade‑offs.

Thanks as well for this suggestion; we definitely agree. We've added a detailed Inference Cost Analysis, which we think will be of interest. Please don’t hesitate if you have any further questions.

Outperforming o1 seems to be another necessary milestone

Please see the section, “What is the role of o1 in the experiments?” in Author responses.

The proposed method essentially tries to inject more explicit priors to guide small LLMs, which seems not generalizable (and also powerful) as o1

We’re not sure we agree with this characterization; this paper is about a general framework that decomposes reasoning into a two-step process with Planner LMs and Follower LMs. We emphasize that these roles can be played by arbitrary models, and that we could, in principle, use o1 either as a Planner or Follower. While certain resource limitations prevent us from doing so at the moment (see Author responses for a discussion), we would love to see a group with the relevant resources explore instantiations of DisCIPL that use large-scale reasoning models like o1 in both Planner and Follower roles.

The system prompt is missing from A.10.

Thanks so much for spotting this issue — we agree this is an important piece of context for reviewers and we sincerely apologize for the inconvenience.

(By way of explanation, the system prompt is written in Markdown and we were using a package to automatically compile it to LaTex; it appears that the final compilation of the submission PDF caused an error at the last minute that we didn’t catch.)

While authors are prevented from updating the PDF during COLM rebuttal, we’ve uploaded the full System Prompt for your reference.

Dear Reviewer,

As we near the end of the discussion period, we wanted to make sure you had a chance to take a look at the rebuttal we posted earlier this week.

We ran several new experiments that specifically address your comments. We hope you will consider these in your final evaluation, and we’d be grateful for any further feedback or follow-up questions you might have.

Thank you again for your time and consideration.

Sincerely,

The Authors

Evaluation scope is too narrow / datasets are small

Our evaluations are similar in scope to evaluations presented in other recent works on novel LLM inference methods (e.g., Tree-of-Thought [1], Stream-of-Search [2]). In particular, [2] (which was an oral spotlight at COLM 2024) is a good example of a paper that introduced a novel LLM inference method and only evaluated on a single domain (Game of 24 / Countdown). In comparison, some of our PUZZLES tasks (e.g., budgeting, trip itinerary planning) are more representative of “real world” tasks than evaluations in prior work (e.g., simple number games, crosswords). This is not to say that we’re not interested in exploring generalization (see below), but rather that we believe the current evaluation scope meets the standard for published papers in this space while enabling us to study the properties of our method in-depth.

Can this method generalize to other tasks; e.g., math reasoning?

Some tasks in the PUZZLES domain do have quantitative reasoning components while still being, essentially, constrained generation tasks (e.g., budgeting). That said, we do believe our architecture (in which a Planner LM generates a plan, which is executed by Follower LMs) could be applied to math reasoning problems. In fact, several recent papers ([3, 4], and cited in our Related Work) have shown that sequential Monte Carlo (SMC) can be used to boost performance on domains such as MATH500 and AIME. These methods typically work by scoring in-progress reasoning chains using off-the-shelf process reward models (PRMs, [5]); it would be interesting to explore whether, (a) DisCIPL Planners can write inference programs that effectively exploit such reward models, if given access to them; or (b) DisCIPL could generate plan programs that boost math reasoning ability without using special-purpose PRMs, e.g. by generating LM-as-judge prompts specialized to the problem at hand. We believe these are interesting directions for future work.

NOTE: OpenReview does not allow file uploads; key figures and supplemental data from these experiments are available at the following anonymous link: self-steering.notion.site

Scaling up the Follower

Several reviewers (DTJV, WVvh, DG1U) commented that they would like to see evaluations of a broader range of Follower sizes to better characterize scaling effects. Accordingly, we re-ran our COLLIE experiments with multiple larger Llama-3.x models to produce 3-point scaling curves:

• 1 Billion: Llama-3.2-1B-Instruct
• 2 Billion: Llama-3.2-3B-Instruct
• 8 Billion: Llama-3.1-8B-Instruct

Key Findings

• Pass@1 scales with Follower size: As expected, both for DisCIPL and Follower-only baselines, Pass@1 improves with Follower model size. However, for COLLIE sentence tasks, Follower-only performance remains much lower than DisCIPL (delta Pass@1 = 0.63), even at the 8B size.

• Coherency improves with Follower size, but only for DisCIPL: Notably, on COLLIE sentence tasks, Coherency decreases with additional scale in the Follower-only baselines. (See the full results tables for Coherency numbers.)

  • Intuitively, the larger Llama models appear to be trying harder to satisfy the constraints, producing less natural text and therefore reduced Coherency scores.
  • In contrast, for DisCIPL-SMC/IS, we observe that Coherency and Pass@1 both increase with Follower size.

  We believe this scaling phenomenon illustrates a unique advantage of our method over standard sampling techniques.

COLLIE Sentence Tasks

COLLIE Paragraph Tasks

NOTE: OpenReview does not allow file uploads; key figures and supplemental data from these experiments are available at the following anonymous link: self-steering.notion.site

While our existing Planner-only baselines already evaluates GPT-4o’s performance in a direct generation setting, reviewers (e.g., 3m39) rightly point out that if we assume access to a strong Planner, there are other relevant baselines to compare to.

Here, we explore two additional baselines that utilize the Planner in more computationally-intensive ways:

• Planner Chain-of-Thought (CoT)
• Planner Program-of-Thought (PoT)

Planner Chain-of-Thought (CoT)

We modify our existing Planner-only system prompt to include standard instructions to think step-by-step.

**System Prompt**
You are helping a user generate text that satisfies constraints. Think step-by-step in order to follow the user's instructions. Your final response should be enclosed in <answer></answer> tags.

Quantitative findings

• On COLLIE Sentence-Level Tasks, we find that CoT provides a marginal boost over GPT-4o; however, overall Pass@1 is still substantially below DisCIPL.
• On COLLIE Paragraph-Level Tasks, we find that CoT does not perform standard sampling with GPT-4o.
  • Overall Pass@1 for both methods is comparable to DisCIPL with scaled-up Followers.
  • Notably, DisCIPL-RS (Llama-3.1-8B) still outperforms GPT-4o here (Pass@1=0.91).

COLLIE Sentence-Level Tasks

COLLIE Paragraph-Level Tasks

Qualitative findings

Qualitatively, we find that chain-of-thought does not enable GPT-4o to overcome existing issues with COLLIE tasks; for instance, counting the number of characters or positions of words.

Example #1: Character counting

**Task Prompt**
Please generate a sentence with exactly 82 characters. Include whitespace into your character count.

**GPT-4o (CoT)**
To create a sentence with exactly 82 characters, I'll count each letter, space, and punctuation mark:

"The sun sets slowly over the ocean, painting the sky with hues of orange and pink delight."

Now, let's count the characters:

1. "The sun sets slowly over the ocean, painting the sky with hues of orange" (70 characters)
2. " and pink delight." (12 characters)

Total: 70 + 12 = 82 characters.

<answer>The sun sets slowly over the ocean, painting the sky with hues of orange and pink delight.</answer>

**Evaluation: Invalid**
- Length is 90 characters (expected 82)

Example #2: Word Positioning

**Task Prompt**
Please generate a sentence:
1) with exactly 15 words;
2) with the 4th, 8th, 11th words to be 'collection', 'Noise', 'Testament' respectively.

**GPT-4o (CoT)**
To create a sentence that meets all the requirements, let's start by laying out the positions of the specific words:

1. (word 1)
2. (word 2)
3. (word 3)
4. collection
5. (word 5)
6. (word 6)
7. (word 7)
8. Noise
9. (word 9)
10. (word 10)
11. Testament
12. (word 12)
13. (word 13)
14. (word 14)
15. (word 15)

Now, fill in the blanks with appropriate words to construct a coherent sentence:

"The museum's vast collection included a fascinating exhibit titled Noise, featuring the Testament of artifacts."

**Evaluation: Invalid**
- word 8 is "exhibit" (expected "Noise")
- word 11 is "featuring" (expected "Testament")

NOTE: OpenReview does not allow file uploads; key figures and supplemental data from these experiments are available at the following anonymous link: self-steering.notion.site

Several reviewers (DTJV, WVvh, and DG1U) asked for more detailed comparisons of the computational cost of DisCIPL with that of our baselines.

For illustrative purposes, we present an in-depth study of inference costs for a representative domain (COLLIE sentence tasks). Our analysis suggests that this is a conservative choice; comparative efficiency advantages of DisCIPL are likely even greater on longer-form generation tasks such as COLLIE paragraph generation and PUZZLES. Detailed cost comparisons for all domains will be included in a subsequent revision.

Dollar cost analysis

We perform an analysis of the dollar cost of the DisCIPL-SMC method, the Follower-only, Planner-only, and o1 baselines from the submission, and the new Planner-only Chain-of-Thought baseline.

Key takeaways:

• In absolute dollar cost, DisCIPL is cheaper than o1: In total, running DisCIPL is marginally cheaper than o1 (\$4.26 vs. \\$4.75 per 100 tasks).
• DisCIPL’s inference programs are 40.1% shorter and 80.2% cheaper to generate than o1 reasoning traces. Programs generated by the Planner LM use 40.1% fewer tokens than reasoning traces generated by o1. In addition, the generated tokens are on average 80.2% cheaper, as they are generated by a less expensive model—GPT-4o. Overall, per 100 tasks, the tokens of generated inference programs cost \$0.91, whereas the tokens of o1 reasoning traces cost \\$4.59.
• The cost of scaling inference-time compute with DisCIPL is negligible: With 32 particles, the Follower LM’s generations account for <0.1% of the overall cost of DisCIPL. Thus, for example, doubling the test-time compute budget (e.g., from 32 to 64 particles) would have a negligible influence on the total cost of DisCIPL. By contrast, doubling the reasoning budget for o1 would nearly double its price, and similarly for Planner-only Chain-of-Thought.
• In the current implementation, DisCIPL’s cost is dominated by the long Planner system prompt: Because GPT-4o has not been trained on LLaMPPL inference programs, we use in-context learning with a detailed system prompt and few-shot examples. We find that 72% of the cost of DisCIPL results from pre-filling the part of this prompt that is static across tasks.
• Nevertheless, DisCIPL benefits significantly from prompt caching. On average, 95.6% of the Planner’s prompt tokens were cached by OpenAI’s servers, resulting in an automatic 50% discount. In a performance-optimized system, these costs could be further reduced by using a local Planner LM with prefix caching. Alternatively, we could finetune the Planner LM on inference programs—in the same way that current reasoning models are finetuned on reasoning traces—to avoid the need for expensive re-prompting. Amortizing the Planner computation in this way would yield significant practical cost savings, up to 72% of the overall cost.

Dollar Cost per 100 Tasks

*DisCIPL-SMC instantiated with a Llama-3.2-1B Follower, GPT-4o Planner, N=32 particles.

The columns in the above table are:

• Answer Generation: The cost of generating the tokens of the solution to the task. For example, in o1, answer generation tokens are those generated after reasoning has completed; for GPT-4o CoT, they are the tokens tagged by the model as constituting the final answer; and in DisCIPL, they are all tokens generated as candidate answers by the Follower LM.
• Reasoning / Program Generation: The cost of generating tokens instrumental to solving the task. For CoT and o1, these are reasoning tokens. For DisCIPL, these are the tokens of the inference program generated by the Planner LM.
• Task Prompt: The cost of pre-filling the task prompt, which describes the particular task being solved. For DisCIPL, this is the cost of explaining the task to the Follower, not to the Planner.
• Planner System Prompt: The cost of pre-filling the Planner’s system prompt in DisCIPL. Some of the prompt is task-specific and not cached, whereas most of the prompt is domain-general and cached across tasks.