L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning

We thank the reviewer for their thoughtful review and for appreciating the intuitions and interpretability efforts in our results. We address your questions below:

What I really appreciate are that this paper tries to interpret the results, and gives nice intuitions on page 6 and 8, 9 to explain their findings. Only used a single LLM.

We are glad you liked the analysis. Based on your feedback, we have also trained a larger 7B model (Deepseek-r1-distilled-7B). The results can be found here.

We see similar trends as 1.5B model: High controllability, strong performance compared to baselines. These results highlight the effectiveness and potential of LCPO to scale to larger model sizes. We will include more information about this in the main paper.

L1 is trained with LCPO. However, CoT is test time. So how train with LCPO? Test/Train??? Please explain more clearly.

To clarify, we train L1 with prompts specifying various token lengths. At test time, length control is achieved simply by specifying the length in the prompt. For exact prompt template you can check Line 149 and qualitative examples in Figure 19, 20.

Provide explanation why OOD generalization occurs.

We hypothesize that LCPO-trained models effectively learn the concept of length control independent of domain specificity. Recent studies (e.g., “SFT Memorizes, RL Generalizes” [1]) suggest RL with CoT training leads to better generalization in reasoning tasks compared to standard supervised finetuning, and further, our LCPO formulation that disentangles Length and Correctness rewards separately may have further led to strong generalization. That being said, providing a full explanation of why OOD generalization occurs would be interesting and valuable future work.

Violin plots in Figure 4 should be bigger. Very hard to read.

Thanks for the feedback! We will update the figures in the camera-ready.

Thank you again for your encouraging feedback and thoughtful suggestions; we are happy to clarify any further points you might have.

We thank the reviewer for their thoughtful review! We appreciate that you find our work interesting and timely, our results solid, and the analysis interesting. We address your questions below:

In LCPO, while the control instruction says “Think for_ ngold,i tokens”, the reward is calculated based on the total output length, rather than the thinking token length (i.e., text wrapped by <think>). The results in Fig 7 relieve my concern, but I still wonder why the reward is not based on comparing the exact thinking length with ngold,i . On the other way around, is the LCPO algorithm sensitive to the specific control instruction design?

We appreciate this insightful point. We chose total output length practically, as this often aligns better with real-world usage scenarios (the user owns the total cost and not just reasoning length). This choice performs well and is expected to work for different instruction designs (as evident from performance in L1-Exact and L1-Max settings). Nonetheless, exploring reward formulations explicitly targeting thinking tokens could indeed be an interesting variation.

Concern about the generalizability of LCPO's length control: The two L1 models were said to be trained under a 4k-token context and then evaluated under an 8k-token context. However, the results presented in Figs 2 and 3 do not include any data points where the L1 models used more than 4k tokens. It seems to imply that, because the L1 models were trained to generate at most 4k tokens, they cannot follow the test-time control for using more than 4k tokens.

Indeed, our experiments confirm LCPO-trained models do not readily generalize to token lengths beyond those seen during training. Exploring strategies to improve length generalization is a promising direction for future research. We will add this discussion to the camera-ready.

A few language corrections:

Thanks for pointing them out! We will fix them in the revision.

What are the values of α in Eqns 1 and 2? In Eqn 2, the range of 0-1 in the clip function makes me wonder if the amount of ngold−ny is normalized in practice (e.g., (ngold−ny)/ngold

We use α = 0.0003 in both equations. The term (ngold - ny) is not normalized; instead, we empirically chose α to balance correctness and length adherence without explicit normalization. We will clarify this further in the updated version.

Why does s1 show a downward trend on GPQA and a flat curve on LSAT?

We note that both datasets are difficult for 1.5B model. Further, S1 often truncates the reasoning in the middle. Given the low performance (close to random guessing), we note that for S1 on both GPQA and LSAT difference in trends is not statistically significant.

What is the experimental procedure of Table 1? Why are there three sets of L1 results on the same dataset? Is the "#Tokens" specified in the control instruction?

Our goal is to compare our models against state-of-the-art models at comparable token lengths. We evaluated multiple (3) baseline models, noted their token lengths, and for each length, we prompted the L1-Exact model to generate with the same token length. Therefore, L1 is listed three times. We will clarify this better in the revised paper.

Thanks again for your thoughtful review and feedback. We look forward to addressing any other concerns you may have!

We thank the reviewer for their thoughtful review and positive feedback regarding the simplicity and effectiveness of LCPO, and our extensive experimental evaluation. We are also glad that the reviewer found no major problems. We address the reviewer’s minor concerns below.

There are repetations in Sec. 4, (Models and Datasets & Evaluation and Implementation Details.)

Thank you for pointing this out. We will update the section to eliminate redundancies.

Details are missing. How do you implement s1 on DeepScaleR?

S1 is a method that forces the reasoning to end by inserting a special </think> token once the maximum tokens is reached and then prompts with “Final Answer.” The exact code used can be found https://anonymous.4open.science/r/L1-COLM-7832/scripts/baselines/eval_s1.py. We follow exactly the same procedure as described by S1’s authors and will explicitly detail this implementation in the camera-ready.

I don't understand Table 1. Why are L1-Exact and L1-Max listed 3 times with different results? Seems they are different runs with different length control. But there are no further demonstrations, and I think it would be more proper to pre-determine the length than pick results afterwards.

Our goal is to compare our models against state-of-the-art models at comparable token lengths. We evaluated multiple (3) baseline models, noted their token lengths, and for each length we prompted the L1-Exact model to generate with the same token length. Therefore L1-Exact and L1-Max are listed three times. We will clarify this better in the revised paper.

How would LCPO compare with SFT/DPO methods with similar length control? Like [1] did?

SFT/DPO, as used in previous works [1], would be ineffective for length control in reasoning models. This is because they rely on generating seed data containing outputs of different token lengths, and relabelling them based on the actual length of the generated response. However, as we show in Table 5 (Appendix B), without training, reasoning models do not follow the length constraint and have a narrow range of output lengths. As a result, offline methods such as SFT/DPO would be ineffective. In contrast, because LCPO rewards are based on closeness to stated length, the narrow distribution of length gradually increases, until following with high precision. Here is the Table 5 for reference:

Thank you again for your insightful comments; please let us know if there are further points we can clarify.

We thank the reviewer for their thoughtful review and appreciation of the importance and effectiveness of LCPO. We particularly appreciate your recognition of our comprehensive experimental setup and clarity in writing. We are delighted you found no major weaknesses.