We are very encouraged by your positive feedback and your recognizing the value of our proposed method! Below we address your concerns:

Please find our detailed responses to your specific questions below. For clarity, we use the following notations:

• W – Weakness
• Q - Question

W1: Would be great if the author could also release the implementation.

Certainly! We value the impact of open-sourcing and are committed to releasing our models, code, and data to benefit the community.

Q1: How to handle the format reward for the sequence without thinking?

Great question! Format specification is an important factor that influences model behavior. In our approach, we standardize both “thinking” and “non-thinking” modes into a unified format:

<think>reasoning process</think><answer>predicted answer</answer>

• For the non-thinking format, we use the structure <think>\n\n</think>, where the content inside <think> is replaced with \n\n.
• In contrast, the thinking format retains the structure <think>reasoning process</think>, where the content explicitly represents the model’s reasoning process.

This unified formulation enables supervised fine-tuning (SFT) to effectively incorporate thought dropout with minimal changes. Furthermore, it facilitates the transition to the GRPO stage by allowing reuse of the same format.

Q2: Why study this in the MLLM domain? Can we extend this in the LLM domain?

Thank you for your suggestion regarding extending our study to the LLM domain.

Why we choice MLLM as a primary testbed:

• We believe that vision tasks are more intuitive and can often be resolved at a glance. In contrast, text-based tasks typically provide more comprehensive or professional information. Visual tasks tend to be more redundant; often, a quick glance is sufficient to determine whether further thought is necessary, akin to human cognition. There is why we choice MLLM to study this problem.

Extension of TON to the LLM Domain: We appreciate your suggestion to extend our study to the LLM domain. In response, we have adapted our TON framework to the LLM benchmark GSM8K. Below, we present the corresponding experiments and results.

The findings indicate that TON significantly reduces response length while maintaining high accuracy. This demonstrates the generalizability of TON across different modalities.

We will include the experimental setup and incorporate further discussion on this extension in our revised submission.

Q3: For the reverse thinking, how to verify the correctness of the thinking process?

Good question! An intuitive approach to verifying the correctness of a thinking process is to prepend it as a prefix and observe whether it enables the base model to generate the correct answer. If the model succeeds under this condition, we can treat the thinking process as valid.

However, based on our investigation, the primary objective of the SFT stage is to instill format-following behavior—such as the ability to either generate a complete reasoning trace or omit it entirely. Our intention is not to intervene in the model’s behavior through external verification or enforcement. This is why we adopt a reverse thinking strategy derived from the base model itself, without introducing a more powerful auxiliary model.

We aim for the base model to self-explore during the GRPO stage, allowing us to observe the genuine gains achieved without relying on external guidance or supervision.

Q4: Why not use third-party (closed-source APIs) to generate the thinking process as the starting point?

Yes, this is certainly possible. One could use third-party closed-source APIs to generate the thinking process simply by replacing the SFT training corpus with externally generated reasoning traces.

However, our focus is on enabling the model to develop its own reasoning abilities without external assistance. By doing so, we ensure that any improvements observed during the GRPO stage stem from the model itself, rather than being distilled from a stronger external model. This setup allows for a clearer evaluation of the model's intrinsic capacity for reasoning and self-improvement.

We appreciate your questions and comments very much. Please let us know for any further questions.

Dear Reviewer bSRJ,

This is the last call for author-reviewer discussion, which will ends on Aug 8. Could you please read authors' rebuttal and confirm whether your concerns have been addressed asap? Thank you.

Best,

AC

Thank you for recognizing the value of our work and for providing constructive and thorough feedback. We value your insights and hope our response addresses your comments.

Please find our detailed responses to your specific questions below. For clarity, we use the following notations:

• W – Weakness
• Q - Question

W1: Rationale for Random Dropout of Reasoning Traces

Thank you for your comments on the complex reasoning case, we also value this most. We appreciate the opportunity to discuss it in greater depth.

1. Does adding the reasoning difficulty supervision make it better?

• By Answer Correctness: We tried a direct approach to classify each sample's difficulty based on whether the base VLM answers it correctly. We then drop thoughts for only the “easy” samples (those answered correctly by the base VLM). To investigate this, we implemented a difficulty-aware dropout strategy and compared its accuracy under our TON training framework.

Somewhat surprisingly, the accuracy of the difficulty-aware dropout version is similar to, and even slightly lower than, our proposed random dropout method. This outcome may be attributed to the challenge of precisely defining “task difficulty”. Human-defined heuristics introduce potential noise and risk interfering with model behavior.

Therefore, we adopt a conservative and principled approach: we treat all samples equally, allowing the model to acquire format-following skills during SFT and to self-explore and learn effectively during the GRPO stage. This strategy ensures that the model’s improvements are intrinsic and not confounded by external definitions or interventions.

2. Despite Random Dropout in SFT, the Model Adapts to Task Difficulty in the RL Stage

As shown in Supplementary Section A.1, even with random thought dropout during SFT, our TON framework enables the model to adapt its skip-thinking behavior during GRPO based on task difficulty and reasoning complexity. We analyze performance across a range of benchmarks—including 3D object counting, GUI-based mobile agent navigation, and geometric math reasoning—spanning a spectrum of task difficulties. Task difficulty is assessed both qualitatively (via intuitive reasoning complexity) and quantitatively (by base VLM zero-shot accuracy), grouped as:

• 60-100 (easy)
• 40-60 (medium)
• 0-40 (hard).

The results of the final skip-ratios and our TON accuracy are as follows:

Our findings indicate that our TON maintains a high skip ratio for easy counting queries, allowing it to bypass unnecessary thinking processes, while keeping a low skip ratio for harder, more professional, knowledge-intensive questions to fully utilize its thinking capabilities.

3. Possible future extension: We acknowledge that the skip-thinking mechanism may introduce the risk of the model bypassing necessary complex reasoning steps, and we plan to address it in future work. (i) A potential direction is to incorporate auxiliary scores such as visual perplexity (PPL) scores as a proxy for estimating task difficulty. (ii) Another possibility is to evaluate whether the model can answer a question correctly both with and without the reasoning trace. If so, the reasoning may be redundant and thus safely skippable.

W2: Would be better to provide quantitative insights into the skip ratios and query types.

Thank you for your valuable suggestion. To address this, we categorize query types based on their originating benchmarks. For example, we identify query types such as:

• Counting (e.g., "How many [objects]?")
• Mobile Agent (e.g., "Download a music app")
• Math Reasoning (e.g., "What is the degree of the angle?")

These query types naturally differ in their reasoning complexity. To quantify this, we assess task difficulty using the base VLM’s accuracy in a zero-shot setting:

• Easy: 60–100%
• Medium: 40–60%
• Hard: 0–40%

We then analyze the model’s skip-thinking behavior during the GRPO stage and observe how the skip ratio correlates with query type and difficulty.

Our findings indicate that our TON maintains a high skip ratio for easy counting queries, allowing it to bypass unnecessary thinking processes, while keeping a low skip ratio for harder, more professional, knowledge-intensive questions to fully utilize its thinking capabilities.

Q1: Why does directly applying GRPO lead to a zero coordinate reward in L217

There is because in AITZ benchmark, we require the model to output well structural action format, must adhere to JSON (lines 222-224),{action: 'ACTION_TYPE', value: 'ELEMENT', position: '[x,y]'}, which is more complex than the math problems with a numerical answer (e.g., 5).

We find that open-source models without SFT format following cannot directly follow such complex instructions, making it difficult to extract answers and determine consistency with the ground truth, resulting in large failures. While RL stage is not good for format following and is better for generalization, thus without SFT, direclty GRPO cannot obtain reward to support training.

We appreciate your questions and comments very much. Please let us know for any further questions.

Thank you so much for digging into our difficulty‐aware dropout ablation. We’d like to clarify two key points:

1. Ablation, not contribution. The “difficulty-aware” dropout—we drop thoughts only on examples the base VLM answers correctly—was included solely as an ablation study at your suggestion. It is not our core contribution. Its purpose was to demonstrate that even a hand-crafted, noisy difficulty signal cannot match the performance of our simpler strategy.
2. Core innovation: random dropout. Our main idea is that randomly dropping reasoning traces during SFT lets the model learn pure format-following, then autonomously decide when to think or skip during the RL stage, all without any manual difficulty labels. The 0.5 gain (51.0 vs. 50.5) shows that human-defined “easy/hard” heuristics often introduce noise and can actually hinder performance compared to a fully random approach.

We hope this clears things up: our strength really comes from doing away with handcrafted difficulty definitions. If you have any thoughts, suggestions, or concerns about this “heuristic-free” angle of TON, we’d be grateful to hear them.

Thanks for the detailed rebuttal. Most of the concerns are addressed except W1. The experiments show random dropout actually outperforms difficulty-aware dropout (51.0 vs 50.5), suggesting that defining difficulty via noisy labels (answer correctness) may not be optimal. Thus, the current version could be improved by a more proper definition of difficulty (instead of directly using noisy labels). I decide to maintain a borderline negative rating.

We appreciate that you found our method to have strong performance. We value your insights and hope our response addresses your comments.

Please find our detailed responses to your specific questions below. For clarity, we use the following notations:

• W – Weakness
• Q - Question

W1: The motivation for “over-dense” chains of thought is not fully substantiated.

We would like to clarify that our motivation is not primarily aimed at optimizing over-dense chains of thought (i.e., making reasoning more efficient), but rather at addressing a more fundamental question:
"When should the model think?" — i.e., identifying whether explicit reasoning is necessary.

As shown in Figure 2, we conduct our motivation study on the AITZ benchmark, which does not exhibit excessively long CoT traces. Despite this, we observe that a non-trivial portion of cases (7.6%) still fail. More notably, 52.1% of the answers remain correct even without any reasoning, and 14.5% are correct only when reasoning is skipped. This implies that explicit reasoning is not always required for optimal performance. Further evidence and expanded analysis supporting this motivation are provided in Appendix A.

We appreciate your insightful comment and agree that applying our framework to settings with long and potentially over-dense CoT is a valuable future direction. We plan to explore this in subsequent work.

Q1: Why choose multimodal as the testbed? Can TON transfer to LLMs?

• We believe that vision tasks are more intuitive and can often be resolved at a glance, so it would be a good testbed to study think-or-not, while language-centric benchmarks are more focused on knowledge, which makes them difficult to articulate in detail. In contrast, text-based tasks typically provide more comprehensive information, thus would be better for long-cot. Visual tasks tend to be more redundant; often, a quick glance is sufficient to determine whether further thought is necessary, akin to human cognition.
• TON is being generic, and we adapt our TON to the text-only datasets based on LLMs. Here, we give the experiments on the text-only tasks, and the results are as below. The results show that our TON reduces the length greatly while retaining the high accuracy. We have included these experiment settings in our revision.

Q2: What specific mechanisms of TON lead to accuracy gains? Considering that redundant reasoning, in principle, should preserve accuracy

First, we challenge the assumption that redundant reasoning steps should preserve accuracy. This is not always true. In fact, saying more does not always mean saying it correctly. As demonstrated in Table 6 and Figures 15 and 16, lengthy but flawed reasoning processes can lead to incorrect answers by increasing the likelihood of hallucinations or logical errors.

Second, we attribute our TON' effictiveness to two reasons.

• (i) TON effectively skips incorrect or redundant thinking processes, alleivating hallucinations. Fig. 2 shows that 52.1% of answers remained correct without the "think" process, and 14.5% were only correct when "think" was omitted. This suggests that explicit reasoning is not always necessary. Sup. A.3 gives examples on GeoQA where the reasoning processes are either incorrect or redundant, leading to wrong answers. Our TON can skip these misleading reasoning steps and output answers directly and automatically.
• (ii) Introducing the non-think enlarge diversity on the distribution space of responses when GRPO sampling. Fig. 3 illustrates that TON can introduce empty thinking into the GRPO algorithm, thereby increasing response diversity within each group. Unlike previous works like DAPO emphasizes advantage distribution $A_i$ by dynamic sampling in the sparse reward space, our TON shifts the focus to the latent distribution space of responses $\pi_\theta(o_i|v,q)$, thus enhancing the diversity of the terms $\alpha, \beta$ in equation 4.

Together, these mechanisms explain how TON improves accuracy—not by blindly preserving all reasoning, but by selectively reasoning only when it adds value.

Q3: What is the novelty of TON compared to length-adapted related works?

(i) TON v.s. length-adapted methods: Length-adapted methods assume reasoning is always needed but optimize for shorter paths. Our TON tackles a more fundamental problem: whether to think or not. Figure 4(d) shows that the reasoning length of TON is similar to the vanilla GRPO.

(ii) Orthogonality and Compatibility. TON’s think-or-not decision is orthogonal to length-adaptive methods. For instance, TON first determines whether reasoning is necessary; if it is, adaptive reasoning methods can then optimize the extent of that reasoning. This synergy holds great promise for future research.

(iii) Lastly, we want to friendly remind that the works you suggested: Thinkless (21 May 2025), LASER (21 May 2025), AdaCoT (17 May 2025) mentioned above were released after the NeurIPS2025 full paper submission (15 May 2025) deadline. meaning that we are concurrent works. We would love to add a discussion of length-adaptive methods in related works in the revision.

We appreciate your questions and comments very much. Please let us know for any further questions.

We are very encouraged by your positive response! Below we have addressed your questions:

Please find our detailed responses to your specific questions below. For clarity, we use the following notations:

• W – Weakness
• Q - Question

Q1: Rationale of Two-stage.

The two-stage framework originated from our exploratory experiments in Section 5.4--Emprical Verfication of SFT Significance in TON.

• We initially attempted to utilize only the GRPO stage without the preceding SFT stage: (1) hybrid prompts where prompt the models to skip think if it is confidence enough and (2) Non-think rewards give reward 1 if it skip the think. Unfortunately, these attempts did not induce skipping;
• Fig. 5d shows a skip ratio of 0, and the training curve in Appendix. G indicates that non-think rewards also remained at 0. We analyzed this phenomenon and attributed it to the VLMs defaulting to a conservative, fully reasoning approach, resulting in a nearly 100% think rate.

This motivated us to inject the non-think prior during the SFT stage using thought dropout, which we found necessary to break this bias toward full reasoning.

Q1: Rationale of Random dropout.

Thank you for raising this question. Below, we outline the motivation and supporting analysis behind our use of random dropout for reasoning traces:

1. Random vs. Difficulty-aware Dropout We also experimented with a difficulty-aware dropout strategy, where reasoning is dropped only for "easy" samples (those correctly answered by the base VLM). The comparison on GeoQA is as follows:

Surprisingly, random dropout performs slightly better. This suggests that hand-crafted heuristics for task difficulty may introduce noise or unintended bias, potentially interfering with the learning process. Random dropout offers a simpler, unbiased alternative that generalizes well across tasks.

2. Adaptive Behavior in GRPO

Despite random dropout during SFT, our model learns to adaptively skip or retain reasoning in GRPO based on task difficulty. For example:

TON maintains a high skip ratio for easy tasks while preserving reasoning for more complex queries, demonstrating that the model internalizes task difficulty and adjusts its behavior accordingly—even without explicit difficulty labels in SFT.

In summary, random dropout is a simple yet effective mechanism to encourage flexible, adaptive reasoning while avoiding the heuristic-based supervision. We will expand this discussion in the revised paper.

Q2: Can TON be scalable to tasks without binary or deterministic rewards?

TON primarily addresses selective reasoning in VLMs by introducing a generic thought-dropout method, due to its generic and concise method, it do not add any regularization on the reward definitions. Thus, the TON framework can still be adapted to uncertain environments in the following ways:

1. Adapting reward beyond binary/discrete metrics: In such cases, we can still define formal reward instead of binary metrics, such as in Mobile navigation, regrading the action grounding, we could use the absolute L1 distance between click coordinate and box as continuous reward measurement.
2. For these tasks without deterministic rewards: This being an open challenge in RL training, a potential solution could be training a critic model or adopting powerful models to predict rewards to supervise training.

But we want to emphasize that as TON does not introduce explicit reward regularization; thus, TON could be flexibly adapted to these scenarios. We would like to explore this extension in future work.

Q3: Deeper Analysis of think and How TON improve it

Thanks for raising insightful question regarding the our finding. Follow your suggestion, we analyze 10 examples in Appendix G.8 and Supp. A.3 with the outputs of zero-short, grpo and our TON on 3 tasks. We have the following obseravations:

• The model may generate hallucinated reasoning that leads to incorrect answers.
  As shown in Table 10, even on simple 3D counting tasks (e.g., "How many items are in the picture?"), the model may produce meaningless or incorrect reasoning steps, ultimately resulting in wrong answers.

• TON exhibits greater robustness with respect to the presence or absence of reasoning. As shown in Figures 15 and 16, TON is capable of providing correct answers even without an explicit reasoning process. We attribute this to two key factors:

(i) TON can selectively skip reasoning steps, thereby reducing the likelihood of hallucinations or logical errors.

(ii) For many VLMs, this suggests that generating a correct answer along with a valid reasoning trace is more difficult than generating the answer alone.

How TON improve this issue?

• TON can skip the reasoning process to alleviate incorrect reasoning, allowing it to provide direct answers for easier questions. In Table 5, models outputs a lengthy but incorrect reasoning process (~1k), which hinders the model's ability to answer correctly. In contrast, TON directly provides the correct answer with minimal reasoning.
• Insights about the model’s internal reasoning reliability: for simpler questions, skipping the reasoning process reduces the probability of errors, offering a more efficient pathway to correct answers.

These analyses provide a deeper understanding of the behavior and advantages of our TON framework. We will incorporate this discussion into the revision.

Q4: Differentiate TON from existing length adaptive reasoning methods in LLM

We thank you for raising such insightful comments.

Firstly, it is necessary to clarify the key differences between our TON to prior adaptive reasoning methods.

(i) How to think v.s. When to think: Adaptive reasoning methods assume reasoning is always needed but optimize for shorter paths. Our TON tackles a more fundamental problem: whether to think or not. Figure 4(d) shows that the reasoning length of TON for those unskipped questions is still similar to the vanilla GRPO.

(ii) Orthogonality and Compatibility. TON’s think-or-not decision is orthogonal to length-adaptive methods. For instance, TON first determines whether reasoning is necessary; if it is, adaptive reasoning methods can then optimize the extent of that reasoning. This synergy holds great promise for future research.

In the future, we would like to include more visual-centric designs, such as extend our ‘dropout’ to visual inputs, let model to skip redundant visual tokens as well.

We appreciate your questions and comments very much. Please let us know for any further questions.

We appreciate your valuable feedback and are glad to hear that you found the selective reasoning problem interesting. We value this rebuttal opportunity to clarify issues and address your concerns.

Please find our detailed responses to your specific questions below. For clarity, we use the following notations:

• W – Weakness
• Q - Question

W1: Consider Thought Dropout with task difficulty.

Thank you for emphasizing the importance of task difficulty—this is indeed a crucial factor, we also value this most, we appreciate the opportunity to discuss it in greater depth.

1. How to measure Task Difficulty?

• By Answer Correctness: A more direct approach is to classify each sample based on whether the base VLM answers it correctly. We then drop thoughts for only the “easy” samples (those answered correctly by the base VLM). To investigate this, we implemented a difficulty-aware dropout strategy and compared its accuracy under our TON training framework.

Somewhat surprisingly, the accuracy of the difficulty-aware dropout version is similar to, and even slightly lower than, our proposed random dropout method. This outcome may be attributed to the challenge of precisely defining “task difficulty”. Human-defined heuristics introduce potential noise and risk interfering with model behavior.

Therefore, we adopt a conservative and principled approach: we treat all samples equally, allowing the model to acquire format-following skills during SFT and to self-explore and learn effectively during the GRPO stage. This strategy ensures that the model’s improvements are intrinsic and not confounded by external definitions or interventions.

2. Despite Random Dropout in SFT, the Model Adapts to Task Difficulty in the RL Stage

As shown in Supplementary Section A.1, even with random thought dropout during SFT, our TON framework enables the model to adapt its skip-thinking behavior during GRPO based on task difficulty and reasoning complexity. We analyze performance across a range of benchmarks—including 3D object counting, GUI-based mobile agent navigation, and geometric math reasoning—spanning a spectrum of task difficulties. Task difficulty is assessed both qualitatively (via intuitive reasoning complexity) and quantitatively (by base VLM zero-shot accuracy), grouped as:

• 60-100 (easy)
• 40-60 (medium)
• 0-40 (hard).

The results of the final skip-ratios and our TON accuracy are as follows:

Our findings indicate that our TON maintains a high skip ratio for easy counting queries, allowing it to bypass unnecessary thinking processes, while keeping a low skip ratio for harder, more professional, knowledge-intensive questions to fully utilize its thinking capabilities.

3. Possible future extension: We acknowledge that the skip-thinking mechanism may introduce the risk of the model bypassing necessary complex reasoning steps, and we plan to address it in future work.

(i) a potential direction is to incorporate auxiliary score such as visual perplexity (PPL) scores as a proxy for estimating task difficulty.

(ii) Another possibility is to evaluate whether the model can answer a question correctly both with and without the reasoning trace. If so, the reasoning may be redundant and thus safely skippable.

W1: Discussion of dropout rates with dataset characteristics.

We have conducted a dataset-aware analysis of dropout behavior as part of our experimental investigation. Specifically,

• [Same dataset, different ratio] In lines 255-263, we present the ablation study and insights into selecting dropout rates of 20%, 50%, and 80%. Our findings indicate a correlation between the dropout ratio and the zero-shot performance across the datasets.

• [Different dataset, different ratio] In Appendix G.5 and G.6, we analyze the training curves of these dropout ratios across datasets of varying reasoning difficulties. we observe that, despite differing dropout ratios, TON consistently shows an increasing skip ratio as training progresses. Notably, the lower dropout ratios demonstrate a rapid increase in skip rates, while the higher ratios remain relatively stable throughout training. TON can dynamically adjust according to reward signals—decreasing the dropout ratio when performance is high and increasing it when performance declines. Thus, we recommend starting with a lower dropout probability for further investigation, such as a 20% ratio for more difficult tasks (i.e., zero-shot accuracy < 40) and an 80% ratio for easier tasks (i.e., zero-shot accuracy > 60).

We thanks for your suggestions. These insights enhance our deeply understanding of TON, and we will highlight them in our revision.

W2: Would be better to have Discussion between Adaptive reasoning methods in LLMs and TON.

Thanks for your suggestion and for mentioning our work with these adaptive reasoning methods.

Firstly, it is necessary to clarify the key differences between our TON to prior adaptive reasoning methods.

(i) How to think v.s. When to think: Adaptive reasoning methods assume reasoning is always needed but optimize for shorter paths. Our TON tackles a more fundamental problem: whether to think or not. Figure 4(d) shows that the reasoning length of TON for those unskipped questions is still similar to the vanilla GRPO.

(ii) Orthogonality and Compatibility. TON’s think-or-not decision is orthogonal to length-adaptive methods. For instance, TON first determines whether reasoning is necessary; if it is, adaptive reasoning methods can then optimize the extent of that reasoning. This synergy holds great promise for future research.

(iii) We further include the representative adaptive reasoning method DAPO [1], which apply the adaptive reasoning with dynamic sampling and the normalziation of the token numbers within the group; We apply it on VLMs for comparison on GeoQA, and the results are as below: Despite both variants yield signifcant improvement over original GRPO, our TON with minimal design, which reduces the length greatly while retaining the high accuracy.

[1] “DAPO: An open-source llm reinforcement learning system at scale.” arXiv preprint arXiv:2503.14476 (2025).

W3: Would be better to apply TON on other or larger models.

Thanks for your suggestion. We add the experiments on InternVL3-2B and 8B as follows. Our TON can retain comparable accuracy while reducing the think length significantly, particularly, it reduces length further on a stronger model (8B), which demonstrates the generalization of TON across different backbones. Due to time limitations, more results on the bigger VLM, such as Qwen2.5-VL-32B, will be included in our next version.

We appreciate your questions and comments very much. Please let us know for any further questions.

Dear Reviewer WjY5,

This is the last call for author-reviewer discussion, which will ends on Aug 8. Could you please read authors' rebuttal and confirm whether your concerns have been addressed asap? Thank you.

Best,

AC

Thanks for your reply. We respectfully disagree with your statements, which we believe are unfair. Our detailed clarifications for each point are provided below.

1. TON uses random thought in SFT, but remains adaptive in GRPO.

• The SFT stage is solely for format following—whether to “think” or not is part of that format—as evidenced in Section 5.4 (Empirical Verification of SFT Significance in TON). We explicitly separate format following from reasoning ability: random dropout in SFT establishes the format, while reasoning ability is developed in the GRPO stage. In GRPO, the model adapts freely, learning when to skip unnecessary steps to optimize RL performance, thus remaining adaptive. In other words, adaptiveness does not need to be introduced in the SFT stage, as each stage targets a distinct objective'.

• This adaptiveness mirrors standard dropout in neural networks—randomness during training still enhances generalization, as the network learns to handle varied conditions.

2. Does human design in SFT really equal an adaptive method?

• Human design inevitably introduces heuristics. In our experiments, the straightforward difficulty-aware design (judging necessity based on correctness) achieved comparable accuracy to—and even slightly lower than—our random dropout method. This suggests that a carefully crafted SFT strategy has limited influence on final performance, making human heuristics largely unnecessary.

• Moreover, creating a single human-defined assumption that generalizes across diverse setups is inherently difficult. In our experiments spanning math, visual, and agent tasks, the more robust and reliable approach is to let the model explore autonomously within its specific environment during the RL GRPO stage.

3. ‘When to Think’ vs. ‘How to Think’?

• The reviewer WjY5 claims that “When to Think” is a subset of “How to Think,” which we disagree with. The distinction between "think" and "non-think" modes is non-trivial. The significance of the “Think-or-not” problem is also recognized by the reviewer vGCC, g1TJ, WjY5, bSRJ. (vGCC highlights that "the valuable practice value in real-world applications.", g1TJ notes that "the idea to equip VLMs with selective reasoning is practical"). In Fig. 5d (G.7 Prompt vs. SFT on different benchmarks), we tested using only the GRPO stage with hybrid prompts—asking the model to skip thinking if confident—but the skip ratio remained 0. We attribute this to VLMs’ strong conservative bias toward full reasoning, leading to an almost 100% think rate.

• Recent top-performing models such as Gemini-2.5-Flash [1], Qwen3 [2], and GPT-5 [3] underscore the importance of routing between thinking and non-thinking modes. Gemini-2.5-Flash and Qwen3 even provide a manual switch to toggle between the two modes, while GPT-5 introduces an automated router that decides based on the user query. These findings highlight the significance and practicality of “When to Think,” showing it is a distinct, impactful, and unsolved challenge—not merely a subset of “How to Think.”

[1] Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities.

[2] Qwen3 Technical Report

[3] GPT-5 System Card

4. Many existing papers on “How to Think” in LLMs.

• Our work focuses on “When to Think” rather than “How to Think”, as the first to use vision-language models as the testbed across diverse setups—math, visual counting, and agent tasks. We show the substantial potential of the think-or-not paradigm in VLMs to improve efficiency without sacrificing performance.

• In an orthogonal direction, the newest LLM methods such as AdaCoT, ThinkLess, and LASER [4–6] focus on how to think. Most works were released on arXiv only after the NeurIPS full submission deadline and therefore do not diminish the novelty, originality, or timeliness of our contribution.

[4] AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning. arXiv:2505.11896 (2025).
[5] ThinkLess: LLM Learns When to Think. arXiv:2505.13379 (2025).
[6] Learn to Reason Efficiently with Adaptive Length-Based Reward Shaping. arXiv:2505.15612 (2025).