LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perception

We thank the reviewer for the thoughtful and constructive feedback. We're encouraged by your recognition of the scalable, automated pipeline we propose and the demonstrated improvements across diverse tasks. We acknowledge your concerns regarding writing and analysis; however, other reviewers have positively recognized this aspect of our work. For instance:

• Reviewer DSU2 says that "the paper has conducted insightful analysis of where the performance improvement comes from ..."
• Reviewer EHkH says that “the writing is very clear and easy to follow, the experiments are thorough and complete ...”

Below, we address each of your concerns:

1. Clarification of the title

What the authors do is to use a system-2 approach to solve vision-centric (perception) tasks that are usually addressed with a system-1 approach.

Yes, this is exactly our intention. Our goal is to tackle traditionally system-1 perception tasks using explicit, structured, system-2-style reasoning. We will revise the abstract and introduction to make this goal more transparent early on.

I don't see where there is a distillation happening here ... It seems here that a better word would simply "synthesize"?

We agree that the term "distill" may be confusing. To clarify: distillation occurs in Stage 3, where a reasoning LLM (R1-Distill-Qwen-32B; see Appendix L676) injects structured reasoning into our dataset. We will revise the text to use "synthesize" for the simple CoTs from the VLM and reserve "distill" for the structured traces from R1.

2. The confusion of verifiable multiple-choice questions

whatis verifiable? .. it's never made clear what this means precisely, nor how this is used or enforced.

By “verifiable,” we mean questions whose answers can be automatically checked via deterministic string matching against predefined options—unlike open-ended or VQA-style questions that rely on potentially noisy LLM-based judgments

This property is essential to our pipeline: in Section 2.4, we use these verifiable questions to determine the correctness of ($a_1$, $a_2$, $z_1$, $z_2$), which is critical to construct SFT and preference data.

3. Key aspects of the proposed generation pipeline

Which key aspects of the proposed data/generation pipeline are important?

Thank you for pointing this out. Our pipeline’s effectiveness stems from the interplay of three stages:

• Stage 1: generating verifiable multiple-choice questions from image captions
• Stage 2: using VLMs to produce simple CoTs aligned with their output style
• Stage 3: refining these with a reasoning LLM to inject deeper cognitive structure.

Each stage contributes uniquely, enabling both reliable evaluation and progressively richer reasoning.

To mitigate the reviewer's concern, We direct the reviewer to Table 1: LongPerceptualThoughts is compared with two other visual reasoning datasets, VL-Thinking [1] and Virgo. We note that VL-Thinking distills data from R1 followed by rewriting and filtering steps, yet still yields negative SFT performance, even on in-domain tasks. In contrast, our approach demonstrates that data synthesized using our pipeline can support effective SFT, suggesting the benefits of our framework for vision-language model training.

[1] Chen et al., SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models

We would like to clarify that the dataset referred to as VLAA-thinking in [1] is a renamed and expanded version of the VL-thinking dataset used in our submission.

4. Analysis of response lengths in Section 3.4

... Shouldn't we expect the responses to always be much longer?

This is an excellent observation. Our paper’s goal is to enable long-form reasoning, and we address this in two parts:

1. Are the LongPerceptualThoughts responses longer than vanilla VLM outputs?

Yes. We compared our dataset to Qwen2.5-VL’s generations on DOCCI, and the LongPerceptualThoughts responses are longer on average. Please refer to this external figure

2. Does fine-tuning lead to longer outputs?

Interestingly, not always. As noted in L165, our DPO training balances correctness and brevity. Thus, even if the model has access to longer CoTs, it may choose more concise outputs in certain cases. Indeed, Figure 3b shows that SFT-trained models tend to generate longer responses, while DPO-trained ones remain relatively compact despite better performance.

5. Toward a more comprehensive study

Main reason to reject: ... No ablations etc. Not sufficient for publication in its current form, ...

While we agree that a comprehensive ablation of hyperparameters in our synthetic data pipeline would strengthen the paper and plan to pursue this in future work, it is not feasible during the rebuttal period due to time and budget constraints.

6. Typos

We will revise the paper accordingly.

We appreciate the reviewer’s positive feedback on our data-centric approach, the significant performance gains across multiple benchmarks, and the improvements on the challenging text reasoning benchmark.

In the followings, we address the reviewer's concerns and questions:

1. Comparison with existing CoT approach that modify inputs

We appreciate the reviewer’s comment and clarify that our work focuses on implicitly equipping VLMs with an internal search mechanism (L37–L41), differing from prior methods [1,2] that modify input images to enhance perception. These approaches are conceptually distinct and complementary to ours.

To directly address the reviewer’s point, we evaluated VisCoT [1] on V* Bench using the official codebase (GitHub link). For a fair comparison, we selected VisCoT-7b-336, which is similar in size to Qwen2.5-VL-7b-Instruct, in particular VisCoT-7b-336.

As shown in the table below, VisCoT-7b-336 slightly outperforms Qwen2.5-VL-7b-Instruct. However, after applying SFT followed by DPO using our proposed LongPerceptualThoughts dataset, we observe a +7 point improvement over VisCoT.

We hope this result helps clarify that while our method shows promising gains, it is complementary in spirit to works like VisCoT, which take a more explicit input-modification approach (also noted in L44). We believe combining these directions may offer further potential.

[1] Shao et al., Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

[2] Zhang et al., MLLMs know where to look: Training-free perception of small details with multimodal LLMs.

2. Ablation study

The ablation study presented is limited, ... Can the authors provide a more comprehensive ablation study ... This would include varying the number of multiple-choice options in Stage 1 and testing different trigger words in Stage 3.

This paper aims to enhance system-2 thinking in VLMs for perceptual tasks by synthesizing a dataset tailored for standard training methods like SFT or DPO. Inspired by the prior works (L139–140), we propose a novel three-stage data synthesis pipeline and demonstrate strong in-domain and out-of-domain gains. Design choices such as the number of options in Stage 1 and trigger words in Stage 3 are intended as tunable hyperparameters. A full ablation of these factors is beyond our current computational resources.

To mitigate the reviewer's concern, we highlight the results in Table 1: LongPerceptualThoughts is compared with two other visual reasoning datasets, VL-Thinking [3] and Virgo. We note that VL-Thinking distills data from R1 followed by rewriting and filtering steps, yet still yields negative SFT performance, even on in-domain tasks. In contrast, our approach demonstrates that data synthesized using our pipeline can support effective SFT, suggesting the benefits of our framework for vision-language model training.

[3] Chen et al., SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models

We would like to clarify that the dataset referred to as VLAA-thinking in [3] is a renamed and expanded version of the VL-thinking dataset used in our submission.

We thank the reviewer for the thoughtful feedback. Below, we address the concerns of Q1 and Q3:

Q1 (Additional Baselines and Benchmarks)

We have now conducted additional experiments on CV Bench and MMVP. As shown in the table below, our method consistently outperforms baselines across these diverse benchmarks, further supporting the generalizability of our approach. If the reviewer has another method in mind in addition to Visual-CoT, please let us know and we will add it.

Q3 (Diverse Chain-of-Thoughts)

We thank the reviewer for acknowledging the promising results. To better emphasize/visualize significant increase in token length in the external figure, we dump their statistics in table format:

Regarding "how the method enhances diverse CoTs", we respectfully point out that our paper includes two analyses that quantify the changes introduced by thought expansion: (1) Diversity of cognitive behaviors (Fig. 3a) and (2) Token Length increase in the above table.

If the reviewer could suggest specific metrics they believe are missing, we would be happy to incorporate such analysis.

We appreciate the reviewer’s positive feedback on our dataset, the potential usefulness of our data generation pipeline for other domains, and the insightful analysis of the dataset’s reasoning patterns.

In the followings, we address the reviewer's concerns and questions:

1. Details and prompts of constructing datasets.

We agree that openness is important to the community. Section E.1 outlines the sampling parameters for each stage, and the prompts used are shown in Figures 5, 6, and 9. We thank the reviewer for noting the missing prompt in Stage 3 of the proposed data synthesis pipeline. We will also be making the code available for reproducibility.

Below is the prompt used in Stage 3, which we will add to the appendix.

User: You are a large language model that answers visual questions by generating a vivid mental image from a text description. Given a visual question along with an image description, create a detailed internal visualization of the image. Then, use this mental image to spatially reason through and answer the question.

- After building the mental image from the text description, you should not explicitly referencing the text description in your internal reasoning. e.g., Avoid saying "The description states ..." within <think>...</think> block.
- Ensure your reasoning is logically sound and leads coherently to the final answer. The steps you follow should clearly support the conclusion you reach.
- Please provide your answer as (X), where X is the letter of the correct option.
- Enclose your final answer within <answer> and </answer> tags.

Assistant: <think> {{ simple_CoT }} {{ cognitive_phrase }}

Variable explained:

• simple_CoT is the extracted thought from Stage 3.
• cognitive_phrase is the pre-defined cognitive cues, such as "Wait,", "Hmm,", and "Alternatively,"

2. Evaluation on different tasks.

To address the reviewer’s request for evaluation beyond multiple-choice, perceptual tasks, we include preliminary results on DocVQA-val [1], a benchmark that better aligns with the desired criteria (non-multiple-choice, reasoning-focused). We sample the first 1,000 examples from the validation set (total: 5,349 QA pairs) and evaluate model outputs using gpt-4o-2024-08-06 as an LLM-as-judge.

As shown in the table below, fine-tuning on LongPerceptualThoughts yields a >30-point improvement in accuracy over zero-shot predictions. This result highlights the generalization benefits of our dataset to broader reasoning tasks beyond perception.

To ensure transparency, we include the exact prompt we used for LLM-as-a-judge evaluation below:

System: You are a rigorous evaluator judging whether AI model predictions match ground truth answers. 
Always respond with 'yes' or 'no' only.

User: You are a strict judge evaluating an AI model's prediction.
All Possible Ground Truth Answer: {{ ground_truth_list }}
Model Prediction: "{{ prediction }}"

Does the prediction correctly and completely match the ground truth?
Respond only with 'yes' or 'no'.

We will include evaluation results using a standardized prompt format in the camera-ready version upon acceptance.

[1] Mathew et al., DocVQA: A Dataset for VQA on Document Images

We appreciate the reviewer’s positive feedback on the clarity of the writing, the completeness of the experiments, and the importance of the research question addressed.

In the followings, we address the reviewer's concerns and questions

1. Varying thresholds of #pixels in Table 1, especially on V* benchmark

As clarified in Section E.1, we set the maximum resolution to 512×512 during both training and inference throughout the paper.

To address the reviewer’s question regarding the removal of the visual token budget, we conducted an ablation by varying the inference resolution while keeping the training setup fixed at a maximum of 512×512. Using the same VLMs from Table 1, we find:

• At 512×512 (aligned with training), our fine-tuned VLM achieves the largest gains.
• Increasing inference resolution introduces a distribution shift, as test inputs may exceed the training-time resolution. Nonetheless, scaling up to 2048×2048 still yields over a 2-point accuracy improvement relative to Qwen2.5-VL-7B-Instruct.

These results indicate that LongPerceptualThoughts remains effective even without a strict visual token budget. We will clarify this in the final version.

2. Hallucination benchmark

We appreciate the reviewer’s concern regarding potential hallucinations introduced by equipping models with additional "thinking" steps. To evaluate this, we adopt the POPE benchmark [1]. POPE is consists of 9,000 examples and specifically targets on object hallucination. It has three splits: random, popular, and adversarial. We follow the exact setup in Table 1 and report both F1 and accuracy on the full benchmark.

As shown in the table below, our method, LongPerceptualThoughts, achieves consistently better performance than the baseline, despite the added thinking tokens. We hypothesize that this improvement stems from the injection of structured cognitive behaviors (see Fig. 3a), which might offer some benefit in managing hallucination. While we observe improvements on POPE, we acknowledge that the relationship between extended reasoning and hallucination remains an open area of investigation.

We agree this is an important direction for developing reasoning-capable VLMs and will include this analysis in the final version.

3. Additional related work

We thank the reviewer for highlighting relevant prior work [2]. Similar to [3,4], which improve vision-centric tasks by helping VLMs "see" better, our approach is complementary: it encourages the model to "think" longer by exploring inter-search mechanisms (see L37–L45).

We will clarify this distinction and add a discussion to Section 4.

[1] Li et al., Evaluating Object Hallucination in Large Vision-Language Models

[2] Zhang et al., MLLMs know where to look: Training-free perception of small details with multimodal LLMs.

[3] Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms.

[4] Shao et al., Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning