Unified Reinforcement and Imitation Learning for Vision-Language Models

Q1. The requirement of multiple large VLMs and LLMs for both discriminator and judge-model training might limit accessibility due to significant computational resource demands.

Compared to the original rule-based GRPO, we agree that RIL introduces increased computational costs during training. However, we would like to emphasize that these costs can be mitigated through efficient technical implementation strategies. While pretraining the discriminator is a necessary step and cannot be avoided from a computational standpoint, we can reduce training costs during the RIL loop by managing model weights efficiently. Specifically, we offload the weights of the small VLMs and the discriminator to the CPU when they are not in use. When needed, we load the appropriate weights from the CPU to the GPU using DeepSpeed API. This allows us to dynamically swap model weights without uploading both architecture to GPU, thereby minimizing computational overhead. This approach is feasible because we use the same model architecture for both the small VLMs and the discriminator, enabling seamless weight replacement and resource-efficient training.

---

Q2. This manuscript is well-written and presents a strong empirical contribution to efficient vision-language modeling. However, the degree of innovation might appear incremental given the existing frameworks. Please further emphasize the innovation and the problems addressed by this paper.

We would like to emphasize two key points regarding the innovation of our work:

• While individual components such as Dr.GRPO and LLM-as-a-Judge have been previously explored, the overall framework we propose is novel in that it enables the distillation of knowledge from multiple large VLMs into a single, smaller VLM. This unified design allows us to leverage the strengths of various large models in a systematic and scalable manner.

• Our approach is orthogonal to advancements in VLM architectures and agnostic to specific VLM series, thanks to the generalized nature of our framework. This makes it broadly applicable and future-proof as new VLMs emerge.

Specifically, RIL provides a generalization effect similar to RL and mimics the benefits of high-quality SFT, guiding smaller VLMs to learn more flexible responses beyond fixed ground-truth answers. This is because RIL basically leverages the diversity of responses generated by large VLMs, which are more likely to produce high-quality answers compared to smaller VLMs, and these generated responses are directly used to simultaneously train both the discriminator and the smaller VLMs, as described in lines 6–13 of Algorithm 2.

Thanks to such sophisticated framework design, the experimental results showed the significant performance improvement on multiple benchmarks with multiple VLM families. Therefore, once the new VLM is developed in the community, this approach can exploit such model to develop lighter and more accurate VLMs.

We will add this explanation to the manuscript to better clarify this important point.

---

Q3. What mechanisms or hyperparameter tuning strategies are critical to maintaining training stability, particularly in adversarial imitation settings?

We have addressed the training stability from multiple perspectives:

• Structured Prompt for LLM-as-a-Judge: Please refer to our response of Reviewer SQq6's Q3.
• Binary Answer Reward: Please refer to our response of Reviewer SQq6's Q3.
• Binary Similarity Reward: Please refer to our response of Reviewer Hg1D’s Q4.
• Quality and Diversity of Generated Responses from Teacher Models: Please refer to our response of Reviewer SQq6’s Q4.
• Hyperparameter: To prevent unstable training, we conducted several ablation studies on the number of discriminator and small VLM training iterations, as well as on the KL divergence hyperparameter, as shown in Table 5(a) and (b).

---

Q4. Could RIL be effectively combined with other recent model alignment methods (e.g., instruction tuning or retrieval augmentation)?

We have already discussed RIL's limitations in the discussion section (lines 285–291) on page 9, noting that the current implementation of RIL primarily focuses on the post-instruction tuning alignment phase. Extending RIL to earlier stages, such as the visual instruction tuning phase, would be a promising direction for future research.

---

Q5. How sensitive is the approach to the selection and quality of large teacher VLMs? Would performance degrade substantially if lower-quality teachers were used?

Building on our answer to Reviewer SQq6’s Q4, we have conducted additional experiments based on your comment regarding the use of lower-quality teachers.

Both the diversity and the quality of teacher outputs are critical to RIL’s success. As the ablation study below shows, RIL’s performance degrades when teacher responses lack diversity or accuracy. (We assume diversity is satisfied by generating a larger number of responses, since we set the inference hyperparameters—temperature, top‑p, and top‑k.)

The first table reports performance as we vary the number of pre‑generated teacher responses used to train the discriminator and drive the RIL loop:

Note that, this result is based on Qwen2.5-VL-RIL-7B (Both).

We also evaluated the effect of introducing incorrect answers by randomly shuffling a percentage of generated responses relative to their questions (a dash “–” indicates mode collapse):

Note that, this result is based on Qwen2.5-VL-RIL-7B (Both).

Additionally, we evaluated the impact of using downgraded versions of "Qwen2-VL-72B" and "InternVL2.5-78B" as lower-quality teachers on RIL’s performance:

Recap: Based on these results and analyses, we argue that our approach is orthogonal to advancements in VLM families and agnostic to VLM series, due to the generalized nature of our framework. Therefore, as new VLMs are developed by the community, our approach can seamlessly incorporate them to train smaller and more accurate VLMs.

In summary,

• RIL benefits substantially from both diverse and accurate teacher outputs.
• Performance improves steadily as the number of high-quality, varied responses increases.
• Conversely, introducing inconsistent or conflicting answers causes significant performance degradation.
• The use of lower-quality teachers leads to noticeably reduced performance.

---

Q6. In the manuscript, "Eq. (x)" and "Eq. x" are used interchangeably. Please standardize their usage throughout the manuscript.

We have identified that "Eq. x" appears on lines 205 and 250. We will revise these instances to "Eq. (x)" to ensure consistency throughout the manuscript.

---

Thank you for your encouraging review. We hope our responses have further clarified the points you raised.

Q1. One limitation is that the paper focuses solely on VLMs; it would have been valuable to see the approach extended to LLMs as well.

We have also applied RIL to LLMs to evaluate its generalization capabilities. Based on these results, we confirm that RIL is effective for LLMs as well.

Note that, rows without parentheses indicate baseline models without any distillation.

---

Q2. Given the large number of tasks in Table 1, it would be helpful if the authors included an average metric to make it easier to assess the overall improvement at a glance.

We will definitely include an average metric to facilitate easier assessment of overall improvements at a glance. As shown in the table below, it is clear that the proposed approach, "w. RIL (Dr.GRPO + GAIL)", achieves the best performance across all VLM families—even when using only a single teacher VLM.

---

Q3. Based on the experiments, it appears that applying SFT before RIL improves post-training outcomes. In Table 1, when comparing against the GRPO baseline, could the authors clarify whether the GRPO baseline also included SFT as a prior step?

Yes. All experiments reported in the manuscript, including the GRPO baseline, were conducted after completing the SFT training phase.

---

Q4. I understand the single-teacher LLM setup, where the discriminator model provides a binary signal indicating whether a response comes from the teacher. Could the authors clarify how this approach extends to the multiple-teacher scenario?

Given a response T1 from one teacher VLM and T2 from another teacher VLM, and a response S from the student VLM, then the discriminator is trained to output zero when S is provided and one when either T1 or T2 is provided. Additionally, the rewards and their advantages for S, T1, and T2 are computed based on similarity and answer quality. The student VLM is updated using RIL’s objective loss. We will clarify the multi-teacher training setup in the revised manuscript.

---

Q5. Finally, could the authors provide a comparison of the speed to achieve good performance versus vanilla GRPO? It would be interesting to see whether the proposed RIL approach leads to faster convergence.

In terms of convergence speed, GRPO actually converges faster than the RIL framework. This is because GRPO does not rely on any external models, such as a discriminator or a teacher model, but instead improves upon itself. This self-reliance imposes a natural limitation on performance improvement, making the convergence appear relatively quick. In contrast, RIL leverages external models to guide the learning process, allowing it to gradually improve and ultimately surpass the performance of GRPO. While this results in slower convergence compared to GRPO, the overall performance gain is significantly higher, as demonstrated across multiple experiments. (% denotes training percentage, and the results describe average metrics for 14 evaluation benchmarks based on Table 1 in the manuscript.)

Note that, this result is based on Qwen2.5-VL-7B.

---

Thank you for your encouraging review. We hope our responses have further clarified the points you raised.

Q1. The discriminator relies solely on limited cues from the response, and no examples of its decision outputs are provided.

While the discriminator only receives the response as input, this design is intentional for the sake of training efficiency. We have observed that incorporating additional input cues (such as the question including image) does not consistently improve performance and instead introduces more randomness. This suggests that the key contribution lies not in carefully tuning the input format, but in leveraging the discriminator itself as a training signal. Therefore, to reduce computational cost, we deliberately shorten the discriminator's input. For greater clarity, we will definitely include qualitative examples (image, question, predicted response, decision) of the discriminator's outputs in the revised version.

---

Q2. Increased Complexity and Cost

Compared to the original rule-based GRPO, we agree that RIL introduces increased computational costs during training. However, we would like to emphasize that these costs can be mitigated through efficient technical implementation strategies. While pretraining the discriminator is a necessary step and cannot be avoided from a computational standpoint, we can reduce training costs during the RIL loop by managing model weights efficiently. Specifically, we offload the weights of the small VLMs and the discriminator to the CPU when they are not in use. When needed, we load the appropriate weights from the CPU to the GPU using DeepSpeed API. This allows us to dynamically swap model weights without uploading both architecture to GPU, thereby minimizing computational overhead. This approach is feasible because we use the same model architecture for both the small VLMs and the discriminator, enabling seamless weight replacement and resource-efficient training.

---

Q3. Insufficient Discussion of LLM Bias and Reward Hacking

To mitigate reward hacking and bias when using LLM-as-a-Judge, we employ a structured, template‑based format: we wrap the user query, the ground truth, and the model’s output in question, ground truth, and generated text tags, respectively. It is then prompted to explain its reasoning and generate its judgment within answer tags. This structured approach has been widely adopted in prior work [1–5]. Additionally, we cut off the answer reward signal of LLM-as-a-Judge to avoid overestimation or reward hacking, which leads to more stable training. The table below shows how performance varies with different granularities of the answer reward, where a dash '–' denotes mode collapse—training failed because small VLMs could not reliably distinguish, for instance, an answer reward of '0.2' from '0.21'.

[1] Saha, Swarnadeep, et al. "Learning to plan and reason for evaluation with thinking-llm-as-a-judge." arXiv preprint arXiv:2501.18099 (2025).

[2] Yu, Jiachen, et al. "Improve llm-as-a-judge ability as a general ability." arXiv preprint arXiv:2502.11689 (2025).

[3] Chen, Nuo, et al. "Judgelrm: Large reasoning models as a judge." arXiv preprint arXiv:2504.00050 (2025).

[4] Whitehouse, Chenxi, et al. "J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning." arXiv preprint arXiv:2505.10320 (2025).

[5] Huang, Hui, et al. "Think-j: Learning to think for generative llm-as-a-judge." arXiv preprint arXiv:2505.14268 (2025).

---

Q4. How sensitive is RIL to the quality or diversity of teacher outputs?

Both the diversity and the quality of teacher outputs are critical to RIL’s success. As the ablation study below shows, RIL’s performance degrades when teacher responses lack diversity or accuracy. (We assume diversity is satisfied by generating a larger number of responses, since we set the inference hyperparameters—temperature, top‑p, and top‑k.)

The first table reports performance as we vary the number of pre‑generated teacher responses used to train the discriminator and drive the RIL loop:

Note that, this result is based on Qwen2.5-VL-RIL-7B (Both).

We also evaluated the effect of introducing incorrect answers by randomly shuffling a percentage of generated responses relative to their questions (a dash “–” indicates mode collapse):

Note that, this result is based on Qwen2.5-VL-RIL-7B (Both).

In summary, RIL benefits substantially from both diverse and accurate teacher outputs. Performance improves steadily as the number of high‑quality, varied responses increases. Conversely, introducing inconsistent or conflicting answers causes a marked degradation in performance.

---

Q5. Are there any observed failure modes during RIL training? If so, how are they mitigated?

Based on our answers to Q3 and Q4 above, we have addressed these failure modes. Briefly, to prevent them: (1) we use a structured prompt with tags and explicit reasoning for LLM-as-a-Judge, and (2) we convert the continuous reward signal into a binary one to stabilize training. Our response to Reviewer Hg1D’s Q4 provides additional details on training stabilization in the discriminator and the similarity reward signal. Furthermore, to avoid failure modes, we have already conducted the hyperparameter tuning (e.g., the number of updating parameter of discriminator and small VLMs in one training iteration, and coefficient of KL divergence), as shown in Table 5(a) and (b).

---

Q6. How is the answer parsed from the responses generated by RIL-finetuned models?

All evaluation benchmarks for VLMs already provide guidelines on how to parse responses and evaluate accuracy or other metrics. We follow each benchmark's parsing rules, in which we typically use pre-defined input prompts such as: "Answer the question using a single word or phrase", "Please answer yes or no", and "Answer with the option’s letter from the given choices directly".

---

Q7. What prompt format is used for the teacher models? For questions with deterministic answers, the similarity and answer rewards may become highly correlated, potentially limiting the extent of knowledge transfer from teacher to student.

As you mentioned, deterministic answers can clearly limit the extent of knowledge transfer from teacher to student. To address this, we sampled 40K examples from the 4M SFT dataset based on answer length, specifically to avoid deterministic or overly simple questions and instead focus on open-ended or advanced ones. We assumed that if a ground-truth answer required more than 256 tokens, it was more likely to represent an advanced problem. Additionally, we filtered for diverse domains—including image description, advanced chart-document reasoning, knowledge understanding, and multi-step or multi-dimensional solutions—to ensure a broad range of complex tasks. In this way, we made a deliberate effort to avoid deterministic questions and promote richer knowledge transfer. We will definitely add this explanation about how we constructed RIL dataset to the manuscript.

---

We sincerely hope that our rebuttals have resolved your concerns and will lead to a reconsideration of the scores.

Q1. The framework lacks clarity in several key aspects. The initialization and training dynamics of LLM-as-a-Judge are not well explained - it's unclear whether it's trained simultaneously with SFT or how its learning evolves during RIL.

We would like to clarify that, as mentioned in line 164, Qwen2.5-32B is used as LLM-as-a-Judge. It is not trained during SFT or RIL steps. This model serves solely to determine whether the predicted response accurately reflects the meaning of the ground truth. We will revise the manuscript to clarify this point and avoid reader confusion.

---

Q2. Additionally, training duration and stopping conditions are not specified.

Using 256 NVIDIA A100 GPUs, pre-training the discriminator on 1.2M samples (40K [number of samples] $\times$ 16 [generated responses] $\times$ 2 [for both teacher and student]) takes approximately 1 to 3 days. The SFT step on the 4M-sample SFT dataset takes around 3 to 5 days. Conducting the RIL loop for the sampled 40K data requires an additional 3 to 5 days using 8 NVIDIA A100 GPUs. We do not apply specific stopping conditions; instead, we train one epoch but conduct four repetitive sampling per question-answer pair.

---

Q3. The paper doesn't adequately explain why RIL produces considerably different improvements across benchmarks.

We believe the performance differences of RIL across benchmarks stem from the nature of the tasks themselves. Some benchmarks require only short or factual answers, for example, "what color of t-shirt is a person wearing in the image?", (e.g, SEED2+ and RWQA), while others demand longer reasoning (e.g., MathVista, MMMU), which covers image description, advanced chart-document reasoning, knowledge understanding, and multi-dimensional or step-by-step solutions. Since reinforcement learning introduces a generalization effect, especially when mimicking the text style of large VLMs, the smaller models naturally learn to produce more coherent, logical, and well-structured responses. This is particularly beneficial in tasks that require reasoning or explanation. Therefore, RIL tends to yield greater improvements on benchmarks that benefit from such reasoning ability.

---

Q4. Converting continuous discriminator scores to binary values may discard informative signals, e.g., to which extent the responses from the student model is indiscernible.

We would like to clarify that using binary rewards enables efficient and stable training by controlling the reward steps applied. As shown in the table, a simpler reward design appears to positively impact training stability and efficiency.

There are two possible reasons for this: (a) it is difficult for models to understand why a score of 0.21 is better than 0.2, and (b) discriminator scores might be overconfident or underconfident—a well-known phenomenon in deep learning. This has led to many studies highlighting the importance of calibrated confidence. However, utilizing continuous rewards is not the focus of our work, and therefore, we do not incorporate any calibration techniques.

---

Q5. Concrete justification beyond "drawing inspiration from prior works" could help strengthen the motivation behind this design.

Prior works [1-2] have argued that natural language response-based distillation is more effective than high-dimensional feature distillation, emphasizing the importance of utilizing the language head—referred to as the verbalization effect. Motivated by this insight, we began by leveraging the similarity in natural language responses between teacher and student models through reinforcement learning and imitation learning. We will incorporate this explanation into the manuscript to better clarify the motivation behind our design.

[1] Lee, Byung-Kwan, et al. "VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[2] Li, Yuhui, et al. "Eagle: Speculative sampling requires rethinking feature uncertainty." arXiv preprint arXiv:2401.15077 (2024).

---

Q6. There seems to be presentation issues with experimental results appearing before the proposed framework is explained (e.g., Table 1 on page 5), and excessive tables with extensive numerical results throughout the main paper. The core contribution could be communicated more effectively by moving comprehensive numerical results to the appendix while retaining only the most critical comparisons in the main text.

We will revise the structure of the paper to ensure that the proposed framework and its components are clearly introduced before presenting the corresponding experimental results, including Table 1. Additionally, we will relocate the more exhaustive numerical results to the appendix and retain only the most essential comparisons to highlight our core contributions.

---

Thank you for your encouraging review. We hope our responses have further clarified the points you raised.