Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning

Dear reviewer gYxm,

Thank you very much for your positive feedback and insightful suggestions on our paper! We are delighted to hear that you appreciated the presentation of our work and our innovative method!

On the computational cost of LoRA

We’d like to note that the computational costs of our LoRA approach are comparable to standard supervised LoRA fine-tuning. A computationally similar scenario, RLHF, has been extensively studied and proven to work well, even in resource-constrained academic labs. Additionally, by switching between different LoRAs with the same base model, we only maintain one set of VLM weights, reducing VRAM consumption to approximately one-third of standard PPO training. Looking ahead, we are excited about future algorithmic improvements to PPO that will further reduce training costs.

On multi-task training

Thank you for pointing this out. Our current framework only allows training in a single environment but is not limited to a single task. For example, experiments in ALFWorld suggest that our framework can perform well in a multi-task environment containing different types, potentially shedding light on its multi-task learning capabilities (in a single environment) as well.

Regarding other weaknesses and questions

1. On the hyperparameter of the CoT tokens. We agree that manually setting a hyperparameter to adjust the log probabilities of the CoT tokens is not algorithmically elegant. This could be improved by providing an auto-tuning mechanism to handle the additional hyperparameter. Due to limited development time, we are unable to conduct such additional experiments during the rebuttal period, and we would like to leave this for future studies.
2. On exploiting the capability of MLLM. We agree that we have only provided a working version of the end-to-end RL training framework, without extensively exploiting the capability of the MLLM. Using our framework, one can extensively try other methods (e.g., different prompting techniques) to exploit the capabilities of the MLLM.
3. On the visual prompting. Thanks for the suggestion! While our current framework does not naturally allow visual prompting (VP), it is indeed interesting to see the performance of incorporating VP into our framework, we would like to leave this for future study.

Concluding remarks

We highly appreciate your appreciation of our work, suggestions for improvements, and potential future studies! Please be assured that we will include an additional paragraph to discuss the computational cost compared to RLHF, and the multi-task training capabilities in the updated version. Although we are only able to address 2 out of your 4 concerns, if you think addressing these 2 concerns or our additional experiments further improve the quality of our work, would you mind kindly improving the rating of our work? However, if you feel this is not sufficient, we understand, as the other suggestions you made are already very important future directions.

Dear reviewer QEzb,

Thank you very much for your high appreciation of our work, we are glad to hear that you appreciated the novelty, technicality, and performance of our paper!

Regarding the explanation of the limited performance on ALFWorld

Besides your appreciation, we would like to provide some potential explanations for the limited performance of the ALFWorld environment as you have mentioned in the weakness and question sections. The ALFworld environment is generally harder than the gym_card environment in these two following aspects:

1. Each game in the gym_card is a single-task environment, while the ALFworld is a multi-task environment, which substantially increases the difficulty for RL training in ALFworld, as the agent needs to learn more diverse actions based on the types of tasks. In fact, showing improvements in the ALFworld environment also sheds light on the multi-task learning capabilities of our framework.
2. In addition, the action space of ALFworld is generally much larger than the action space in gym_cards, which makes the learning process of ALFworld more difficult. More specifically, see Figure 17 on page 19 for example: (1) the agent is required to choose from more than 40 admissible actions at every step; (2) and at each step the admissible action set may be different.

Concluding remarks

We highly appreciate your suggestion for the additional discussion on the limited performance of the ALFWorld, and we will definitely include them in the discussion section. If you think our response addresses your concerns or our additional experiments improve the quality of this work, would you mind kindly further improving your rating? If not, please let us know as well, and we would like to further engage and improve our work based on your future suggestions! Thank you again for your appreciation and your insightful suggestions for improving our paper!

Thank you for providing a thoughtful response to my question. I could understand this paper more. I maintain my initial rating. By the way, if the authors will give a name to the proposed method, it will be easier for readers to refer to it. :-)

Dear reviewer RYx1,

Thank you very much for your high appreciation of our work. We are delighted to hear that you found our results in CoT and SFT insightful.

General response

In addition to your appreciation, we plan to incorporate the following results and discussions into the updated paper based on your valuable suggestions and questions:

1. We have conducted additional experiments on the recently released Cambrian-1 model, which has significantly better vision recognition capabilities. Detailed results are provided below.
2. We will include a more comprehensive explanation of the discussion on RL and SFT in Alfworld, with further details also provided below.

Experiments on additional VLM

Thank you for your suggestion. We conducted additional experiments using the recently released Cambrian-1, a VLM with enhanced visual capabilities, on the EZPoints and Points24 tasks. We provide all experiments using the same setup in the paper but with Cambrian-1-8b as our backbone VLM.

As shown in the table above, the final performance improves when the backbone VLM has better visual capabilities (EZP: 50.0% -> 54.0%, P24: 2.3% -> 9.1%), which further demonstrates that (1) our framework can adapt to other VLMs as well; (2) VLMs with better visual capabilities will enjoy better final performance. Note that we have not extensively swept the hyperparameters on Cambrian-1 due to the limited time, we believe the performance of Cambrian-1 can be further improved after more parameter sweeping.

To see how Cambrian-1 is better than Llava-1.6 quantitatively, we ran additional experiments to compare the number recognition accuracy in EZP and P24. More specifically, we evaluate the accuracy of the original llava-1.6-7b / Cambrian-1-8b checkpoints (0-shot) and the supervised fine-tuned (SFT) checkpoint. The results below are tested with the same prompt in the paper and averaged among 1000 trials. Each trial is considered correct, if all numbers (or cards) are correctly recognized. For example, in the EZpoint and Points24, a trial is considered correct if the VLM recognizes the numbers in all cards.

In conclusion, we showed that (1) our framework can still improve the performance of another VLM (Cambrian-1-8b); (2) and using Cambrian-1-8b, a mode with better visual capability, as the backbone will achieve much better performance compared to Llava.

Explanations on why RL does not improve across all the SFT tasks in ALFworld

Thank you very much for raising this insightful question. One potential explanation for this phenomenon is that the ALFWorld is a multi-task environment, where each task in gym_cards is trained in a single-task environment, which makes ALFWorld substantially harder than tasks in gym_cards. Following this line of thought, it would also be interesting to explore how RL can improve the multi-task learning capability of VLMs using the proposed framework. We sincerely thank the reviewer for raising this intriguing question, and we will include it in the discussion section in our updated draft.

Concluding remarks

We highly appreciate your suggestion for additional experiments on another VLM and discussions on the performance of ALFworld. We will definitely include them in the discussion section in the future update. If you think our additional experiments and discussion address your concerns, would you mind kindly further improving your rating? Thank you again for your appreciation and your valuable suggestions for improving our paper!

Dear reviewer MKrq,

Thank you very much for your valuable review and your questions! We will definitely integrate them into the updated paper to address your concerns.

General response

We sincerely thank you for your appreciation of our work and your insightful suggestions to improve our paper. The primary contribution of our work is to provide the first integrated system that enables end-to-end RL fine-tuning on VLMs, incorporating customized prompts as inputs. Hence, we directly utilized PPO, which has been well studied in RLHF, rather than comparing the effects of different prompts or RL algorithms within our framework, as these topics are not the primary focus of our paper. However, we highly value your suggestions regarding the effects of different prompts and comparisons of various RL algorithms. These are indeed important topics for future studies within our framework, and we will incorporate more discussions on these subjects as future work in the updated paper draft. Once again, we greatly appreciate your attention to detail and your thoughtful feedback.

To answer your questions:

Q.1:

Regarding the presentation in Figure 3.

In the appendix, it is observed that CoT not only provides thought processes but also manually encodes key states, such as the “current number” and “target number” in the NumberLine task (Figure 9). This crucial information is hidden in the main text (See Figure 3).

A: Thank you for bringing to our attention the presentation issue in Figure 3. We appreciate your careful review. The main purpose of Figure 3 is to provide a template for our expected input and output, for facilitating the post-processing described in Section 4.2, rather than detailing the design of each task-specific prompt. We apologize for any confusion this may have caused. To improve clarity, we will update the figure to include an additional key, "{optional task-specific prompts}," before "thoughts" to elaborate on the difference in our task-specific prompts carefully.

The authors need to clearly differentiate the source of CoT's effectiveness in improving reasoning—whether it stems from manually labeled key states or the textual thought process itself. This distinction should be experimentally validated and presented in the main text rather than being relegated to the appendix.

A: Once again, we apologize for any confusion in our presentation and thank the reviewer for highlighting this point. Our intention is not to compare the performance of different prompts, but to present an end-to-end RL fine-tuning paradigm that effectively utilizes customized CoT prompting for VLM. However, we agree that studying the effects of different customized prompts using intermediate reasoning steps [1,2] could potentially further improve the performance, and we would like to leave that for future research.

Q.2:

Moreover, regarding the CoT issue, Figures 11 and 13 show that CoT encodes not only the key states used for decision-making but also information "directly obtained from the formula in the image." There seems to be no reasonable justification for this, as theoretically, this information can be derivable from the images. An effective RL method should be capable of optimizing and identifying this critical information through rewards.

A: We sincerely thank you for your valuable suggestions to further improve the presentation of our work. We agree that the formula can be directly obtained from the image in principle; however, the backbone VLM (Llava-1.6b) struggles to recognize the formula from the images, which causes the entire pipeline to fail.

To demonstrate this, we provide additional evaluation results on the visual recognition capabilities of EZPoints, and Points24 in the Table below. In the table below, we have conducted additional experiments to evaluate the accuracy of the visual recognition for cards and equations in EZPoints and Points24, on the original llava-1.6-7b checkpoint (0-shot) and the supervised fine-tuned (SFT) checkpoint.

The results in the table below are tested with the same prompt in the paper and averaged among 1000 trials. For recognizing numbers, each trial is considered correct, if all numbers (or cards) are correctly recognized. For example, in the EZpoint and Points24, a trial is considered correct if the VLM recognizes the numbers in all cards. For recognizing equations, each trial is considered correct if the VLM successfully recognizes the equation in the image.

As shown in the table above, while the VLM can reasonably recognize the cards or numbers from the images, it encounters significant difficulties in recognizing the equations – even for EZPoint, the accuracy of recognizing equations is roughly 12% after supervised fine-tuning. Hence, if we ask the VLM to recognize both the numbers and the equation at the same time, the probability of the agent being able to recognize both the equation and the cards throughout the entire trajectory becomes very low – e.g., in the EZPoint environment, the probability of the agent that can successfully recognize the equations for at least 3 steps, is less than $0.12^3\approx0.0017$.

Page 1/ 2, to be continued.

Continued from page 1.

Therefore, our framework cannot provide any performance gain due to the limited visual capability of the backbone VLM. We have also briefly discussed such failure examples in Appendix B.5, where our framework fails when the backbone VLM is unable to recognize the key information for the task.

Please be assured that we will include the aforementioned table and an additional discussion on how the visual capabilities of the backbone VLM affect the final performance of our method in the updated version. We appreciate your suggestion for improving the presentation of our paper.

Q.3:

Did your baselines (GPT4-V and Gemini) include CoT information?

A: Thank you for the question. Yes, our experimental baselines with GPT4-V and Gemini use the same task-specific prompt for each task, as we discussed in line 711 of Appendix B.2.

Q.4:

To validate the effectiveness of the proposed RL algorithm, it would be beneficial to compare it with other RL methods, such as "OFFLINE RL FOR NATURAL LANGUAGE GENERATION WITH IMPLICIT LANGUAGE Q LEARNING" and "ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL." If using the authors' SFT model with your CoT, can we replace the proposed RL algorithm with these RL methods to achieve optimization?

A: Once again, we sincerely thank the reviewer for suggesting additional comparison experiments with ILQL [3] and ArCHer [4]. We agree that our end-to-end RL training framework can, in principle, adopt other RL algorithms besides PPO. Studying how different RL algorithms affect VLM training is indeed an important topic for future research. However, due to the limitations of computational resources and the nature of training large models, we are unable to provide results for more RL algorithms on VLMs during the rebuttal period. As mentioned in footnote 2 on page 6, each curve in our experiment takes 30 hours on 4 A100 GPUs to run. This time estimate only covers running the experiments and does not include the integration of a new RL algorithm into VLM training, which generally requires significantly longer development time (e.g., for this project, we spent more than 2 months developing the framework just for PPO). Similar computation cost issues have also been discussed in [3,4]. In section 5.7, section 6 of [4], and section 7 of [3], computational budget is a major bottleneck for comparing different RL algorithms on large model training. Moreover, our backbone VLM (llava-1.6-7b) is substantially larger than the backbone models used in ArCHer and ILQL. For example, ArCHer conducted most experiments on GPT-2, a 1.5b model (Section 6 in [4]), and ILQL conducted all experiments on GPT-2 (Appendix A.4 in [3]). We truly appreciate the reviewer’s suggestion to compare different RL algorithms within our framework, and we will definitely update our paper by discussing the comparison with different algorithms in future work.

Concluding remarks

We would like to thank the reviewer again for the insightful suggestions for improving our paper. Please be assured that we will update our paper to better (1) clarify the usage of our task-specific prompts in Figure 3; (2) add more discussion on the computational limitation for comparing with other RL algorithms; and (3) include the additional evaluation results of the visual recognition accuracy of llava-1.6-7b as presented before.

Please let us know if our response addresses your concerns, if so, would you mind kindly improving your rating for our work? If not, please let us know as well, and we would like to further engage and improve our work based on your future suggestions!

[1] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35 (2022): 24824-24837.

[2] Fu, Yao, et al. "Complexity-based prompting for multi-step reasoning." The Eleventh International Conference on Learning Representations. 2023.

[3] Snell, Charlie, et al. "Offline rl for natural language generation with implicit language q learning." The Eleventh International Conference on Learning Representations. 2023.

[4] Zhou, Yifei, et al. "Archer: Training language model agents via hierarchical multi-turn rl." ICML, 2024.

Page 2/2.

I would like to appreciate the authors’ faithful rebuttal. Here is my follow-up response:

"Our intention is not to compare the performance of different prompts, but to present an end-to-end RL fine-tuning paradigm that effectively utilizes customized CoT prompting for VLM. However, we agree that studying the effects of different customized prompts using intermediate reasoning steps [1,2] could potentially further improve the performance, and we would like to leave that for future research."

I do not mean to study different customized prompts. Mentioning "different customized prompts" is because your current implementation is problematic in this regard. When stating “We provide the first integrated system that enables end-to-end RL fine-tuning on VLMs”, at least we expect that the RL fine-tuning techniques enhance the visual understanding capabilities. Otherwise, it can be just a fine-tuning technique for LLMs. Unfortunately, I cannot find solid evidence in the current experiments to support this claim.

However, the backbone VLM (Llava-1.6b) struggles to recognize the formula from the images, which causes the entire pipeline to fail.

It is acceptable to report failed results. If your baseline also excludes this manually extracted information, they are under fair comparison. However, keep the information in the experiments gives the community a false sense of success and burdens future researchers who aim to solve this problem directly.

Suggestions: After removing those manually extracted information, perhaps the authors could demonstrate that RL training improves the accuracy of information extracted by VLMs from images. This would provide direct evidence that this is a viable RL training method for VLMs.