FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback

We thank the reviewer for acknowledging our proposed benchmark and RLHF framework. We address the reviewer's concerns below:

• Clarification on out-of-sample generalization: For the human-feedback prediction model, we do not claim that it can generalize to out-of-distribution data. Instead, we claim that it shows out-of-sample generalization. We evaluated the out-of-sample generalizability of our human-feedback prediction model in the following experiments:

  1. We computed the statistics of the predictions of trained feedback prediction model on the train set with that of human-annotated data. We show the results below. From the table, we can see that the model predictions are similar to the human-annotated data.

  # Fig-Caption Pairs	Human Feedback	Median	Mean	Std	Min	Q1	Q3
  Actual	438
  		Helpfulness	3	3.01	1.19	1	2
  		Takeaway	2	2.16	1.22	1	1
  		Visual	2	2.11	1.08	1	1
  		OCR	4	3.83	0.80	1	4
  Predicted	106,834
  		Helpfulness	2.89	2.89	1.07	-1.27	2.17
  		Takeaway	1.95	2.06	1.03	-1.06	1.33
  		Visual	1.91	2.02	1.01	-1.23	1.31
  		OCR	3.88	3.84	0.83	0.018	3.32

  2. We also conducted a 5-fold cross validation experiment using our annotated samples. We constructed a train/val set of the original 438 annotated samples by randomly sampling 219 samples to be selected as train set and remainder as validation set. We report our results (mean squared error and standard deviation over 5 runs) below:

  Helpfulness: $0.082 \pm 0.12$ | Visual: $0.076 \pm 0.2$ | Takeaway: $0.087 \pm 0.17$ | OCR: $0.095 \pm 0.13$

• Baseline selection: Our main goal with this work is to provide a benchmark for developing and evaluating new methods that addresses the problem of model alignment with human preferences for vision-language tasks. To this end, we selected 10 baseline models from 3 different classes of models (OCR-only, Figure-only, and Figure-Caption), that have been shown to perform well for the general image-to-text tasks. We believe that our current baselines and RLHF method would provide the researchers with the knowledge needed to further develop better performing models and evaluate how different models perform when utilizing human-preference data. To this end, we believe our baseline models are justified and we hope that our proposed benchmark and RLHF method leads to further innovation in this area.

• Utility of Human-Feedback prediction model: When compared with the online RL formulation of RLHF, both our offline-RLHF formulation and the online-RLHF methods require a reward prediction model pre-trained on human-preference data. However, the key difference between the two approaches is that once we infer the reward signal for each figure-caption pair, in our proposed offline RLHF method, we don't require the reward model for policy optimization. We proposed the offline upside-down RL method for the RLHF formulation as it would allow other researchers to compare their image-captioning models easily without the need to setup and train separate reward models.

  We don't claim that our proposed RLHF formulation is better or worse than the online RLHF method. However, as shown in previous works [1] offline RL methods are more stable, scalable and easier to use than online RL methods. We believe that for researchers focusing on developing new models for image-to-text task, our RLHF framework allows a simpler framework for evaluating their models for the human-preference realignment tasks.

General Comment: We truly appreciate the concerns raised by the reviewer as we specifically focused on these questions when designing our benchmark. We ensured that we provide results with different baselines and thoroughly evaluated the feedback prediction model. We believe that our work is a strong step in benchmarking image-to-text models and RLHF methods. Thus we hope that reviewer positively reconsider their initial rating.

We thank the reviewer for acknowledging the strengths of our work. We address the concerns below:

• Confusion regarding framework description: We referred to the two components (Human Feedback Prediction model and Upside-Down RLHF method) as a framework because they are jointly used to align an image-captioning model’s output with human preference. The human feedback prediction model is used to generate human feedback scores which is then used as reward signal along with the state ( input figure image) and action (output figure caption) to define the single-step trajectory of experience in a contextual bandit environment [1]. The triplet of ( reward, state, action) is then used to update policy (image-to-text model) using the Upside-Down RL formulation [2]. However, to avoid confusion, we will modify the text to make it more clear how they share synergy.

• Novelty of the work: We highlight the novelty of our work below:

  1. Our work provides a new benchmark for developing and evaluating new methods addressing the problem of model alignment with human preferences for vision-language tasks. We believe that our benchmark would allow more in-depth evaluation of new RLHF methods as, currently, the majority of benchmarks addressing human-preference alignment problems are specific to NLP literature.
  2. Furthermore, we propose a novel baseline for a simple but efficient implementation of RLHF principles in the form of offline upside-down RL. Our proposed RLHF method is novel as it addresses the RLHF problem as an offline reward-conditioned behavioral cloning. We show that this novel formulation is not only easy to use, but also demonstrates strong performance on the proposed benchmark. Formulating the RLHF problem as offline-RL also allows us to efficiently evaluate different image-to-text models as we are not dependent on the reward model for policy optimization once we have generated the (reward, action, state) trajectory. This should enable researchers to easily evaluate their own caption-generation models without worrying about the reward model.
  3. We want to highlight that the main goal of this work is to provide a new benchmark for evaluating different methods that incorporate human feedback for human preference-aligned model output. To this end, our contribution of new benchmark with human preference data evaluating different models and proposing a simple but effective RLHF method should be positively considered.

• Narrow applicability: We agree with the reviewer that having a benchmark based on general image/video-to-text tasks is important. We specifically focused on figure-captioning task as one of our goals with the proposed benchmark is to allow researchers to evaluate different RLHF frameworks. To this end, we would like to obtain high-quality human-feedback data that is not ambiguous. For image captioning task with natural images, it is difficult to avoid ambiguity when collecting human preference data. This ambiguity emerges because it is difficult to evaluate different captions for a given natural image objectively. There could be multiple valid captions that correctly explains an image. Hence the human preference data can have high variance, making it difficult for downstream applications. On the other hand, scientific figures are more structured, and we can utilize domain experts to collect high-quality human preference data. This allows us to create a benchmark that can be used to accurately compare different models and RLHF frameworks.

  Nonetheless, our current framework can be easily extended to general image-captioning tasks, as we use general-purpose image-to-text models, and our proposed RLHF method can be applied to the new domain too. However, we consider this extension as out of scope because of the above-mentioned reason.

General Comment: We again thank the reviewer for acknowledging our work. We truly appreciate the concerns about the applicability of our method as we also share the same thoughts that a similar framework targeting general image/video captioning is important. We believe that our proposed benchmark is a strong step in this direction and would allow the researchers to efficiently benchmark their methods.

We hope that we answered the reviewer's concerns and that they positively reconsider their rating.

[1] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.

[2] Jurgen Schmiduber. 2019. Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions

We thank the reviewers for acknowledging the effectiveness of our method and the usefulness of our proposed benchmark. We address the reviewer’s concerns below:

• Computational Efficiency: We stated that our offline reward-conditioned behavioral cloning method is computationally efficient to online RL methods for human-preference alignment of model predictions in the context that, for once, we have a trained reward model (Human-feedback prediction model), and we have inferred the reward signal for individual (figure, caption) tuple to define our trajectory of (reward, state, action), we do not require the reward prediction model for optimizing the policy. Thus the model optimization step is only dependent on the image-to-text model selected as policy. On the other hand, for online-RL methods, we require the initial supervised fine-tuned model, the policy model, and the reward model together to optimize the policy. Thus, from the perspective of runtime and compute memory requirements, our proposed offline-RL method for learning from human feedback is computationally efficient.

  Furthermore, previous work [1], [2] in the context of natural language generation have highlighted that online-RLHF method is often unstable and substantially increase the complexity of the training setup because online methods often require more computational resources per step due to real-time interactions with environment and may require multiple gradient updates per step. Offline RLHF approach can be computationally more efficient per sample because they process a fixed dataset without the need for real-time interactions with the environment. We will clarify this better in the text.

• Novelty Concerns and Evaluation of reader-alignment after RLHF training: Based on our qualitative and quantitative, we do find that our proposed RLHF method does improve the quality of the captions. To further evaluate the generated captions, we conducted a small scale human evaluation experiment. Specifically, we randomly select 100 figures from the Test set of our proposed benchmark and generate captions using the BLIP and BLIP-RLHF models. We present the triplet of Figure, corresponding BLIP and BLIP-RLHF generated captions (after randomizing the order of the two captions) to 10 human subjects. Each human subject is asked to rank the two captions based on which caption they think is better. We ask the subjects to specifically consider helpfulness, visual-descriptiveness, OCR alignment and takeaway while ranking individual pair of captions. To guide the subjects, we first explain each metric [helpfulness, visual-descriptiveness, OCR alignment and takeaway] and present each human subject with 100 samples from our human annotated dataset with individual figures, groundtruth caption and the corresponding metric scores (recorded in 5-point Linkert scale). From our study, we find that on average 85% of the time, BLIP-RLHF generated caption was selected as the better caption relative to BLIP generated caption. From our small-scale study, we conclude that RLHF does improve the quality of the captions when compared to fine-tunning existing vision language models for the task of figure-caption generation.

  We want to highlight that our work provides a new benchmark for developing and evaluating new methods that addresses the problem of model alignment with human preferences for vision-language tasks. We believe that the availability of our benchmark would allow more in-depth evaluation of new RLHF methods as currently, the majority of benchmarks addressing human-preference alignment problems are specific to NLP literature. To this end, our contribution of a new human-preference alignment benchmark, evaluation evaluating models and proposing a simple but effective RLHF method should be positively considered.

General Comment: We thank the reviewer again for their helpful feedback. We hope that we were able to address the concerns. We believe that our benchmark, coupled with multi-dimensional human preference data encompassing various aspects of human preference, and the proposed offline RLHF framework, will provide researchers with valuable tools to develop and evaluate improved methods for both image-to-text models and alternative RLHF schemes. Thus we hope that reviewer positively reconsider their initial rating.

[1] Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. 2023. Offline rl for natural language generation with implicit language q learning.

[2] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.