TLDR: Token-Level Detective Reward Model for Large Vision Language Models

W1: Although the paper introduces the concept of token-level feedback, this approach feels like a natural extension of existing reward modeling techniques, rather than a fundamentally new innovation…

A1: We politely disagree that TLDR is not a new innovation. Though fine-grained feedback has been explored in NLP, the current token-level reward functions are separated models from the LLM. There is no token-level reward model for on-policy RLHF training, let alone for vision-language models. We believe TLDR establishes the stage for vision-language on-policy RLHF training. We have further clarified the differences with previous work in the revised Sec 2.

W2, W3, W4: This paper didn't compare the proposed model with existing hallucination detection and mitigation methods, such as Woodpecker or other state-of-the-art techniques…The proposed hallucination evaluation didn’t show superiority over the existing hallucination evaluation methods.

A2: Thank you for directing us to other hallucination detection and evaluation literature. The main contribution of TLDR is serving as a token-level reward model. We are excited to show the token-level reward model can also help on hallucination detection, hallucination self-correction, and speed up human correction for the wrong words in captions. We agree it’s interesting to compare TLDR to other methods designed for hallucination mitigation. While hallucination self-correction is not our main focus, we will leave it for future study.

W5: Lack of comprehensive survey of hallucination on Large Vision-Language Models.

A5: Thank you for pointing out the missing references, we have added the listed references to the revised Sec 2 and Sec 5.3.

W6: Lack of evaluation for the synthetic data.

A6: For each taxonomy enumerated in Sec. 4, we manually inspect 30 examples with synthetic negative tokens. Taxonomy small_object introduces 1 example, where there was no small objects in the image, and the synthetic caption introduces an unnatural new object that is not small. Taxonomy spatial_relationship introduces 1 example, where the synthetic (negative) caption is a paraphrase of the original (positive) caption. All other taxonomies’ synthetic negative captions pass manual inspection that they both fit into the dedicated taxonomy and the perturbation is successful (i.e., not the paraphrase of the original caption).

W7: The models used to validate the self-correction with TLDR models are too few

A7: Thank you for the suggestion. We used Llama-3.2-Vision-11B and Qwen2-VL-7B as extra models for self-corrections, and we updated Table 4 in the paper as follows.

The results validate that TLDR can assist models in better self-correcting themselves than naive binary reward models.

Again, we are grateful for the reviewer’s thoughtful comments and suggestions. We are willing to answer any questions the reviewers may have during the discussion.

We are happy to see that the reviewer found our paper well-written and organized and appreciated the idea of developing a token-level reward model for LVLMs. We are also pleased that they recognized the potential applications of our model, including hallucination evaluation, self-correction, enhanced model performance, and its ability to significantly speed up caption annotation.

W1, W2: Backbone choices for TLDR

We want to have a lightweight reward model that can provide meaningful feedback for larger models, and PaliGemma's 3B size is a nice fit. Llama-3.2-Vision models release date (Sep. 25) was too close to the conference submission deadline (Oct. 1). During the rebuttal phase, in order to showcase the effectiveness of TLDR training, we ran experiments using Llama-3.2-Vision-11B as the backbone model for TLDR, and here are the llama-only results, with the full results updated in the revision and the global response.

Table 1: Evaluation of Model Performance

Table 5: TLDR automatically performs likelihood optimization

W1: Model choices for human evaluation and self-correction

Why we chose miniCPM: We want to have more samples where the model hallucinates. For bigger models like GPT-4o, the hallucination rate is too low to draw significant amount of samples. so we used captions generated by 7B models such as miniCPM. we conducted extra experiments with Phi-Vision and Qwen2-VL-7B, with a special focus on false negative (FN) type of errors, and averaged among three human annotators, we find the TLDR model has a sentence-level FN rate of 8.7%, 10.5% and 9.8%, on MiniCPM, Phi-Vision and Qwen2-VL-7B respectively.

Why we chose GPT-4V for self-correction: Most open-weight multimodal models still suffer from instruction following, such as correcting the captions. In this sense, we need a model that is able to self-correct provided with some signals. We didn’t choose GPT-4o because, again, it rarely hallucinates. To make the self-correction experiments more convincing, we apply self-correction with TLDR to two new models, Llama-3.2-90B-Vision and Qwen2-VL-7B, both of which can self-correct given some signals. Here are the results of them

We still find TLDR can assist these two models in better self-correcting themselves than naive binary reward models.

We thank the reviewers for their thoughtful comments and suggestions. We are happy to see that the reviewer appreciated the clear motivation behind our work and recognized the innovation in shifting to token-level feedback to address the limitations of binary annotations. We are also pleased that the clarity of our method section was acknowledged, as well as the interesting finding that fine-tuning a vision-language model with token-level rewards improves its overall performance.

W1: Whether TLDR can be called a reward model

In the context of PPO, a reward model refers to a separate machine learning model that estimates a numerical reward score for an agent's action or output, based on how well it aligns with the desired behavior. In short, we believe that as long as a separate model can provide positive and negative feedback, that is aligned with a desired behavior, it can be called a reward model. In our context, TLDR resembles the reward model setting for PPO training while giving token-level rewards. Although we didn’t perform large-scale RLHF, which could be very costly, we provide an approxy in Sec 5.4, whereas training the TLDR model secretly tunes the backbone models to align with the reward model head – which we call a likelihood optimization method. We believe this oversimplified trick is also an RLHF procedure that is less costly and could improve model performance, as shown in Tables 5 and 6. We also believe TLDR shed lights for future work in developing a complex and effective system using token-level rewards.

W2: Need human annotations for model quality.

We manually examined 100 Phi-vision generated caption sentences with at least one “bad” tag, only 10.5% of the sentences are actually correct (i.e., with at least one “bad” tag wrongly marked). Similarly, we conducted the human evaluations TLDR’s detection on Qwen2-VL-7B generated captions as well, and the false negative rate there is 9.8%.

W3: Evidence for Equation (8)

We highlight our person R correlation test in the caption of Table 3: the correlation score between $−\ log (\mathcal H_T)$ and MMMU score is 0.902 with a p-value of 3.45e-4, where $\mathcal H_T$ is the token-level hallucination rates. We also provide a new Fig. 5 in the appendix to show this linear trend that, given the models we evaluated (including newly added Qwen2-VL), $\mathrm{MMMU} = -6.51 \times \log(\mathcal H_T) + 3.96$. Such empirical finding induces our conjecture in Equation (8).

Again, we are grateful for the reviewer’s thoughtful comments and suggestions. We are willing to answer any questions the reviewers may have during the discussion.

Dear Reviewer KRLQ,

We would like to kindly remind you to take a look at our rebuttal and we are willing to discuss further to address any questions or concerns you may have. Thanks!

Best, Authors of TLDR

We thank the reviewer for their comments, kind words, and recognition. We are glad that the reviewer praised the shift from binary to token-level rewards for providing more precise and interpretable feedback, highlighted the novel hallucination rate metric, and noted the model’s utility in improving self-correction and speeding up data annotation. We are also happy to see that the reviewer shares the same insights with us about TLDR’s potential as a data correction tool.

Concerns about limitations in interpreting complex, text-rich images

In this work, we mainly studied natural images, and whether training on natural images can extend robustly to diagrams, OCR, and more complex text-rich images is still unknown. However, we believe that as long as the training and testing distributions are similar, i.e., extending TLDR training set to text-rich images, it should also succeed. We will leave this thoughtful extension to future works, and we thank the reviewer for pointing this out.

Effect of $\tau$

When $\tau = 1$, the LoRA $\alpha$’s for finetuning TLDR and running inference with TLDR backbone model are the same, so in this case, it’s the finetuned TLDR reward model, and its performance is shown in Table 1. Inspired by the reviewer’s question, we conducted extra experiments to show that backbone models, after tuned with TLDR, can reduce their hallucination rates significantly. We created a new Table 6 in the paper as follows,

The reviewer also asked why when $\tau$ increases beyond, for example, 0.25, the model performance decreases. Let’s take one step back to understand what $\tau$ represents. When $\tau = 0$, the merged model is the original backbone model (PaliGemma or Llama-3.2-Vision), while when $\tau = 1$, the merged model is the finetuned TLDR model’s backbone, which is optimized to provide token-level rewards. In this regard, $\tau$ can be viewed as the interpolation coefficient between a vision language model ($\tau=0$) and a reward model ($\tau = 1$). When $\tau$ is chosen as a good midpoint, it takes the benefit of both – can generate fluent and meaningful text, and can hallucinate less. When $\tau$ increases beyond, it shifts its behavior towards being a reward model, and its language abilities could decrease, resulting in lower scores (blink, isobench, and hallucination rates) that are calculated from text generations.

Again, we are grateful for the reviewer’s thoughtful comments and suggestions. We are willing to answer any questions the reviewers may have during the discussion.

Dear Reviewer PRNS,

We would like to kindly remind you to take a look at our rebuttal and we are willing to discuss further to address any questions or concerns you may have. Thanks!

Best,

Authors of TLDR

Reviewer f5mm also suggested more experiments with BLINK. Since the suggested visual correspondence task is multi-image, which is not naturally supported by many models, we conducted extra evaluations on the Object Localization task, together with our new Llama-3.2 backed TLDRs. We updated Table 5 as such

We find that the TLDR model with both PaliGemma and Llama-3.2 backbone can be improved with the token-level reward training for free.

Updated Hallucination Evaluation Benchmark Table

As reviewer f5mm also suggested, we should evaluate the Llava series and Qwen VL series. We decided to perform an extra evaluation on the Qwen2-VL family with 2B and 7B versions. We decided not to evaluate the Llava series since their training data were derived from GPT-generated texts, which is a clear violation of OpenAI’s terms of service. We updated Table 3 in the paper as follows,

As f5mm also pointed out, MMMU scores may not be fully representative of model performance. We chose MMMU as the score to compare because it has the most coverage of models so far. We add extra scores from MEGA-Bench [1]. As reviewer KRLQ suggested we should provide more evidence for Eqn. (8), and we first highlight our person R correlation test in the caption of Table 3: the correlation score between $−\ log (\mathcal H_T)$ and MMMU score is 0.902 with a p-value of 3.45e-4, where $\mathcal H_T$ is the token-level hallucination rates. We also provide a new Fig. 5 in the appendix to show this linear trend that, given the models we evaluated (including newly added Qwen2-VL), $\mathrm{MMMU} = -6.51 \times \log(\mathcal H_T) + 3.96$.

[1] Chen, Jiacheng, et al. "MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks." arXiv preprint arXiv:2410.10563 (2024).