Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment

Q1 The paper overuses formula derivations while glossing over specific methodological details, making the overall writing quite confusing. Especially in section 3.2 and equation 5, there’s a lack of necessary logic, making it difficult to understand.

A1 Thank you for your valuable feedback and for pointing out the specific issues with Section 3.2 and Equation 5. We understand that an overemphasis on formula derivations, without sufficient explanation of the underlying methodology, can make the writing less accessible. To address this concern, we have made the following improvements in the revision version:

• Improved Logical Flow: We have revised Section 3.2 to better articulate the connection between the equations and the methodological details, ensuring that each step in the derivation is accompanied by an intuitive explanation. This makes it clearer how the formulas contribute to the proposed approach.
• Clarity for Equation 5: We have expanded the discussion surrounding Equation 5 to include the rationale and context behind its formulation.

Q2 I understand that the data construction method mentioned in the article still relies on fuzzy matching with ground truth based on entities. This seems to be a rather manual and primitive way of data construction, potentially introducing significant systematic errors. Have you tried more precise token selection methods?

A2 We believe there may be a misunderstanding regarding our methodology. Could you clarify what specific concerns or aspects you are referring to? I’d like to emphasize that we do not manually construct the data, nor do we rely on fuzzy matching with ground truth entities. Actually, we do not mention data construction in our paper. If there are specific concerns or aspects we might have overlooked, we would greatly appreciate further clarification to address your concerns comprehensively.

Q3 Overall, I find the experiments still insufficient. The experimental baselines are quite outdated, and the tested benchmarks are not comprehensive enough. Particularly, why is there a claim about handling hallucinations, but the final experiments lack an evaluation of hallucinations? Also, why is there almost no improvement on POPE? What’s the reason for this?

A3 Thank you for your valuable feedback on our work. We have carefully addressed your concerns as follows:

• Updated Baselines: We sincerely appreciate your suggestion to include more recent baselines. Following your feedback, we have added the latest baselines, including HA-DPO and RLAIF-V, in Table R1 below. As demonstrated in the updated results, our method exhibits competitive performance compared to these state-of-the-art approaches.
• Comprehensive Benchmarks: Thank you for pointing out the potential confusion in our benchmark setup. To address this, we have ensured our setup strictly follows the settings from LLaVA, encompassing a broad range of evaluation datasets. Specifically, the last three evaluation metrics in Table 2, such as CHAIR_S, CHAIR_I, and COCO_Caption, focus on assessing hallucination. To make this clearer, we have updated the table titles and explicitly noted that a smaller CHAIR value indicates better performance.
• Concern on POPE: We appreciate your careful observation regarding the performance on the POPE dataset. This dataset predominantly consists of simple QA tasks with single-word ground truth answers. In such cases, the difference between sentence-level and token-level rewards is minimal, limiting the advantage of our fine-grained approach. We acknowledge this findings and have included this experimental discussion in our revision.

Table R1: Performance Comparison Across Baselines

Thank you for your valuable insights. We have improved Section 3.2 and Equation 5 by enhancing logical clarity and providing detailed explanations. We added recent baselines (HA-DPO, RLAIF-V) and clarified hallucination-related benchmarks (e.g., CHAIR). Results in Table R1 underscore our method’s competitive performance. We look forward to your valuable feedback.

Q3 Regarding Section 3.1 Overall, this section looks a bit unclear to me. Firstly, the calculation process of token-level reward is not introduced, leading to confusion. Whether it is token-level CLIPScore? (additional question: whether per-sentence average of token-level reward is the same as sentence-level reward?) The authors state that sentence-level rewards show a weak correlation with conventional metrics, implying that sentence-level rewards are not effective. However, metrics like BLEU and ROUGE are known to have limitations in measuring semantic similarity, consider using CiDEr or SBERT semantic similarity score can make the results more convincing. Also, the paper does not clarify whether token-level rewards have a stronger correlation.

A3 Thanks for your response. We will address your concern individually.

Overall, this section looks a bit unclear to me. Firstly, the calculation process of token-level reward is not introduced, leading to confusion. Whether it is token-level CLIPScore? (additional question: whether per-sentence average of token-level reward is the same as sentence-level reward?)

Thank you for pointing out this issue and for highlighting the need for clarity in our explanation of the token-level reward calculation process. We sincerely appreciate your detailed feedback. In the Section 3.1 of revised version, we have clarified that token-level rewards are obtained by calculating the similarity between the token-level text embeddings and the image embeddings from the VLLM’s vision encoder. In contrast, sentence-level rewards are computed based on the similarity between the embedding of the entire sentence and the image embedding.

The authors state that sentence-level rewards show a weak correlation with conventional metrics, implying that sentence-level rewards are not effective. However, metrics like BLEU and ROUGE are known to have limitations in measuring semantic similarity, consider using CiDEr or SBERT semantic similarity score can make the results more convincing.

Thank you for your valuable feedback and for suggesting the use of alternative metrics such as CiDEr and SBERT semantic similarity score to strengthen our evaluation. We appreciate your observation regarding the limitations of BLEU and ROUGE in measuring semantic similarity. We have conducted additional experiments using metrics such as CiDEr and Meteor score to provide a more comprehensive evaluation. These results, originally presented in Figures 8 and 9 of the Appendix, have been moved to Figure 2 in the main text to address the reviewer's concern.

Also, the paper does not clarify whether token-level rewards have a stronger correlation.

To address the reviewer's concern, we calculate the average sum of token-level rewards in a sentence and explore its relationship with conventional evaluation metrics at the sentence level. As shown in Figure 10, we compare the correlation between sum of token-level rewards and conventional evaluation metrics with the correlation between sentence-level rewards and conventional evaluation metrics. We observe that, compared to sentence-level rewards, token-level rewards exhibit strong correlation with conventional evaluation metrics.

Q4 Issues with Clarity and Readibility

A4 Thank you for your feedback. We have revised the manuscript to clearly define symbols like $S$ and $\mathcal{N}$ in Section 3.3.2 when first introduced, and restructured the section 3.1 so that reward concepts are introduced before Figure 1.

Q1 The theoretical analysis assumes that the CLIP provides perfect vision-langauge alignment, which is a bit unrealistic as CLIP models are known to have alignment errors in practice. Under this assumption, it’s unsurprising that it can be proved that incorporating the CLIPScore improves the performance. Also, the presented theorem cannot support the benifit of fine-grained reward, which is the core point of this paper. Additionally, the framework also relies on linear relationships between inputs and outputs and the infinite data setting.

A1 Thank you very much for your thoughtful feedback and for highlighting this crucial point. We sincerely appreciate your attention to detail. You are absolutely correct that the assumption of perfect vision-language alignment in CLIP is not entirely realistic, as alignment errors are well-documented in practice. This simplification, however, is widely adopted in related studies [1, 2] to enable a more manageable theoretical analysis. Without this assumption, a rigorous proof becomes significantly more challenging. However, this assumption provides a tractable starting point to analyze the impact of vision feedback on VLLM performance. We acknowledge your observation that the presented theorem does not directly support the benefits of fine-grained rewards, which is a core focus of this paper. To address this, we have included a new proof in Appendix A.3.2 in the revised version that explicitly demonstrates how token-level rewards contribute to enhanced performance, providing stronger theoretical support for our claims.

[1] Nakada, R., Gulluk, H. I., Deng, Z., Ji, W., Zou, J., & Zhang, L. (2023, April). Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference on Artificial Intelligence and Statistics (pp. 4348-4380). PMLR.

[2] Zhou, Y., Fan, Z., Cheng, D., Yang, S., Chen, Z., Cui, C., ... & Yao, H. (2024). Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622.

Q2 The introduction of the MDP perspective seems unnecessary and is not well-justified within the context of the paper. Framing language generation as an MDP may not capture the long-range dependencies and context inherent in language, potentially violating the Markov property. At the end of Section 3.3.1, the author state that “This perspective highlights how fine-grained rewards can be applied ...”--I am unsure about how introducing the MDP framework demonstrates this point.

A2 Thank you for your insightful feedback! To address your concerns, we present the point-to-point response as follows.

The introduction of the MDP perspective seems unnecessary and is not well-justified within the context of the paper.

Thanks for your insightful feedback to help us improve clarity of our presentation. Introducing the MDP framework is crucial for our method because it provides a formal structure to apply fine-grained (token-level) rewards during the generation process. By modeling the VLLM as an agent in an MDP, we can:

• Define States and Actions: The states represent the current context or history $y_{<t}$, and the actions are the possible next tokens $y_t$. This precise definition is essential for applying RL algorithms that operate on state-action pairs.
• Apply Rewards at Each Decision Point: The MDP framework allows us to assign rewards at each time step t, corresponding to each token generated. This is particularly important for our fine-grained feedback mechanism, where we want to encourage or discourage specific token-level outputs based on the alignment with the visual content.
• Leverage RL Algorithms Effectively: By framing the problem within an MDP, we can employ standard RL optimization techniques like PPO to adjust the policy based on the rewards received. This facilitates stable and efficient training.

Framing language generation as an MDP may not capture the long-range dependencies and context inherent in language, potentially violating the Markov property.

We appreciate your thoughtful observation regarding the limitations of framing language generation as a Markov Decision Process (MDP), especially concerning the capture of long-range dependencies and context inherent in language, which might indeed challenge the Markov property. However, we believe our approach holds considerable merit. Recent developments suggest that MDP frameworks can be refined and extended to effectively account for long-range dependencies in language generation, offering a robust foundation for modeling such complexities[1,2,3].

[1] Liu G, Ji K, Zheng R, et al. Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization[J].

[2] Cao Y, Sheng Q Z, McAuley J, et al. Reinforcement learning for generative ai: A survey[J].

[3] Wu J, Ning L, Liu L, et al. RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs[J].

Thank you for your thoughtful comments. We have clarified token-level reward calculations and their distinction from sentence-level rewards (Section 3.1), incorporated CiDEr and SBERT for better semantic evaluation, and improved clarity of Section 3.3.2 symbols and structure. Results now show stronger token-level reward correlations (Figure 10). We look forward to your feedback.

Q2 Efficiency of FiSAO should be analysed as well. And how the dependency of pretrained vision model affects the performance?

A2 Thank you for your valuable feedback! We have conducted experiments to analyze the efficiency of FiSAO, with the detailed results provided in Table R1. A comprehensive analysis of these findings can be found in Appendix A.2.7 in the revised version.

Table R1: Computational Efficiency Across Methods

When excluding inference time, our method is more efficient than others, with a faster reward computation process. For the overall time (including inference), the efficiency of our approach is comparable to POVID and VLFeedback. However, our method delivers better performance due to the use of fine-grained token-level rewards, highlighting its practical advantages despite similar overall efficiency.

Additionally, we sincerely appreciate you bringing attention to the impact of the pre-trained vision model on FiSAO's performance. To explore this, we conduct two experiments:

• a: We adopt DINO-v2 as the reward model to provide fine-grained feedback for training LLaVA. From Table R2, it is evident that our method (Fisao (Dino)) achieves superior performance across multiple metrics, such as the highest MME$^{\mathrm{P}}$, MME$^{\mathrm{C}}$, and SEED values, while significantly reducing CHAIR${\mathrm{S}}$ and CHAIR${\mathrm{I}}$. The corresponding analyses and results are included in the revised version under Section A.2.5.

Table R2: Performance Comparison Across Baselines

• b: Due to the time limitation we are unable to train Fisao using SigLip scores as rewards. However, to address the reviewer's concern, we visualized the distributions of hallucinated and correct tokens using SigLip scores, as shown in Figure 12.** A detailed analysis has been included in the revised version in Appendix A.2.5.** We observe that SigLip demonstrates better differentiation between hallucinated and correct tokens compared to CLIP. This suggests that our approach is potentially highly adaptable when applied to MLLMs incorporating multiple visual encoders. Regarding the modifications or considerations necessary to handle such configurations effectively, the advantage of our method is its simplicity and flexibility. By following the reward calculation approach in Equation (8), our framework can seamlessly incorporate feedback from any visual encoder, including SigLip or others. This adaptability makes our method versatile and effective across various settings and encoder architectures.

Q1 In the comparison with other state-of-the-art methods, FiSAO did not achieve results on MM-Vet that are comparable to leading approaches. It would be beneficial to include an analysis of the factors contributing to this outcome and to outline potential strategies for improvement in future iterations. Understanding the specific challenges faced in this context could guide refinements to FiSAO and the design of models.

A1 Thank you for your thoughtful feedback and for highlighting this important observation. We greatly appreciate your recognition of the need to analyze FiSAO's performance on MM-Vet, as this helps us identify areas where our approach can be refined. MM-Vet primarily involves simple QA tasks, where the ground truth is often a single word, making it difficult to leverage the advantages of token-level rewards. We believe this represents an interesting direction for future exploration.

We appreciate your detailed feedback. We have analyzed FiSAO's efficiency and provided computational overhead results (Table R1). We also explored the dependency on pre-trained vision models with DINO-v2 and SigLip (Appendix A.2.5). These findings underscore FiSAO's adaptability and effectiveness in diverse multimodal configurations. We look forward to your further insights.

Q1 Table 1 does not seem to be very representative of relevant literature. Neither the motivation nor the related work covers existing methods for NLP token-level reward [1,2]. Would be nice to also cover concurrent works for VLMs [3,4] so that differences with recent work are better highlighted (albeit not taken into consideration as a weakness in my review, to the contrary, this shows this area of research is very important).

A1 Thank you for your constructive feedback. In response, we have expanded related work sections in Appendix A.4.3 to discuss these references, highlighting the differences and contributions of our approach more comprehensively. We greatly appreciate your insightful comments, which have helped us improve the quality of our work.

Q2 It is unclear why not use OpenCHAIR which addresses the limitations of CHAIR for benchmarking object hallucinations.

A2 Thank you for your valuable feedback. We have incorporated additional experiments using OpenCHAIR to benchmark object hallucinations. The results have been included in Table R1. We can observe that Fisao achieves the lowest CH$_i$ (9.6) and CH$_s$ (45.6), outperforming other baselines in reducing hallucinations.

Table R1: Performance on Object Hallucination Benchmarks (OpenCHAIR)

Q3 Figure 2 alone does not robustly support the observation that sentence-level rewards are unreliable indicators of model performance. To be exact, Figure 2 suggests there exists no linear relationship between the CLIP-based sentence rewards and the conversational metrics. Sentence-level CLIP scores might still have value, but in ways not captured by simple linear correlation with BLEU or ROUGE. Perhaps the figure could also show the linear correlation for token-level rewards to make a comparative argument, which would be stronger.

A3 Thank you for your thoughtful and detailed feedback! We greatly appreciate your attention to the nuances of our analysis and your suggestion to strengthen the comparative argument by including linear correlation for token-level rewards.

To address the reviewer's concern, we calculate the sum of token-level rewards in a sentence and explore its relationship with conventional evaluation metrics at the sentence level. We have made corresponding modifications and clarifications in the revised version. In Appendix 2.2.2, we compare the correlation between sum of token-level rewards and conventional evaluation metrics with the correlation between sentence-level rewards and conventional evaluation metrics. As shown in Figure 9, we observe that, compared to sentence-level rewards, token-level rewards exhibit strong correlation with conventional evaluation metrics.

Q4 Theoretical analysis assumes linear transformations and relationships between the latent representations of the image and text modalities (e.g., in the data generative model for v and t). It also relies on orthogonal matrices U_v and U_t​ for projecting latent variables to high-dimensional representations. While this simplifies the theoretical analysis, it assumes a degree of independence or decorrelation between features that may not exist in complex multimodal settings. There are other simplifications such as sub-Gaussian noise, Gaussian generative likelihood, an infinite data setting, and modeling token-level feedback as a regression problem, all of which understandably simplify (and make possible) the analysis but perhaps should be stated clearly as assumptions.

A4 Thank you for your detailed and perceptive feedback! We greatly appreciate your careful examination of our theoretical analysis and your thoughtful suggestions for clearly stating the assumptions made. In response to your observations, we have explicitly outlined these assumptions in the revised Section 3.2, ensuring that the degree of independence or decorrelation between features, the sub-Gaussian noise, and Gaussian generative likelihood are clearly acknowledged as necessary conditions for simplifying the analysis.

Thank you for your constructive comments. We have thoroughly addressed your concerns and added comprehensive discussions on related works (Appendix A.4.3), benchmarking with OpenCHAIR (Table R1), and clarified assumptions in theoretical analysis (Section 3.2). Detailed improvements are reflected in Figures 9, 12, and Tables R2-R3. We look forward to your further feedback.

Q6 Your approach primarily focuses on object hallucination. I am curious to know whether it also helps mitigate other types of hallucinations, such as those related to actions or spatial relationships. Additionally, could you elaborate on how the size of the expanded set C contributes to the observed performance improvements?

A6 Thank you for your insightful question! We greatly appreciate your emphasis on exploring how our approach might mitigate various types of hallucinations, as this broadens the understanding of its potential impact. We will address your concerns one by one:

Your approach primarily focuses on object hallucination. I am curious to know whether it also helps mitigate other types of hallucinations, such as those related to actions or spatial relationships.

While our approach primarily targets object tokens, it inherently enhances the overall vision-language alignment in VLLMs, which can indirectly help reduce hallucinations related to actions or spatial relationships to some extent. By ensuring that the objects mentioned in the generated text are present in the image, the model develops a more accurate grounding in the scene, providing a foundation for better action and spatial relationship reasoning. We acknowledge that our method may not fully address hallucinations involving actions or relationships, as these areas rely on more complex contextual reasoning that pre-trained encoders are not specifically optimized for [1,2]. We have addressed this limitation in Appendix A.5 of the revised version and added it as an avenue for future exploration.

[1] Kuo W, Cui Y, Gu X, et al. F-vlm: Open-vocabulary object detection upon frozen vision and language models[J]. arXiv preprint arXiv:2209.15639, 2022.

[2] Lewis M, Nayak N V, Yu P, et al. Does clip bind concepts? probing compositionality in large image models[J]. arXiv preprint arXiv:2212.10537, 2022.

Additionally, could you elaborate on how the size of the expanded set C contributes to the observed performance improvements?

Regarding the expanded set $C$, we agree that its size plays a crucial role in the performance improvements. A larger and more comprehensive $C$ enables the model to more robustly identify and align object tokens, which enhances its ability to understand and generate contextually appropriate descriptions. However, when the set is expanded beyond a certain point, the performance improvements become limited.
To substantiate this point, we compare the model's performance before and after the expansion of $C$ in Table R2. The results indicate that the expanded set $C$ leads to noticeable improvements in LVLM performance across various benchmarks. However, the subsequent performance gains become marginal when we further add additional object labels to $C$. This suggests that the original set $C$ covers a sufficiently comprehensive range of common objects, and further expansion may not significantly enhance the model’s capabilities.

Table R2: Performance Impact of Expanded Set $C$ on LVLM Benchmarks

These findings highlight that while expanding $C$ improves performance, the impact diminishes once the set becomes sufficiently inclusive of common objects.

Q5 Currently, new MLLMs leverage not only CLIP-based models but also other visual encoders, such as DINO-v2 (self-supervised) and SigLip. How adaptable is your approach when applied to MLLMs that incorporate multiple visual encoders? Could you explain any modifications or considerations necessary for your method to handle such configurations effectively?

A5 We are deeply grateful for your feedback. It’s excellent that you’ve highlighted the potential for adapting our approach to MLLMs with multiple visual encoders, such as SigLip and DINO-v2. To explore this, we conduct two experiments.

• a: We adopt DINO-v2 as the reward model to provide fine-grained feedback for training LLaVA. The results are presented in Table R1, and corresponding analyses and results are included in the revised version under Appendix A.2.5.

Table R1: Performance Comparison Across Baselines

• b: Due to the time limitation we are unable to train Fisao using SigLip scores as rewards. However, to address the reviewer's concern, we visualize the distributions of hallucinated and correct tokens using SigLip scores, as shown in Figure 12 in Appendix A.2.5. A detailed analysis has been included in Appendix A.2.5 of the revised version. We observe that SigLip demonstrates better differentiation between hallucinated and correct tokens compared to CLIP. This suggests that our approach is potentially highly adaptable when applied to MLLMs incorporating multiple visual encoders. Regarding the modifications or considerations necessary to handle such configurations effectively, our framework can incorporate feedback from any visual encoder, including SigLip or others, by adhering to the reward calculation approach outlined in Equation (8). This adaptability makes our method versatile and effective across various settings and encoder architectures.

Q4 In Figure 4, you illustrate the comparison of reward distributions for generated objects on LLaVA-1.5 before and after training. It seems reasonable that training the model with an RL algorithm on the same reward function results in higher average rewards. However, I am unclear about the purpose of showing the shift in reward distributions. Could you elaborate on the rationale behind presenting this distribution shift? Additionally, could you provide a detailed explanation for the design of Equation (8) and the reasoning behind its specific form?

A4 Thank you for your insightful question! We greatly appreciate your attention to the rationale behind presenting the shift in reward distributions and the design of Equation (8), as it helps us refine our analysis. We will address your comments sentence-by-sentence as below:

However, I am unclear about the purpose of showing the shift in reward distributions.

The purpose of showing the shift in reward distributions in Figure 4 is to illustrate how our method improves the alignment between the visual and language modalities of the model. Before training, the model tends to generate objects with lower reward values. This indicates that the outputs are not well-aligned with the preferences encoded in the reward function. After training with our method, the reward distribution shifts to the right, meaning that the generated objects have higher average rewards. This shift reflects improved alignment and demonstrates that the model learns to generate objects that are consistent with the image descriptions. By visualizing this change, we aim to highlight the effectiveness of our approach in enhancing vision-language alignment and producing higher-quality outputs. In the revised version, we have refined the description in Section 4.3 to ensure the rationale for presenting the distribution shift is explicitly stated.

Additionally, could you provide a detailed explanation for the design of Equation (8) and the reasoning behind its specific form?

Equation (8) calculates fine-grained rewards to help the model refine its outputs based on the visual input. The reward function evaluates each generated token's score, which indicates how well the token matches the visual input $v$, against two reference scores: the average score for correct objects $\mu_{\text{gt}}$ and the average score for hallucinated objects $\mu_{\text{hal}}$. These reference scores are obtained from the model's visual encoder. To distinguish correct outputs from incorrect ones, a margin $\lambda$ is added to create thresholds. If a token’s score is below $\mu_{\text{hal}} - \lambda$, it gets a negative reward, suggesting it is likely incorrect. If the score is above $\mu_{\text{gt}} + \lambda$, it gets a positive reward, indicating it aligns well with the visual input. Tokens with scores between these thresholds receive no reward, as their accuracy is uncertain. We have included a detailed explanation of Equation (8) (Equation (12) in the revised version ) to clarify its specific design and the reasoning behind its form.

Q1 In Figure 2, you show that the correlation between CLIP-based sentence rewards and conventional evaluation metrics is weak. Could you also present the correlation between token-level rewards and conventional evaluation metrics?

A1: Thank you for raising this insightful question. We greatly appreciate your question, as it provides valuable insights to improve the presentation of our work.
In response to your concern, we calculate the average sum of token-level rewards in a sentence and explore its relationship with conventional evaluation metrics at the sentence level. We have added a detailed analysis in the revision in Appendix A.2.2 and present visualization results in Figure 10. For the visualization experiments, we follow the same settings described in Section 3.1. As shown in Figure 10, we observe that, compared to sentence-level rewards, token-level rewards exhibit strong correlation with conventional evaluation metrics.

Q2 In the Table 3 experiment, it would be valuable to see the performance improvements across all baselines when using your approach. While line 429 mentions that the LLaVA backbone with your method surpasses existing approaches, it is worth noting that the original LLaVA-1.5 already performed well compared to other models. Demonstrating consistent improvements across various VLLMs could serve as strong evidence of your approach’s generalizability.

A2 Thank you for pointing out this important observation. We agree that demonstrating consistent improvements across various VLLMs is crucial for validating the generalizability of our approach. We have incorporated it and added new experimental results for InstructBLIP, presented in Table 3 of the revised version. As shown in Table R1, our approach achieves consistent improvements across multiple VLLMs, including LLaVA-1.5 and InstructBLIP, further demonstrating the generalizability and effectiveness of our method.

Table R1: Performance Comparison Across VLLMs

Q3 It is clear from Table 3 that your approach does not allow LLaVA-1.5 to surpass the higher-performing baselines on the MM-Vet benchmark. Additionally, Table 4 shows that fine-grained rewards do not yield an advantage on some benchmarks, including MME^c, POPE, and CHAIR. I am particularly interested in understanding which types of benchmarks or backbone architectures benefit most from your approach and where it shows superior performance.

A3 Thank you for highlighting these critical points and for pointing out areas where our approach has limitations. Your feedback has helped us improve the clarity of our work. Below, we provide our responses to address your concerns, organized based on different benchmarks:

• MM-Vet and POPE: These data primarily involves simple QA tasks, where the ground truth is often a single word, making it difficult to leverage the advantages of token-level rewards. This limitation could be further explored in future work, such as incorporating new reward computation methods that extend beyond object-based considerations.

• CHAIR: For CHAIR, as shown in Table 4 in our paper, the performance on CHAIR metric improves after using fine-grained rewards.
  The reviewer’s confusion might stem from the fact that we did not clarify that a smaller CHAIR value indicates better performance. We apologize for any confusion caused by the lack of clarification regarding the interpretation of CHAIR values. We have addressed this issue in the revised version by adding arrows in all the Tables for clarification.

We sincerely thank you for your constructive comments, which have significantly improved the clarity and depth of our work. We have carefully addressed your concerns in the rebuttal, supplemented by additional detailed experimental results, visualizations, and expanded analyses, as summarized in Appendix A.2.2 and A.2.5. We look forward to your valuable feedback.

My biggest concern is that the improvement due to the algorithm itself is limited. From the results in Table R2, the performance improvement appears marginal compared to the original LLaVA. A naive approach of augmenting the data used for preference fine-tuning with the Expanded Set C could potentially yield similar results. I am unable to identify a clear advantage over this naive approach. As a result, I will maintain the original score.