PerPO: Perceptual Preference Optimization via Discriminative Rewarding

Q1: In Table 1, the performance of the LLaVA-Next model with PerPO shows a noticeable decrease. Could the authors explain the reasons behind this performance drop?

We are uncertain about the performance drop you mentioned: whether it refers to the decreased improvement of LLaVA-Next over LLaVA-v1.5 under the PerPO setting or the decline in LLaVA-Next’s metrics on the RefCOCO+ validation set. We address both phenomena separately in the following text.

• LLaVA-Next, enhanced with augmented training data and a refined image processing paradigm, naturally exhibits commendable object grounding capabilities. This, to some extent, constrains the potential improvements achievable through post-training. In contrast, LLaVA-v1.5, owing to its comparatively weaker inherent capacities, offers ample room for enhancement via PerPO. In summary, the effectiveness of PerPO training is closely tied to the model’s innate capabilities. Nonetheless, PerPO consistently delivers significant improvements over SFT and DPO across all configurations.

• Despite our method’s general superiority, we noted a performance dip on the RefCOCO+ validation set. Analysis revealed two likely causes:

  1. Underrepresentation in Preference Data: RefCOCO+ constitutes less than 25% of our constructed preference dataset, significantly lower than RefCOCO and RefCOCOg. This disparity may lead to suppression of its unique representation.
  2. Distinct Prompt Distribution: RefCOCO+ exhibits a markedly different prompt distribution compared to RefCOCO and RefCOCOg. Specifically, RefCOCO and RefCOCOg focus more on queries incorporating spatial relationships (e.g., indicating an object’s location on the left side of an image). In contrast, RefCOCO+ emphasizes queries about absolute object positions.

  The combination of this training discrepancy and the smaller representation results in a degree of distribution shift during evaluation. Nevertheless, the model’s consistent performance on RefCOCO+'s testA and testB sets demonstrates its inherent robustness.

Q2: Could the authors provide results on the effect of scaling up the model on the discriminative reward?

According to your suggestions, we have supplemented the experimental results on the LLaVA-Next-13B model. The comparative results are shown in the table below. As can be seen from the table, the model continues to achieve a definite advantage as its scale increases. Due to constraints on time and computational resources, we will supplement our study with experiments on LLaVA-Next-34B in future work. However, we posit that our methodology is decoupled from the model’s parameter count, and thus can be effectively applied to models of any scale.

We sincerely appreciate your careful and constructive reviews. We have responded to each of your comments and questions as follows:

W1: The paper argues that the primary limitation in the visual discrimination capabilities of MLLMs is due to a lack of explicit perceptual alignment, but it does not provide sufficient evidence to support this claim. There is no experimental evidence that this issue arises from the alignment stage rather than other factors such as instruction tuning or an insufficient pre-training dataset. I recommend that the authors present more compelling evidence to confirm the assertion that the main challenge in the "limitations in MLLMs' visual discrimination" is mainly related to "perceptual alignment."

Thank you for pointing out this issue. Indeed, we would like to emphasize the following two points:

1. We acknowledge the importance of pre-training and instruction fine-tuning for visual discrimination. However, we emphasize that PerPO aims to push the boundaries and further unlock perceptual potential. Despite extensive training, many models, like GPT-4o, struggle with visual tasks such as object grounding and boundary drift, highlighting the limitations of current models. In deterministic tasks like mathematical reasoning, language models leverage strategies like diverse negative supervision [1] or fine-grained process rewarding [2] through RLHF to exploit intrinsic capabilities. This analogy supports our approach. Moreover, as shown in Figure 1b, the Best-of-N results reveal the model’s greater potential, validating our method. Therefore, we propose PerPO to activate the visual perceptual potential acquired during pre-training and instruction tuning, addressing gaps left by existing methods.

2. Compared to pre-training and instruction fine-tuning, PerPO demonstrates significant advantages in data <u>construction costs</u> and data <u>utilization efficiency</u>. First, while pre-training and instruction fine-tuning enhance model perception, they demand large, precise datasets. In contrast, PerPO employs a scalable approach for rapid, cost-free data acquisition based on temperature sampling and discriminative rewarding. Second, as shown in the table below, doubling dense OCR data from 25k to 50k yields a slight edit distance improvement from 0.67 to 0.64. However, PerPO, using only 1.8k preference data for LLaVA-Next-25k-7B model, significantly reduces the edit distance to 0.58. This demonstrates PerPO’s ability to substantially boost performance while minimizing training costs.

   Methods	Edit Dist ↓	F1 ↑	Prec ↑	Rec ↑	BLEU ↑	METEOR↑
   LLaVA-Next-25k-7B	0.67	0.47	0.71	0.37	0.16	0.28
   + PerPO	0.58	0.54	0.73	0.44	0.23	0.36
   LLaVA-Next-50k-7B	0.64	0.51	0.74	0.41	0.18	0.31
   + PerPO	0.56	0.56	0.75	0.46	0.24	0.36

References

[1] Shao et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. [2] Setlur et al. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning.

Dear Reviewer oqhZ,

We sincerely thank you for your thorough review and the professional insights you have provided. We value your feedback greatly, and we hope that our responses have effectively addressed your concerns.

As the discussion period is nearing its end, we would like to remind you that if there are any remaining issues requiring clarification or resolution, please do not hesitate to reach out. We are committed to addressing any questions you may have.

Should you find that our responses have satisfactorily resolved your concerns, we would be truly grateful if you could consider a higher rating, as it is crucial to the final evaluation of our submission. Thank you once again for your time and dedication.

Best, Authors

We sincerely appreciate your careful and constructive reviews. We have responded to each of your comments and questions as follows:

Q1: Could you provide a more detailed explanation of the specific enhancements in MLLM's visual discrimination capabilities by PerPO?

To address your inquiries, we conducted a more detailed analysis of PerPO's impact on both specialized and general capabilities:

1. OCR Performance Analysis: We categorized predictions on the dense OCR test set based on their edit distance from ground truth (GT): 0-0.5, 0.5-0.75, and 0.75-1.0. As presented in the table below, with PerPO, predictions with edit distances <0.5 increased from 18% to 23%, while those >0.75 dropped from 24% to 15%. This indicates that PerPO primarily reduces errors in the worst-performing cases, enhancing overall OCR performance by effectively constraining the poorest predictions.

   	0-0.5	0.5-0.75	0.75-1.0
   without PerPO	18%	58%	24%
   with PerPO	23%	62%	15%

2. General Capability Assessment: We compared models before and after PerPO alignment using GPT-4o and human judgments across three dimensions: response accuracy (RA), instruction adherence (IA), and hallucination reduction (HaR). Results indicate that PerPO mainly facilitates improvements in response accuracy and hallucination reduction. This demonstrates that our perceptual alignment encourages models to focus more on image content, effectively mitigating hallucinations stemming from image-unconditional responses.

   	RA	IA	HaR
   Win rate as judged by GPT-4o	80%	51%	66%
   Win rate as judged by human users	84%	53%	70%

We plan to include additional visualizations in the appendix to further elucidate PerPO's mechanisms and validate its effectiveness. We trust these efforts will address your concerns comprehensively.

Q2: How does the model perform when presented with complex visual tasks that require intricate discrimination capabilities?

To evaluate the model's performance on complex visual tasks, we expanded our testing to include more general visual task benchmarks such as VQAv2, MM-Bench, MM-Vet, and MMMU. The additional test results are presented in the table below. Evidently, our method demonstrates strong performance on both general and complex visual tasks.

Among them, MM-Vet is a widely used metric. It comprehensively assesses model performance across six domains: recognition, OCR, knowledge, language generation, spatial awareness, and mathematical reasoning. The following table details a comparison across various sub-tasks. Our method excels across multiple tasks, indirectly suggesting an enhancement in the model's perceptual capabilities.

We sincerely appreciate your careful and constructive reviews. We have responded to each of your comments and questions as follows:

Q1: While this paper provides many empirical results (e.g. the authors constantly referring to fig.1), the authors only explain their method with intuition, while a more in-depth or theoretical analysis is expected for a paper like this. Only empirical results are not convincing enough.

Thanks for your comments. Indeed, we have conducted a theoretical and in-depth analysis in the manuscript. And our theory is also built upon existing methods such as DPO and LiPO. Figure 1 is referenced to offer more robust support through experimental results. The more detailed explanation is given below.

• In Section 3, we have provided an in-depth analysis of the method from theoretical perspective. As mentioned from line 216 to line 241, we argue that the theoretical foundation underlying PerPO as a preference optimization method is its equivalence to Empirical Risk Minimization (ERM), which is widely regarded as highly effective in traditional computer vision. Simultaneously, from a practical standpoint, we substantiate the effectiveness of using discriminative rewards for perceptual preference optimization, as detailed in Section 5.
• Notably, the core of our method is to design a reward function for the perception of MLLMs, which is subsequently optimized using DPO and LiPO methods. Our theoretical framework is also built upon these established principles.

Q2: Since this paper simply uses the scalar task-specific discriminative score as the reward, why don't the authors compare their method to PPO while using the reward they defined? It seems equation (8) also seeks to optimize a list of accumulated scalar rewards like PPO did. Could the authors provide more thorough theoretical analysis on the difference of PerPO and PPO, especially in terms of optimization objective and smoothness?

Regarding your concerns, we provide explanations from the following perspectives. We hope this clarification meets with your approval.

1. Comparing with PPO may be unnecessary: The discriminative reward designed in our method can indeed be used for PPO training; however, this is not our primary focus. In fact, the key contribution of our paper lies in proposing a novel discriminative reward to enhance MLLMs' perceptual performance and employing a direct preference optimization method for training.

2. Comparing with PPO is unfair: PPO optimizes using environmental reward signals via policy gradients and value function estimation. Though broadly effective, it often requires substantial resources and tuning for complex tasks. DPO is more widely applied in MLLMs [1, 2, 3], employing direct optimization of user or system preferences to refine the policy. Compared to PPO, DPO offers certain effectiveness and stability while significantly conserving resources [4]. Our approach, based on direct preference optimization, differs significantly from the proximal policy optimization, making an equivalent comparison unfeasible.

3. The essence of PerPO's optimization objective: Equation (8) demonstrates that with our proposed reward, the optimization objective is equivalent to Empirical Risk Minimization (ERM). In other words, the essence of PerPO is to optimize discriminative rewards, which is not the same as PPO to maximize reward expectations through iterative strategy updates. This also theoretically explains why our method enhances the discriminative capabilities of MLLMs.

References [1] Wang et al., mDPO: Conditional Preference Optimization for Multimodal Large Language Models. [2] Liu et al., Mia-dpo: Multi-image augmented direct preference optimization for large vision-language models. [3] Zhu et al., Self-supervised visual preference alignment. [4] Ivison et al., Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback.

Q3: From Figure 2(b), the performance on RefCOCO+ seems to saturate after data size > 5k, and more such data can hurt general vision-language capabilities (shown by lowered LLaVA^(W) scores). Therefore, including more finetuning data does not seem to support PerPO in this scenario. Do SFT and DPO show similar trends? Is PerPO only better than SFT and DPO when the finetuning data scale is small?

Thank you for your thorough review and for raising professional questions. In the following text, we analyze the reason behind this phenomenon and provide additional experiments using SFT and DPO. We hope our response will meet with your approval.

• The reason for this phenomenon: As you mentioned, when the quantity of object grounding data significantly increases, the model's specialized ability improves, but its generalization ability decreases, which seems natural. Further training with extensive specialized data can severely shift the model distribution, leading to the forgetting of pre-training knowledge and a decline in performance on the LLaVA^(W). We believe that in this case, the general capabilities of the model also decreases under SFT or DPO training. Our approach, however, ensures that the model achieves excellent performance in both specialized and general capabilities with an appropriate amount of high-quality aligned data.
• The performance of SFT and DPO: To investigate whether DPO and SFT exhibit similar performance trends, we conducted additional experiments using datasets across different scales (3k, 5k, 7k, 9k, and 11k). For a fair comparison with PerPO, we also evaluated on LLaVA^(W) and RefCOCO+. The specific experimental results are presented in the tables below. From these, we can derive two insights.
  1. As the data size increases, SFT and DPO models exhibit a similar trend. They also forget previously learned knowledge and decline in general capability.
  2. PerPO outperforms both DPO and SFT across general and specialized capabilities, regardless of fine-tuning data scale.

Q4: The tested MLLM and visual perception tasks are somewhat limited (only LLaVA-1.5 and LLaVA-NeXT on object grounding and dense OCR). Thus, the generalizability of PerPO may be unclear. It would be better to include results on more MLLMs (e.g., Cambrian-1, LLaVA-OneVision, Qwen2-VL) and visual tasks (e.g., LISA or GLaMM on segmentation).

We sincerely appreciate your valuable suggestions. To validate the generalization capability of our method, we conducted experiments on other MLLMs for object detection, including LLaVA-OneVision [1] and Vary-toy [2]. The experimental results based on LLaVA-OneVision are presented in the table below, demonstrating that the PerPO method achieves a significant advantage over other comparative methods. Due to time constraints, the evaluation of Vary-toy is still ongoing, and we will update the experimental results in due course.

Although our method has been validated solely on object detection and dense OCR tasks, it is theoretically applicable to other discriminative visual tasks. We regret that we cannot provide quantitative results for diverse tasks due to time and computational constraints. Nevertheless, as highlighted in the Limitations and Future Work section, we plan to explore its applicability to broader tasks in the future.

References [1] Li et al., Llava-onevision: Easy visual task transfer. [2] Wei et al., Small language model meets with reinforced vision vocabulary.

We sincerely appreciate your careful and constructive reviews. We have responded to each of your comments and questions as follows:

Q1: The evaluation of general vision-language comprehension is based on LLaVA-Bench-in-the-Wild (LLaVA^(W)), a very tiny benchmark with less than 100 samples. The scores may not sufficiently reflect the MLLM image understanding ability. Larger, widely adopted benchmarks such as VQAv2 and MM-Bench are preferred.

Thank you for your thorough review and valuable suggestions, which have been instrumental in enhancing our manuscript. According to your feedback, we have conducted additional tests on VQAv2, MM-Bench, MM-Vet, and MMMU to evaluate the model's general visual capabilities. As presented in the table below, our PerPO method consistently surpasses SFT and DPO across these metrics. This further demonstrates that the general visual perceptual capabilities of MLLMs can be improved following PerPO training.

Q2: In Section 5.1, it is claimed that "discriminative reward also aligns well with human," but the results are evaluated by GPT-4o, not human users.

Thank you for pointing out this issue. We regret the omission of human user assessment in our prior work. Given the high consistency between GPT-4o and human judgments, we previously only employed GPT-4o directly to represent human preferences [1, 2, 3]. To address any potential discrepancies between GPT-4o and human users, we supplemented our analysis with human evaluations. For the selected 500 multimodal questions, we invited 20 experts and scholars specializing in computer vision, natural language processing, and human-computer interaction to provide independent assessments.

Specifically, for each question, we calculated average scores across three dimensions: response accuracy, instruction adherence, and hallucination reduction. The winning response was determined based on the magnitude of these average scores. Finally, we synthesized the evaluations from all 20 expert assessors and calculated PerPO’s ultimate win rate by aggregating their judgments. The evaluation results of GPT-4o and human users are presented as follows. Obviously, the results are similar, with our method achieving over 50% win rates across all test sets.

References [1] Wu et al., GPT-4o: Visual perception performance of multimodal large language models in piglet activity understanding. [2] Huang et al., Human-AI Collaboration Supporting GPT-4o Achieving Human-Level User Feedback in Emotional Support Conversations: Integrative Modeling and Prompt Engineering Approaches. [3] Kosaka et al., When ChatGPT-4o Is (Less) Human-Like: Preliminary Subjective Rating Tests for Psycholinguistic Research.