MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?

To further clarify the human verification process, we proceed to provide a more detailed point-by-point explanation of the data curation below, along with a statistical summary in Table-r. 2.

• VLM pre-process: Specifically, as described in Appendix A in the paper, we first gather corresponding image pairs for each perspective through different algorithms we propose. This results in a substantial number of samples, with each perspective containing a similar quantity. Then our first step for quality control is to adopt a powerful VLM (LLaVa-NeXT-34B) to pre-process the data and filter out the wrong preference pairs (e.g. for the alignment subset, we only include those image pairs where the positive sample completely aligns with the prompt and the negative sample includes hallucinated entities or relationships). In this step, we aim to ensure the overall correctness of the image pairs, while not considering if they are challenging enough or have high quality. The samples we obtain in this process are 6260, 4852, 5964 pairs for the alignment, safety, and quality perspective, and 140 groups for the bias perspective.

• Human verification: Next, we engage human verifiers to evaluate each preference pair, considering both images alongside the corresponding prompt. Specifically, we ask 12 humans split into four groups to annotate for each subset. In this step, the verifiers are tasked not only with confirming the correctness of the pair (e.g., ensuring the chosen image in the alignment subset fully aligns with the prompt) but also with assigning a difficulty rating from 0 to 5. This rating reflects how challenging they perceive the pair to be, based on the premise that the reason for the preference is clear and verifiable. The greater the difficulty for the model to distinguish between the images, the higher the rating. This process results in 2,489, 2,271, and 1,680 validated pairs for the alignment, safety, and quality perspectives, respectively, as well as 105 groups for the bias perspective. All pairs are verified for accuracy by human evaluators, with each accompanied by a difficulty rating.

• Benchmark Confidence Sampling: Although the current dataset is verified and ready for use, its size poses significant computational and time-related challenges. To address this, we draw inspiration from [7], which suggests that usually only a subset of the benchmark samples are sufficient to provide a certified and reliable evaluation for each model. To implement this, we use three surrogate models (MiniGPT4-v1, InternVL-Chat-V1.2, and LLaVA-V1.2) to run inferences on the dataset, progressing from higher-difficulty samples to lower-difficulty ones. We then calculate the confidence interval (variance) of each model's performance on the dataset. Using a threshold of 0.1, we ensure that each subset contains sufficiently enough samples to provide a confident estimate of model performance within this interval. This approach not only ensures that the more diverse and challenging samples are prioritized, but also guarantees an efficient and sufficient sample size for evaluation while maintaining statistical reliability. As a result, we obtain 724, 574, and 1,121 validated pairs for the alignment, safety, and quality perspectives, respectively, as well as 18 groups for the bias perspective.

We then compile these samples to form the final evaluation set for each perspective in MJ-Bench. This rigorous quality control pipeline ensures that the collected samples and resulting evaluations are reliable, challenging, and efficient.

To demonstrate the quality of our dataset, we fine-tuned a text-to-image model (SD-1.5) directly using the preference pairs from MJ-Bench, showcasing the value of the data samples in our dataset. We compared this model with the SD-1.5 base model and the SD-1.5 model fine-tuned using GPT-4o feedback, with the results presented in Table-r. 3. Based on human judge feedback, the model fine-tuned with MJ-Bench significantly outperforms the one fine-tuned with GPT-4o feedback in alignment, safety, and bias perspectives, while achieving comparable performance in the quality perspective. This demonstrates the high quality and reliability of our dataset. Additional case studies and comparisons of the outputs from the three models are provided in Figures 7, 8, and 9 in the appendix.

(continued in our next response)

We hope the above additional clarifications of the related works and the unique contribution of our works have addressed the reviewer's concern.

[1] Chen, D., Chen, R., Zhang, S., Liu, Y., Wang, Y., Zhou, H., ... & Sun, L. (2024). Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. arXiv preprint arXiv:2402.04788.

[2] Wu, X., Huang, S., & Wei, F. (2024). Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation. arXiv preprint arXiv:2404.15100.

[3] Wang, H., Xiong, W., Xie, T., Zhao, H., & Zhang, T. (2024). Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts. arXiv preprint arXiv:2406.12845.

[4] Jiao, Q., Chen, D., Huang, Y., Li, Y., & Shen, Y. (2024). Img-diff: Contrastive data synthesis for multimodal large language models. arXiv preprint arXiv:2408.04594.

[5] Zhou, Y., Cui, C., Rafailov, R., Finn, C., & Yao, H. (2024). Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411.

[6] Lee, T., Yasunaga, M., Meng, C., Mai, Y., Park, J. S., Gupta, A., ... & Liang, P. S. (2024). Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36.

W2: The paper employs human evaluators for experimental evaluation multiple times, yet it fails to report the number of human evaluators involved. Since human evaluators may introduce bias, it is recommended to report this metric. If the number is small, it is advised to increase the scale of human evaluators.

We sincerely appreciate the reviewer for raising this concern and have updated the paper to include detailed information about the usage of human evaluators. To provide a direct answer, we employed human evaluators at two key stages:

1. Human Verification During Annotation: For data annotation, we engaged 12 human verifiers divided into four groups. Each verifier was assigned a specific portion of the images to verify, with no overlap between individuals, ensuring efficiency in the annotation process. While we acknowledge the reviewer's concern that relying on a single annotator per image might introduce bias, the large number of samples requiring verification (as detailed in Table-r. 2 below) makes this approach the most practical and efficient. Furthermore, we conducted a quantitative analysis by fine-tuning a model on our curated dataset, which also demonstrated the high quality and reliability of the dataset. We present the results in Table-r. 3 below.

2. Human Evaluation of End-to-End Fine-Tuned Models:
   For evaluating the end-to-end fine-tuned models, we again engaged the same 12 human evaluators. Each evaluator ranked six images generated by SD-1.5 models fine-tuned on feedback from six different judges. To reduce bias in this stage, each set of images was reviewed by at least three evaluators, and their rankings were averaged to produce the final results. This approach ensures the reliability of the evaluation while maintaining efficiency.

(continued in our next response)

To further expand the contribution and novelty of our paper, inspired by the VisionPrefer method which trains a reward model on their curated preference dataset, we designed an additional experiment where 80% of the MJ-Bench data was randomly split (except for bias, where we use 64 groups of the data filtered out from the confidence filtering stage specified below) to train a MoE-based judge model, following the method in [3]. The model incorporates four experts, each responsible for a specific perspective, with a gating layer to aggregate scores across each perspective trained via BT objective. Then we use the remaining 20% of the data as a test set. Results are reported in Table-r. 1.

From Table-r. 1, we observe that the MoE-based judge trained on MJ-Bench outperforms other models in alignment, safety, and bias perspectives in terms of w/ tie scores while being very close to GPT-4o on the quality subset. These findings highlight the advantages of MoE structures for handling multi-objective feedback and underscore the high quality of MJ-Bench data samples. However, the results also suggest that scaling up MJ-Bench, particularly in the quality subset, could further enhance performance, potentially surpassing GPT-4o. Due to time constraints, we plan to train our reward model on a larger held-out training set and evaluate it on the full MJ-Bench test set to compare against more models. Unfortunately, since the authors of VisionPrefer have not open-sourced the VP-Score model, we were unable to include it in our benchmark evaluation.

Table-r. 1 Additional evaluation results of our MoE-based judge model trained on a split from MJ-Bench. We evaluate and compare a subset of the models with best performance from Table 2 in the paper using the rest of the data as test set. The best performance are in bold.

Specifically, regarding the reviewer's concern about "an in-depth analysis of the selection of these four aspects," we would like to provide the following justifications:

• Focus on Critical Challenges in Image Generation Models:
  Our selection of the four aspects is rooted in the most critical challenges faced by image generation models, as identified in recent studies and benchmarks, such as [1]. These challenges can be broadly summarized into the following four key issues:

  1. Text-Image Misalignment, where models generate plausible entities that contradict the instructions (commonly referred to as hallucination).
  2. Unsafe Content, where outputs include harmful, toxic, sexual, or violent concepts.
  3. Low-Quality Generation, characterized by blurry or unnatural artifacts in the images.
  4. Bias and Stereotypical Outputs, where models produce biased results favoring or opposing certain demographic groups.

  Each of the "aspects" in our dataset is carefully curated to address these challenges, recognizing the critical role of feedback in evaluating and improving current image generation models.

• Comparison with Concurrent Works:
  Concurrent works, such as VisionPrefer, also focus on subsets of the aspects we address. For example, VisionPrefer studies "prompt-following," "harmlessness," "fidelity," and "aesthetics," which correspond to the "alignment," "safety," and "quality" perspectives in our benchmark. However, MJ-Bench uniquely includes an additional bias perspective, addressing a crucial issue that remains underexplored in current image generation models. Furthermore, unlike VisionPrefer, which focuses on four broad categories, MJ-Bench offers a more granular and exhaustive evaluation. Each perspective is further decomposed into multiple subcategories, providing a deeper and more comprehensive assessment of multimodal feedback.

(continued in our next response)

Dear Reviewer Rmzq,

Thank you so much for your valuable suggestions to help improve our paper! We deeply appreciate your recognition of the novelty of our work and its critical contribution to advance research in RLAIF for text-to-image generation. Below, we have provided a detailed, point-by-point response to your questions, and hope these could help address your concerns.

W1: The paper's novelty seems limited, as the whole pipeline has been proposed in the previous method. Besides, The selection of these four aspects lacks in-depth analysis.

We sincerely appreciate the reviewer’s concern and would like to provide a more comprehensive explanation to highlight the novel contributions of our work. First, we emphasize that while many studies have explored reward models and judge feedback, MJ-Bench is the first to deeply investigate and comprehensively benchmark these models’ feedback for image generation by introducing a large-scale, high-quality dataset with fine-grained categories—addressing a critical yet previously unexplored topic. To further clarify our unique contributions, we have addressed the perceived overlap with prior methods and provided a detailed response to the question of novelty and the rationale behind the selection of the four evaluation aspects.

While various benchmarks have been developed to assess the capabilities and limitations of general MLLMs, most of these primarily evaluate the generative capabilities of multimodal foundation models themselves rather than their capacity to serve as evaluative judges. Prior works have highlighted that FMs may perform significantly differently in generative tasks compared to classification or feedback tasks, such as providing reward signals. This distinction complicates the direct application of generative benchmarks to evaluative roles as a judge.

Several preliminary efforts, such as [1], have explored using FMs as judges, but these works primarily focus on textual responses from LLMs and VLMs, failing to account for their multimodal feedback in the context of image generation models. Similarly, concurrent work like VisionPrefer [2] investigates reward models for image generation but focuses solely on curating a dataset with four broad subsets, lacking the fine-grained granularity required for detailed multimodal feedback assessment. Other works, such as [4] and [5], attempt to improve text-image alignment with MLLM feedback but rely on preference datasets generated through simple heuristics, without ensuring the diversity, and quality of the datasets.

In contrast, MJ-Bench introduces several novel contributions that distinguish it from previous methods:

• First Benchmark on Multimodal Judge Feedback for Image Generation:
  While prior works have primarily focused on textual feedback for tasks such as text generation or image captioning, MJ-Bench uniquely evaluates feedback from multimodal judges in the context of image generation—an important yet underexplored task for the RLAIF community.

• Comprehensive and Fine-Grained Categorization:
  Unlike earlier benchmarks that only provide large-scale datasets with broad categories, MJ-Bench offers a large dataset with over 20 subcategories. Each subcategory captures a specific aspect of feedback from multimodal judges, enabling a more comprehensive and detailed evaluation. This granularity provides valuable insights for future research.

• High-Quality Annotation and Data Selection:
  MJ-Bench employs a rigorous data sampling and quality control pipeline, ensuring that every data sample is not only of high quality but also sufficiently challenging to serve as a benchmark. Our process includes VLM pre-processing, human verification, and confidence interval sampling to maintain both diversity and reliability.

• Comprehensive Evaluation Framework:
  As noted by the reviewer, we have conducted extensive evaluations, including:

  • Automatic metrics based on a large-scale fine-grained dataset of chosen-rejected triplets.
  • End-to-end human evaluations on models fine-tuned with such judge feedback via DPO/DDPO.
  • Feedback quality assessments at various feedback scales.
  • Consistency evaluation of model preferences across different input modes.
  • Human evaluations through multiple formats, including voting and ranking.
    In addition, we have included numerous case studies in the appendix to provide intuitive insights into our evaluation results.

(continued in our next response)

W3: Table 2 and 3 show the results of SD1.5 finetuned with DPO and DDPO, which, according to my understanding, is the "evaluation" to the proposed benchmark. However, I found that the LMM that work best as the reward (Table 2) is not aligned with the LMM having the highest evaluation scores based on the benchmark (Table 1). Does it mean the proposed benchmark is not good to evaluate T2I reward models? In addition, in Table 3, GPT-4o and GPT-4v achieve the best results with DPO and DDPO, respectively. I think this result suggest that the evaluation of reward model should be performed for certain RLHF methods. I suggest the authors to provide more discussion about this.

We thank the reviewer for raising this important concern. As the reviewer noted, Tables 2 and 3 indeed provide an end-to-end evaluation of our benchmark. We greatly appreciate the reviewer’s observations regarding the results in Tables 2 and 1, and we would like to provide further details about the evaluation pipeline to explain why the results are reasonable and demonstrate the reliability of our benchmark. Specifically, we attribute the discrepancies between the results in Tables 1 and 2 to the following reasons:

• Only pairs with explicit preference are being used: For fine-tuning the model via DPO, we only use pairs with an explicit preference (i.e., those without ties). For instance, the discrepancy observed by the reviewer, particularly in the safety perspective, stems from models like Gemini Ultra having a high rejection rate for unsafe inputs. This results in a low accuracy for safety w/ tie (13.1), as many outputs are classified as ties, despite having a high accuracy w/o tie (95.1). Since we rely on the w/o tie results as the guideline for fine-tuning via DPO, the results in Table 2 are more aligned with the w/o tie column in Table 1.

• Human bias during evaluation: A notable trend in Table 2 is that GPT-4o tends to be ranked higher by human evaluators across all perspectives, whereas the performance of other models generally follows the trends observed in Table 1. We suspect this bias arises because, during the ranking task (illustrated in Figure 16 of the appendix), some annotators were informed about which model generated each image (e.g., the top-right image corresponds to GPT-4o). Since GPT-4o is widely recognized as a SOTA model for providing judge feedback, this knowledge may have influenced some annotators to favor GPT-4o’s outputs and introduced certain biases, leading them to consistently higher rankings for images provided by GPT-4o. Consequently, this introduces a bias in the evaluation, skewing results in GPT-4o's favor.

• Bias of the score-based evaluation: Another factor contributing to the discrepancy is the method used for scoring. Human evaluators were asked to assess six images simultaneously and assign a score to each within a [0, 10] scale, which was later used to calculate rankings. However, this approach may also introduce bias, as individual annotators may interpret and apply the numerical scale differently based on their personal perspectives. Additionally, evaluating multiple images at once can increase the cognitive load on annotators, potentially affecting the consistency and reliability of their evaluations, and further contributing to the biased results.

Following the reviewer’s suggestions, we aim to address the bias and annotation difficulty by adopting a simpler yet effective evaluation metric. Instead of scoring multiple images simultaneously, human annotators are asked to compare only a pair of images: one generated by the fine-tuned model and the other by the base SD-1.5 model (consistent across all evaluations of different models). We then calculate a win rate against the SD-1.5 for each model, with the results presented in Table-r. 3 below. This approach is more intuitive for annotators, reduces cognitive load, and minimizes bias introduced by individual interpretations of numerical scales. The results shown in Table-r. 3 align more closely with those in Table 1, with HPS-v2.1 and Gemini Ultra providing the most accurate feedback for the alignment perspective, GPT-4o excelling in Safety and Quality, and LLaMA-3.2-11B-Vision performing best in Bias. These additional results have been included in the paper revisions, and we hope they better demonstrate the effectiveness of our dataset and address the reviewer’s concerns.

(continued in our next response)

Dear Reviewer eath,

Thank you so much for your valuable suggestions to help improve our paper! We deeply appreciate your recognition of the novelty of our work and its critical contribution to the multimodal AI feedback community. Below, we have provided a detailed, point-by-point response to your questions, and hope these could help address your concerns.

W1: In terms of the presentation, I am unsure whether claiming evaluating "reward models" is a better idea than claiming evaluating "LMMs". Close-sourced LMMs such as GPT-4o/4v/Claude are not widely adopted T2I reward models due to their high cost and low throughput. In addition, Table 10 summarizes the number of evaluation questions for each category of the benchmark, which is of the pivotal importance and should be put in the main paper.

We sincerely appreciate the reviewer’s thoughtful feedback. We agree that since our work focuses on a specific capability of "LMMs" to function as "reward models," both claims are valid. Specifically, if we adopt a broader definition of "reward models" that emphasizes their functional role in providing feedback, as characterized by prior works [1, 2], this can encompass both closed-source LLMs and LMMs like GPT-4o/4v, as well as open-sourced models such as LLaMA-3.2-11B-Vision-Instruct, despite the high inference costs and low throughput noted by the reviewer. While online RL algorithms like DDPO demand timely and responsive feedback, the low throughput issue highlighted by the reviewer can be mitigated using offline RL approaches, such as pre-collecting a dataset of trajectories and applying the reward models for preference annotation and then fine-tuning via DPO. However, we also acknowledge the reviewer’s point that, while we evaluate their ability to serve as reward models, we are ultimately assessing the capabilities of "LMMs" themselves. We have updated the manuscript to clarify this distinction in line with the reviewer’s suggestions.

Regarding the second suggestion, we completely agree that Table 10, which summarizes the dataset, is critical for effectively conveying key details of our dataset. We have moved this table to the main paper, specifically to Section 2.3, as suggested by the reviewer.

We hope these detailed explanations address the reviewer’s concerns, and we would be delighted to engage in further discussions should the reviewer have additional questions or feedback.

[1] Lambert, N., Pyatkin, V., Morrison, J., Miranda, L. J., Lin, B. Y., Chandu, K., ... & Hajishirzi, H. (2024). Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787.

[2] Mahan, D., Van Phung, D., Rafailov, R., Blagden, C., Lile, N., Castricato, L., ... & Albalak, A. (2024). Generative reward models. arXiv preprint arXiv:2410.12832.

Thank you for your thoughtful response and for raising this important concern! We totally agree with the reviewer that ensuring a balanced and unbiased dataset distribution is crucial for fair and reliable evaluations, and we have indeed carefully considered this when we developped the dataset. Our detailed responses are as follows:

Is this an issue in the proposed benchmark?

The unbalanced sample size in Table 10 is not an inherent issue of the benchmark but a specific design choice aimed at achieving efficient evaluation without compromising accuracy. As detailed in Table-r.1 above in response part 3, our dataset curation process involves three stages where the human verifiers first filter 8K samples guaranteed with high quality and a balanced distribution across subcategories to ensure fairness in evaluation and comparability across models. However, due to the high cost associated with evaluating such a large dataset, we aim to further enhance sample efficiency to facilitate the broader usability of our benchmark. Specifically, we adopt a Confidence Subsampling process, where we use three surrogate models to further search for a lower bound of the sample size for each subcategory such that the variance in performance estimation in each subset remains within a small statistical confidence threshold, guaranteeing both accuracy and efficiency in evaluation. While this process results in an unbalanced distribution across subcategories, it guarantees sufficient sample sizes in each subcategory to provide confident performance estimates efficiently. To validate this approach, we conducted empirical experiments with top-performing models, which showed a very small performance gap between the efficient subset and the full dataset, confirming its robustness. Based on these findings, we use the efficient subset to report the results in the paper.

How can this issue be addressed?

To address the concern, we will ensure to release both the full balanced dataset version and the efficient subset to accommodate the user's diverse needs. Besides, we will ensure to include evaluations of the full balanced dataset in the updated revision of the paper to provide a comprehensive performance analysis, ensuring transparency and trust in the benchmark’s fairness and applicability.

We sincerely hope that these actions will address the reviewer’s concerns while maintaining the benchmark’s utility for diverse use cases. Thank you again for your valuable feedback, and we hope these clarifications and proposed measures meet your expectations!

Table-r. 2 Additional evaluation results of our MoE-based judge model trained on a split from MJ-Bench. We evaluate and compare a subset of the models with best performance from Table 2 in the paper using the rest of the data as test set. The best performance are in bold.

Here we provide some additional related works. [3] and [4] also explore improving text-image alignment with MLLM feedback but rely on preference datasets curated through simple heuristics, without ensuring data diversity or maintaining high-quality standards. Furthermore, they only leverage the preference data to train a text-to-image model without proposing a reward model.

Another line of research focuses on providing more grounded scores for text-image alignment through decomposition [5,6], which involves breaking down complex prompts into multiple atomic predicates and verifying each individually, thereby enhancing the robustness of the feedback. Additionally, some probability-based methods [7] find that templating the prompt into binary questions and evaluating the likelihood of answering yes can result in a more stable scoring. We have also included the evaluation of these models in Table-r. 3.

Specifically, we observe that the methods suggested by the reviewers have outperformed the previous methods on certain perspectives while still struggling with certain tasks. We detail our analysis and findings of the updated results point-by-point.

• Decomposition significantly boosts judge performance on text-image alignment: Table-r. 3 indicates that decomposition-based methods DSG and T2I-CompBench have superior performance on alignment and image quality perspectives, which require a fine-grained judgment of the elements and their corresponding attributes and relationships in the image. By decomposing a complex prompt into multiple atomic predicates and verifying them individually, these methods are highly interpretable and guaranteed to have better performance in providing feedback for alignment accuracy. Furthermore, we observe that DSG w/ entailment dependency also outperforms the non-dependent case, demonstrating the significance of logical reasoning in providing grounded feedback for text-image alignment. The fact that w/ dependency case has better performance also validates the high-quality and challenging nature of our dataset, as our images are verified by human annotators so that in each image the logical constraints in the prompt are satisfied. However, while the feedback is more accurate, the decomposition-based methods typically incur higher inference costs as they require extra decomposition and multiple MLLM-prompting, and thus may not be suitable to serve as online judges.

• Probability-based methods reduce the numerical sensitivity of the feedback score: While VQAScore [3] can effectively address the "bag-of-words" issue in CLIP-based models, our evaluation in Table-r. 3 indicates that the performance w/o tie is much higher than w/ tie, indicating VQAScore results in a much larger tie rate than the other models (we set a threshold of 0.1 to determine tie). This indicates that by formatting the prompt into a Does this figure show {text}? question, while it increases the evaluation consistency of a single image, it inherently reduces the gap between two similar images that are specifically curated to be hard to distinguish in our challenging benchmark, thus resulting in a higher tie rate. These results indicate that VQAScore might not serve as a good judge for preference-based text-to-image post-training but can provide an adequately accurate evaluation of a single image.

(continued in our next response)

Dear Reviewer 3DFZ,

Thank you so much for your valuable suggestions to help improve our paper! We deeply appreciate your recognition of the novelty of our work and its critical contribution to advanced research in RLAIF for text-to-image generation. Below, we have provided a detailed, point-by-point response to your questions, and hope these could help address your concerns.

W1:Please provide a comprehensive discussion that covers the related work. For example, some studies have also made efforts in aligning text-to-image models with feedback from MLLMs, such as VisionPrefer[1], which also discusses the varying effectiveness of different MLLMs as annotators for alignment data in text-to-image tasks.

Thank you for this thoughtful suggestion. We sincerely appreciate the reviewer for bringing this valuable work to our attention and we fully agree that a more comprehensive discussion of related works would better highlight the significance of our contributions. In response, we have updated the related works section in the paper revision to include additional recent studies, and we summarize the newly incorporated works below.

Regarding the datasets for providing feedback on text-to-image generation, VisionPrefer [1] is a concurrent work that introduces a large dataset of preference data spanning four aspects. This dataset is subsequently used to train a CLIP-based reward model, VP-Score, to provide feedback for image generation. While VisionPrefer is related to our work, our contribution differs in several key ways. First, VisionPrefer primarily provides a large preference dataset with four aspects, lacking the fine-grained subcategories needed for a comprehensive evaluation of multimodal judges' feedback. In contrast, MJ-Bench not only covers these perspectives but also decomposes each into over 20 scenarios, ensuring a more exhaustive evaluation framework. Second, VisionPrefer relies solely on GPT-4v for data annotation which cannot provide any guarantee of the data quality, whereas our dataset undergoes a rigorous three-stage curation process: (1) VLM pre-processing, (2) human verification, and (3) confidence interval-efficient sampling. Detailed statistics of the data split in each stage are provided in Table-1. below. This pipeline ensures the reliability, diversity, and difficulty of the dataset, making it a more robust resource for fine-grained multimodal evaluation. Inspired by their approach which trains a reward model, VP-Score, on their curated preference dataset, we designed an additional experiment where 80% of the MJ-Bench data was randomly split (except for Bias, where we use 64 groups of the data filtered out from the confidence filtering stage) to train a MoE-based judge model, following the method in [2]. The model incorporates four experts, each responsible for a specific perspective, with a gating layer to aggregate scores across each perspective trained via BT objective. Then we use the remaining 20% of the data as a test set. Results are reported in Table-r. 2.

From Table-r. 2, we observe that the MoE-based judge trained on MJ-Bench outperforms other models in alignment, safety, and bias perspectives in terms of w/ tie scores while being very close to GPT-4o on the quality subset. These findings highlight the advantages of MoE structures for handling multi-objective feedback and underscore the high quality of MJ-Bench data samples. However, the results also suggest that scaling up MJ-Bench, particularly in the quality subset, could further enhance performance, potentially surpassing GPT-4o. Due to time constraints, we plan to train our reward model on a larger held-out training set and evaluate it on the full MJ-Bench test set to compare against more models. Unfortunately, since the authors of VisionPrefer have not open-sourced the VP-Score model, we were unable to include it in our benchmark evaluation.

Table-r. 1 Statistics of the data curation procedure and quality control.

(continued in our next response)

• Decomposition-based methods increase safety risks and do not have much effect on bias: (1) Safety: we further observe in Table-r. 1 that decomposition-based methods have much lower performance from a safety perspective than other one-time inference models such as LLaMA-3.2-11B-Vision-Instruct and GPT-4o. This is because decomposing an unsafe prompt into several atomic descriptions will potentially reduce the harmfulness of the prompt and instead make the prompt more stealthy, leading to such models providing a higher overall score for unsafe images that align better with the prompts. However, one-time inference MLLMs can evaluate the unsafe prompt in its entirety and can thus provide more accurate feedback toward a safer alignment. (2) Bias: we find that decomposition-based methods have similar performance to their base models such as InstructBLIP and BLIP and do not show any significant improvement. This is primarily because we evaluated the bias perspective with straightforward prompts such as a portrait of a doctor which cannot be further decomposed, thus resulting in these decomposition-based methods being degraded to have the same performance as their base models used for individual predicate evaluation.

• MLLMs with more advanced vision encoders have significantly higher performance on alignment: While the evaluation results on the four most recent MLLMs indicate they have better performance on almost all perspectives, we find that models with more advanced vision encoders such as LLaMA-3.2-11B-Vision-Instruct with a specifically fine-tuned image coder and InternVL2-26B with an extra-large 6B vision encoder (compared to 300M vision encoder of the 8B model) consistently yield better performance on text-image alignment and image quality perspectives, demonstrating the importance of such vision encoders to provide grounded information for the subsequent judge reasoning. We are also interested in evaluating the most recent MLLMs such as Transfusion [7] which encodes images through diffusion mechanisms in an interleaved manner once they are open-sourced.

• Instruct-tuned models provide more accurate feedback on safety and bias: We also notice that Instruct-tuned models and post-aligned models such as LLaMA-3.2-11B-Vision-Instruct have better performance in terms of safety and bias, indicating they have embedded such values that align better with human perspectives and can provide better feedback towards a safer and debiased generation.

(continued in our next response)

Dear Reviewer qiUB,

Thank you so much for your valuable suggestions to help improve our paper! We deeply appreciate your recognition of the novelty and comprehensiveness of our work! Below, we have provided a detailed, point-by-point response to your questions, and hope these could help address your concerns.

W1: Comprehensiveness and fairness of the evaluation: The paper evaluates models like CLIP, LLaVA, and GPT-4, but lacks some popular alternative alignment evaluation models such as the Decompose method (Davidsonian Scene Graph) and the Answer Probability Method (VQAScore).

Thank you for this insightful suggestion! We totally agree with the reviewer that more comprehensive evaluation results including the decomposition-based approaches and probability-based method will strengthen the scope and fairness of our benchmark. Therefore, we conduct additional experiments to include the two most advanced decomposition-based methods: Davidsonian Scene Graph (DSG) [1], and T2i-compbench [2], as well as a probability-based method VQAScore [3] following the reviewer's suggestion. To keep our benchmark updated with the most advanced methods, we further include four additional SOTA open-sourced MLLMs, i.e. LLaMA-3.2-11B-Vision-Instruct [4], InternVL2-8B [5], InternVL2-26B and MiniCPM-V-2_6 [6], and evaluate their performance as a judge on MJ-Bench. We present the additional evaluation results in Table-r. 1 below, where we also include GPT-4o from the paper as a reference. We provide two parallel evaluations of DSG in terms of whether the logical entailment dependency is available or not (e.g. the proposition “there is a motorcycle” is a parent of “the motorcycle is blue”: this dependent’s truth (i.e. whether the motorcycle is blue) can only be evaluated if the parent is true (i.e. there is a motorcycle at all)).

Specifically, we indeed observe that the methods suggested by the reviewers have outperformed the previous methods on certain perspectives while still struggling with certain tasks. We detail our analysis and findings of the updated results point-by-point.

• Decomposition significantly boosts judge performance on text-image alignment: Table-r. 1 indicates that decomposition-based methods DSG and T2I-CompBench have superior performance on alignment and image quality perspectives, which require a fine-grained judgment of the elements and their corresponding attributes and relationships in the image. By decomposing a complex prompt into multiple atomic predicates and verifying them individually, these methods are highly interpretable and guaranteed to have better performance in providing feedback for alignment accuracy. Furthermore, we observe that DSG w/ entailment dependency also outperforms the non-dependent case, demonstrating the significance of logical reasoning in providing grounded feedback for text-image alignment. The fact that w/ dependency case has better performance also validates the high-quality and challenging nature of our dataset, as our images are verified by human annotators so that in each image the logical constraints in the prompt are satisfied. However, while their feedback is more accurate for alignment, the decomposition-based methods typically incur higher inference costs as they require extra decomposition and multiple MLLM-prompting, and thus may not be suitable to serve as online judges.

• Probability-based methods reduce the numerical sensitivity of the feedback score: While VQAScore [3] can effectively address the "bag-of-words" issue in CLIP-based models, our evaluation in Table-r. 1 indicates that the performance w/o tie is much higher than w/ tie, indicating VQAScore results in a much larger tie rate than the other models (we set a threshold of 0.1 to determine tie). This indicates that by formatting the prompt into a Does this figure show {text}? question, while it increases the evaluation consistency of a single image, it inherently reduces the gap between two similar images that are specifically curated to be hard to distinguish in our challenging benchmark, thus resulting in a higher tie rate. These results indicate that VQAScore might not serve as a good judge for preference-based text-to-image post-training but can provide an adequately accurate evaluation of a single image.

(continued in our next response)