Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators

We thank the reviewer for their constructive feedback. Below we address each concern in detail. We look forward to any further discussions.

Q1. [Inclusion of latest baseline models] It would be better if the experimental results included stronger baselines such as the latest Gemini 2.5 Pro.

A1. Thank you for the suggestion. We have presented Gemini-2.5-Pro results in the table below. The latest Gemini-2.5-Pro model shows mixed superiority over previous versions.

However, we would like to emphasize that our ultimate goal is not to surpass proprietary models on conventional benchmarks, but to enable capabilities that they currently lack. Kirby-Judge is designed to be adaptable to new, underexplored domains, such as molecular tasks, where existing models cannot be directly applied—which we detail further in Section 4 (Broader Impact).

 

Q2. [Additional related work] Misses some related work. For example, there are various multimodal judge or reward benchmark works worth discussing or evaluating on: Multimodal RewardBench, in addition to MLLM-as-a-Judge and VL-RewardBench, etc. discussed in the paper.

A2. Thank you for the suggestion. We will include the mentioned works and discussion in our related work section. Additionally, we have extended our evaluation by including results on a recent MLLM evaluation benchmark, Multimodal RewardBench (Yasunaga et al., 2025) and JudgeAnything (Pu et al., 2025).

Multimodal RewardBench (Yasunaga et al., 2025) focuses on image understanding, while JudgeAnything (Pu et al., 2025) covers a wide range of any-to-any tasks. JudgeAnything includes not only image/video/audio understanding (I → T, V → T, A → T) but also image/video/audio generation (T → I, T → V, T → A), image editing (I → I), image-to-video (I → V), and image-to-audio (I → A) tasks.

Multimodal RewardBench:

JudgeAnything:

 

Q3. [Intuition on multimodal generalization] Could the authors elaborate more on the intuition of generalizing from textual reasoning data to multimodal judge capability?

A3. Thank you for the question. Our intuition is motivated by two key lines of recent research:

1. It has been shown that when multilingual LLMs are pretrained to encode shared representations across languages, fine-tuning on a specific task in one language often leads to improved performance on the same task in other languages, even without additional data. This demonstrates the potential for knowledge transfer and skills to generalize across different domains when the underlying representations are unified.

2. Works such as s1 (Meunnighoff et al., 2025) and LIMO (Ye et al., 2025) have shown that pretrained LLMs embed rich knowledge and reasoning capabilities, which can be "elicited" with only a small amount of additional reasoning supervision. This pattern has also been observed in preference alignment research, such as LIMA (Zhou et al., 2023), where even a small set of high-quality instructions can unlock strong task performance in pretrained models.

Based on these findings, our core intuition is that a well-pretrained multimodal LLM already possesses unified cross-modal representations and general reasoning ability. Therefore, with only a small amount of high-quality textual reasoning data, we are able to evoke a model to elicit multimodal reasoning-based judgment capabilities, without the need for large-scale, modality-specific annotations. Our work empirically verifies this hypothesis.

 

Q4. [Question on curated training dataset] How was the seed training data curated? Could the authors provide some concrete examples of the data? Given that the data is mainly synthetic, how do you ensure its high quality and avoid issues such as model collapse?

A4. Thank you for your question. Our seed training data was curated by applying a set of quality criteria, which we systematically investigated through various ablation studies (see Figure 2 and Section 2.2). These criteria proved effective for producing high-quality synthetic data without suffering from mode collapse in our experiments. To further address concerns about possible collapse or forgetting, we additionally evaluated our models on visual question answering (VQA) tasks unrelated to judgment, specifically TextVQA and OK-VQA. As shown below, Kirby-VL-7B exhibits only a slight decrease on TextVQA and even outperforms the base model on OK-VQA.

These results suggest that our well-structured methodology does not cause model collapse or catastrophic forgetting. Also, we want to highlight that we have included the curated seed data in the supplementary material for your reference.

We thank the reviewer for their constructive feedback. Below we address each concern in detail. We look forward to any further discussions.

Q1. [More recent benchmarks on image/video understanding] Have authors evaluated Kirby-Judge on newer image and video understanding benchmarks like MM-RewardBench or JudgeAnything to more comprehensively validate its performance?

A1. Thank you for your suggestion. In response, we have extended our evaluation to include results on two recent and comprehensive MLLM evaluation benchmarks: Multimodal RewardBench (Yasunaga et al., 2025), which focuses on image understanding, and JudgeAnything (Pu et al., 2025), which covers wide range of any-to-any tasks. JudgeAnything includes not only image/video/audio understanding (I → T, V → T, A → T) but also image/video/audio generation (T → I, T → V, T → A), image editing (I → I), image-to-video (I → V), and image-to-audio (I → A) tasks. The results are summarized in the table below.

Multimodal RewardBench:

JudgeAnything:

Across both benchmarks, Kirby-Judge achieves comparable performance to proprietary models while consistently outperforming existing open-source baselines. However, we would like to emphasize that our ultimate goal is not to surpass proprietary models on conventional benchmarks, but to enable capabilities that they currently lack. Specifically, Kirby-Judge is designed to be adaptable to new, underexplored domains, such as molecular tasks, where existing models cannot be directly applied—which we detail further in the following response (A4).

 

Q2. [More recent benchmarks on video generation] Have authors evaluated Kirby-Judge on newer benchmarks like Q-Eval-100K and Video-Bench for video generation and image editing tasks?

A2. Thank you for your suggestion. We carefully considered the recent benchmarks you mentioned. However, to the best of our knowledge, the official GitHub (or Hugging Face) repository for Q-Eval-100K does not provide access to their test split, which is necessary for a fair and standardized evaluation. For Video-Bench, their evaluation framework involves using both MLLMs and LLMs simultaneously, which is not directly compatible with our evaluation protocol.

Instead, the JudgeAnything results (as shown in A1) show promising performances of Kirby-Omni-7B in text-to-video (T → V) and image-to-video (I → V) generation tasks. We currently believe GenAI-Bench (Table 3 in the paper) is the most widely used and representative benchmark for video generation and image editing. Our results show that Kirby-Judge is effective across a wide range of visual tasks, including video and image generation, image understanding, image editing, audio generation, and molecule understanding.

 

Q3. [More recent baselines] Did authors consider including recently proposed MLLM-as-a-Judge models, such as InternLM-XComposer2.5-Reward or CAREVL, in your experimental comparison to more thoroughly assess the competitiveness of Kirby-Judge?

A3. Thank you for the suggestion. Recent models like CaReVL and InternLM-XComposer2.5-Reward (IXC-2.5-Reward) are primarily designed only for pairwise comparison and trained on large-scale visual-language pairs, whereas Kirby-Judge is built to generalize across broader evaluation formats, including single-score grading and batchwise ranking, trained with ~1K text data.

Importantly, Kirby-Judge demonstrates strong generalization not only in evaluation format, but also across domains. While many existing judge models are limited to vision-language tasks, our work uniquely extends to molecular generation (potentially to other modalities like 3D space), where curated preference data is scarce. Despite these differences, Kirby-Judge remains competitive even when evaluated against recent models, as presented in our results below.

 

Q4. [Practical utility of Kirby-Judge for response selection] Have you evaluated the practical utility of Kirby-Judge as a response selector at inference time, for instance by selecting the best answer from diverse model outputs?

A4. As described in Section 4 (Broader Impact) and Figure 4, we have already demonstrated the use of Kirby-Judge as a best-of-N selector for inference-time scaling, particularly in the molecular domain. This shows that Kirby-Judge can effectively select the best response from a set of diverse outputs in practical scenarios. Furthermore, we demonstrated that Kirby-Judge can also serve as an off-the-shelf reward model for DPO training, where Kirby-Mol-Llama effectively guides Mol-Llama's preference optimization, highlighting its applicability to downstream training.

We appreciate your constructive comments. We have rephrased the question for ease of reference and provide our corresponding responses below. We look forward to any further discussions.

Q1. [Comparison with base model performances] Could the authors include the results of the base models (Qwen2.5-VL-7B, Qwen2.5-Omni-7B) in the main tables to clearly show how much improvement is achieved by Kirby-Judge?

A1. Thank you for your comment. As shown in the table below, we directly compared our Kirby-Judge models (Kirby-VL-7B and Kirby-Omni-7B) to the base instruction-tuned models (Qwen2.5-VL-7B and Qwen2.5-Omni-7B) across multiple benchmarks. With only a small amount of text-only data, Kirby-Judge achieves substantial performance improvements over the base models—for example, up to +32.38 on VL-RewardBench (Reasoning) and +14.85 on GenAI-Bench (Edition). This demonstrates that our fine-tuning approach is highly effective compared to simply prompting the instruction-tuned base models.

 

Q2. [Large standard deviation of Kirby-Omni-7B] In Figure 6, the standard deviation of multiple runs (especially for Kirby-Omni-7B) appears to be large. How does this compare to the baseline models? Some further analysis on model prediction stability or robustness would be helpful.

A2. Thank you for your question. The table below reports the average (standard deviation) of model accuracy across multiple runs, where $k$ is the number of reasoning paths used for majority voting. We observe that the baseline instruction-tuned Omni model (Qwen2.5-Omni-7B) already exhibits high variance, and this characteristic is inherited by Kirby-Omni-7B. In contrast, both the baseline VL model and Kirby-VL-7B show much lower standard deviation, indicating greater stability. Importantly, for both VL and Omni variants, the Kirby-Judge model either matches or reduces the standard deviation relative to its base model at each value of $k$. These findings suggest that prediction stability (as measured by standard deviation) is mainly a property of the base model. Our method does not introduce additional instability and, in some cases, even improves evaluation consistency compared to the baseline.

 

Q3. [Dependence on base model capabilities] As acknowledged in the supplementary material, the effectiveness of Kirby-Judge is influenced by the reasoning capacity of the backbone model. Could the authors comment on how your approach performs with weaker or smaller backbone models, and whether you conducted related experiments?

A3. Thank you for your insightful question. We agree that the performance of Kirby-Judge is closely tied to the multimodal understanding and reasoning ability of its backbone MLLM. To directly assess this, we conducted additional experiments with a weaker backbone: we trained Kirby-VL-3B by fine-tuning Qwen2.5-VL-3B on our curated seed dataset and compared its performance to Kirby-VL-7B and other larger models (see Table 7 and Appendix C.1 in the supplementary material). Notably, Kirby-VL-3B demonstrates strong generalization and reasoning ability, achieving better results than LLaVA-NeXT and Qwen2.5-VL-7B, but underperforms compared to our Kirby-VL-7B model.

Additionally, we further experimented with another weaker backbone, Qwen2-VL-7B, in place of Qwen2.5-VL-7B. This resulted in a slight further performance drop: 0.538 → 0.524 (MLLM-as-a-Judge, pair w tie) and 46.1 → 42.5 (VL-RewardBench, General), but still highly competitive to the open-source baselines. These results confirm that while the choice of backbone affects the performance, our Kirby-Judge consistently brings significant gains over the base models, even with smaller or weaker MLLMs.

Moreover, as the reviewer pointed out, reasoning-capable MLLMs are rapidly evolving. Recent models like DeepSeek-R1-Distill show promising reasoning ability even at smaller scales, and our framework is designed to benefit from these ongoing advancements. Once a stronger or more efficient backbone becomes available, our method can be seamlessly applied using only a small amount of high-quality text-based supervision.

 

Q4. [Larger Kirby-Judge] Another minor limitation is that this work does not show whether this method can be further scaled to larger models (e.g., 32B).

A4. Thank you for pointing this out. We have trained Kirby-VL-32B, based on Qwen2.5-VL-32B-Instruct (see table below), and these results confirm the scalability of our framework. Although we trained the 32B model using parameter-efficient LoRA (Hu et al., 2021), which is generally less effective than full fine-tuning due to limited computational resources, it outperforms the fully fine-tuned 7B version on most evaluation benchmarks and is competitive with GPT-4o. This validates that judgment quality improves as the backbone’s reasoning capacity increases.

Thank you for your insightful feedback. We have rephrased your comments for simpler reference and have included our respective responses. We look forward to any further discussions.

Q1. [Comparison with base model performances] Did authors compare the performance of your base models and fine-tuned models on judgment tasks, and did you assess how text-only fine-tuning affects their multimodal understanding on standard vision-language benchmarks?

A1. Thank you for your comment. As shown in the table below, we directly compared our Kirby-Judge models (Kirby-VL-7B and Kirby-Omni-7B) to the base instruction-tuned models (Qwen2.5-VL-7B and Qwen2.5-Omni-7B) across multiple benchmarks. With only a small amount of text-only data, Kirby-Judge achieves substantial performance improvements over the base models—for example, up to +32.38 on VL-RewardBench (Reasoning) and +14.85 on GenAI-Bench (Edition). This demonstrates that our fine-tuning approach is highly effective compared to simply prompting the instruction-tuned base models.

To further address concerns about potential catastrophic forgetting or degradation of multimodal abilities, we also evaluated our models on standard visual question answering (VQA) tasks that are unrelated to judgment, specifically TextVQA (Signh et al., 2019) and OK-VQA (Marino et al., 2019). As shown below, Kirby-VL-7B exhibits only a slight decrease on TextVQA and even outperforms the base model on OK-VQA. These results suggest that our well-structured methodology preserves the multimodal capabilities of the base model and does not cause forgetting.

In summary, our experiments demonstrate that Kirby-Judge achieves significant improvements over the base models on judgment tasks, while largely maintaining (and in some cases improving) performance on non-judgment multimodal benchmarks.

 

Q2. [Effect of dataset size] How does the amount of text-only data used for fine-tuning affect generalization and multimodal performance? Did you perform ablation studies on this?

A2. Thank you for the comment. As shown in Section 2.2 and Figure 2(b), we have provided an ablation study on the effect of dataset size for both text-only (PandaLM, JudgeLM) and vision-language (VL-RewardBench, MJ-Bench) judge evaluations. We find that text-only tasks benefit from larger datasets, showing continuous improvement as the number of samples increases. In contrast, for vision-language tasks, performance improves up to a certain threshold (1K), but adding more data beyond that point results in a slight and gradual performance drop, likely due to catastrophic forgetting. These results highlight the importance of careful dataset size selection, and we have designed our methodology with these findings in mind.