On Path to Multimodal Generalist: General-Level and General-Bench

We are more than excited to receive your very strong recognition of our work, which means a lot! Also thanks for your detailed review and valuable suggestions. We have done our best to address all concerns and are happy to engage in further discussion to improve the clarity and quality of our work.

---

Q1. General-Bench focuses on capability while overlooking critical risks (e.g., hallucination, bias amplification). Could high-scoring models in your framework inadvertently reward unsafe behaviors? Should 'safety synergy' be introduced as a separate dimension in the newly proposed framework?

A: Thank you for the reviewer’s thoughtful and forward-looking comment. When designing the scope definition and task list curation for General-Bench, we paid special attention to mitigating potential bias by ensuring diversity in data sources, annotation methods, and task formats. Moreover, we prioritized using well-established datasets with high-quality annotations, and in some cases, relied on manual annotations to further minimize the risk of hallucinated content.

Regarding the safety perspective, we explicitly focused on evaluating positive and safe skill sets when defining the skill scope of our benchmark. During data collection and task construction, we carefully reviewed whether the expected model outputs could involve safety-sensitive content. If any safety risks were detected, we excluded those instances from the final benchmark.

We sincerely appreciate the reviewer’s suggestion to consider "safety synergy" as a separate evaluation dimension. We agree that safety is a critical component of trustworthy multimodal AGI systems and should be an integral part of generalist benchmarking. In future versions of General-Bench, we plan to incorporate safety-focused tasks and metrics to better evaluate models not just on capability, but also on responsible and safe behavior.

---

Q2. Since the benchmark contains more than 700 tasks, which is tremendous, evaluating models on General-Bench likely requires massive compute. Does this contradict the push for sustainable AI? Have you quantified the carbon footprint, and if not, should this be a mandatory disclosure for future benchmarks?

A: Thank you for raising this important concern. We agree that sustainability is a crucial consideration in designing and deploying large-scale AI benchmarks.

Although General-Bench includes over 700 tasks, it is intentionally designed to be modular and flexible. Model developers are not required to run evaluations on the entire task suite. In practice, given prior knowledge of a model’s capability boundaries, developers can select a relevant subset of tasks for evaluation to obtain a fair and rigorous comparison across other models.

We also acknowledge the importance of quantifying carbon footprint in building more responsible and sustainable AI systems. While we have not yet included CO₂ impact reporting in the current version, we plan to integrate optional carbon estimation tools in future releases, following emerging best practices in large-scale evaluation.

---

Q3. In the current setting, is each modality equally important on the path to AGI? The authors mention that they would like to incorporate more modalities into this benchmark. The current modalities seem to be the primary modalities for intelligence, while the newly added modalities do not seem as important as the current ones. If each modality is equally important, the current configuration may not fairly reflect the model's capabilities.

A: Thank you for this thoughtful question. We would like to clarify that General-Bench does not assume all modalities are equally important on the path toward AGI, nor do we assign equal weight to each modality in our evaluation. On the contrary, we recognize that different modalities play distinct and complementary roles in intelligent systems, and their importance can vary depending on the task, environment, or application context.

The currently included modalities—language, image, video, audio, and 3D—have been extensively studied and are more readily available at scale, which naturally draws more research attention. However, other modalities such as heatmaps, sensor data, charts, or even tactile signals also encode rich, structured information about the world. These may be particularly crucial in specialized domains such as medical diagnosis, physical reasoning, or embodied intelligence.

Our motivation for incorporating more modalities is not to suggest that each one is equally critical for AGI, but rather to build a broader and more flexible evaluation framework that can assess how well models generalize across diverse input types. We appreciate the reviewer’s suggestion and will revise the paper to make our position on modality importance and evaluation scope more explicit and transparent.

---

Q4. Typos: Line 2328: all task -> all tasks ...

A: Thanks！Will correct this.

We sincerely thank the reviewer for your time, meaningful questions, and constructive suggestions. Also, your recognition of our paper means a lot to us, which is the source of power to push us forward and enhance this work/project for a greater meaning to the community. We address each concern in detail below and hope that our clarifications help improve your evaluation of our work.

---

Q1. The five-level classification method lacks clarity, as it does not provide well-defined criteria for each level. In particular, the concept of "synergy" between different modalities is not clearly articulated, making the classification ambiguous. For instance, Unified-IO-2 and Next-GPT, which are any-to-any modality generative models capable of producing visual and audio outputs from multimodal inputs, are categorized as Level-2, suggesting they lack synergy in comprehension and generation. Meanwhile, DeepSeek-VL and LLaVA-One-Vision, which do not even possess visual generation capabilities, are classified as Level-3, implying they exhibit synergy across tasks. This inconsistency raises concerns about the validity of the classification framework, as it does not appear to align with the actual capabilities of these models. A more precise and well-justified definition of "synergy" is necessary to ensure the classification accurately reflects the models’ multimodal abilities.

A: Thank you for the detailed and thoughtful feedback. Due to space constraints, we provided the full definitions and criteria for each of the five levels in Appendix C. We kindly refer the reviewer to this section for a comprehensive explanation.

First, it is important to note that the levels in our framework are not mutually exclusive, but rather hierarchical. That is, a model classified at Level-3 will also have valid scores at Level-2, and so on. In Table 1, Unified-IO-2 and Next-GPT are shown as example models under Level-2 to illustrate that they satisfy the baseline criteria for this level—not to imply they are limited to Level-2. In fact, as shown in Table 19 (Appendix), both Unified-IO-2 and Next-GPT also receive valid Level-3 scores, indicating that they exhibit synergy across comprehension or generation tasks. However, they do not appear at Level-4, which requires a synergy in comprehension and generation.

Secondly, as for Level-3, the definition explicitly includes models with synergy in comprehension and/or generation. Although DeepSeek-VL and LLaVA-One-Vision do not support generative outputs, their performance on comprehension tasks exceeds that of single-modality specialists, thereby qualifying them for Level-3 based on comprehension synergy alone.

We appreciate the reviewer pointing out this potential source of confusion, and we will revise the paper to clarify the hierarchical nature of the levels, emphasize that the examples in Table 1 are illustrative rather than exclusive, and provide clearer articulation of "synergy" as it applies to comprehension and generation tasks.

---

Q2. However, the analysis in Section 5.2 feels somewhat superficial, especially considering the large-scale experiments conducted in this study. The observations are fairly straightforward and lack deeper insights, making them less compelling. It would be much more engaging to see more unique insights drawn from the vast amount of experiments conducted.

A: Thank you for the valuable feedback. Due to space limitations in the main paper, we have provided more in-depth analyses and insights in Appendices B.5, B.6, and B.7, including detailed discussions on synergy across skills, modalities, and comprehension/generation dimensions. In addition, we include fine-grained performance breakdowns for each model across individual skills, which allow for more precise diagnosis of model weaknesses and emerging capability trends. These detailed results offer a solid foundation for identifying underexplored areas and informing future research directions.

In the revision, we will work to highlight additional non-obvious patterns and findings in the main text to better reflect the depth of our analysis.

We appreciate you carefully reviewing our paper and raising meaningful questions and constructive suggestions. Below we address your concerns one by one and are open to further discussion. Hope you can raise your evaluation.

---

Q1. Discussion of related evaluation benchmarks (HELM, VHELM, etc.)

A: Thank you for highlighting this. We acknowledge the contributions of holistic benchmarks like HELM, VHELM, and MMBench, which provide open, standardized infrastructures for evaluating vision foundation models. Our Generalist Benchmark aligns with this direction and is designed to serve as an open platform enabling broad community participation and transparent comparison of MLLMs. We will include a more detailed comparison with these existing efforts in the revision.

---

Q2. The benchmark’s large scale makes evaluation costly and impractical for rapid iteration...

A: We believe the benchmark's scale is a strength, reflecting diversity across tasks, modalities, and formats. For any single task, we limit evaluation to ~500 instances—computationally reasonable for most models.

To enhance usability, we will further propose a multi-scope evaluation structure with three leaderboard types:

• Scope-1: Full-spectrum leaderboard (current General-Bench), for general-purpose MLLMs.
• Scope-2: Modality-specific leaderboards, for modality-specialized models.
• Scope-3: Fine-grained task-cluster leaderboards under each modality.

All rankings are computed via our General-Level framework, allowing users to select appropriate evaluation scope based on their model’s capacity and resource constraints. This makes the benchmark scalable and adaptable: lightweight scopes allow fast iteration; broader scopes offer more visibility at higher computational cost—it's up to the user.

We do not plan to release dev sets for each task, as this benchmark is strictly designed for zero-shot evaluation. Model development and training are left to the developers.

---

Q3. The assumption that synergy enables generalists to outperform SoTA specialists is unrealistic.

A: We understand the concern regarding synergy and its comparison to SoTA specialists.

As explained in Appendix C.3 (“Rationality of Scoring Relaxation”), our design follows two steps:
(1) Define synergy as the core metric for capability levels;
(2) Propose a practical scoring method, given real-world model training constraints.

Ideally, synergy could be defined as a model performing better on a joint distribution $P_\theta(y|A,B)$ than on $P_\theta(y|A)$ or $P_\theta(y|B)$ separately. However, isolating such distributions is infeasible, as large generalist models are already jointly trained on many tasks. Retraining to cleanly separate task spaces is impractical.

Thus, we relax the evaluation: we treat cases where generalists match or exceed SoTA specialist performance in a task (without in-domain fine-tuning) as indirect but valid signals of synergy—i.e., effective cross-task/modality generalization.

This is supported by multiple studies showing generalists can outperform fine-tuned specialists:

• [1] shows GPT-4, via prompt engineering alone, achieves SoTA in medical QA benchmarks without fine-tuning.
• [2] shows OpenMedLM, via prompting techniques, surpasses prior fine-tuned open-source models on multiple medical benchmarks.
• Flamingo [3] outperforms fine-tuned specialist models on six vision-language tasks.

These findings support our core assumption: the stronger a model’s synergy capability, the more likely it is to surpass SoTA specialists when synergy is effectively activated. This avoids costly pairwise modeling and enables practical, scalable synergy evaluation.

[1] Can Generalist Foundation Models Outcompete Special-Purpose Tuning?
[2] OpenMedLM: Prompt Engineering Can Outperform Fine-Tuning in Medical QA
[3] Flamingo: A Visual Language Model for Few-Shot Learning
[4] Perceiver IO: A General Architecture for Structured Inputs & Outputs
[5] Segment Anything

---

Q4. Multimodal synergy does improve language—models like Vicuna, Qwen2, and LLaMA benefit from image-text data.

A: We appreciate this feedback and apologize for the unclear statement. We do not deny that multimodal data can improve language understanding. Our point is more specific: such improvement has not yet enabled models to outperform SoTA NLP specialists on core language tasks.

There is a clear distinction between enhancing language performance and exceeding SoTA NLP models. While models like Vicuna, Qwen2, and LLaMA show better results with image-text pretraining, our large-scale evaluation shows they still fall short of outperforming fine-tuned language specialists. Therefore, our statement does not contradict existing evidence. But for sure we will refine the statement for clarity in the revision.

---

Q5. Typos: Line 60 “Yhe” -> “The”

A: Thanks！Will correct this.

We greatly appreciate the reviewer’s recognition of our work and the thoughtful, constructive feedback. Below, we provide point-by-point responses to each comment. Hope you can reevaluate our work if you feel the response is effective and useful.

---

Q1. The “Limitations and Future Investigation” section is helpful, but future research directions could be elaborated further.

A: Thank you for this helpful suggestion. In addition to the Limitations and Future Investigation at section 7, we have also included a more actionable guide in Appendix C.5: “Path to Advancing Higher in General-Level”. This section outlines concrete guidelines for how future MLLMs can progress through the levels in our framework, offering clear directions for advancing synergy, modality coverage, and generalization—key factors in moving toward Level-5 multimodal generalists.

---

Q2. Do the “skills” in Tables 3–7 refer to meta-tasks? Are they directly mapped to the specific tasks in Appendix Table 103?

A: Not really, the “skills” listed in Tables 3–7 refer to meta-tasks, but they do not directly correspond one-to-one with the specific tasks in Appendix Table 103. Instead, each skill (i.e., meta-task) includes multiple specific tasks. For example, the skill Crack Detection includes the specific tasks Tire Crack Detection and Road Crack Detection. This hierarchical organization allows us to abstract task capabilities at a higher level, making the benchmark both manageable and scalable.

---

Q3. Why are pure language LLMs included in Table 9, given that the focus is on MLLMs?

A: We intentionally included language-only LLMs in the comparison to provide a reference point. This helps readers assess the performance gap between unimodal LLMs and MLLMs on NLP tasks. It also highlights a key finding of our benchmark: multimodality has not yet enabled MLLMs to outperform SoTA LLMs on core NLP tasks, which is critical for understanding the current limitations of multimodal synergy and what it would take to reach Level-5 performance.

---

Q4. Why do GPT-series models not achieve top rankings across all levels? And why do only three models reach Level 4? These results feel somewhat counterintuitive.

A: This is an excellent question. Although GPT models demonstrate strong performance in several areas, our analysis shows that they often excel in specific task types, such as vision comprehension, where they behave like specialists. However, they lack broad modality and task support, and in many cases, do not even support certain modalities or task formats.

Our General-Level scoring framework is based on two core principles:

1. The model should be a true generalist, i.e., capable across a wide range of modalities and task types.
2. Synergy is the key metric—performance gains must come from meaningful cross-modal, cross-task integration.

GPT models, while powerful, do not consistently meet these criteria across all levels, which explains their limited presence at Level 4 and absence from the top in other levels. Thus, the result is not only reasonable but aligns with our framework’s goals: to reward generality and synergy, not just isolated task excellence.

We greatly appreciate the reviewers' recognition of our work and the valuable feedback provided. Below, we provide detailed responses.

---

Q1. But I think the normalization and metric mapping methods are mentioned without thorough empirical validation, leaving their effectiveness in accurately comparing heterogeneous tasks not 100% substantiated.

A: We would like to clarify that the proposed normalization and metric mapping techniques are primarily designed to enable fair and consistent comparisons across heterogeneous tasks. This is necessary because certain evaluation metrics, such as FID, are not naturally bounded within the [0, 1] range, and lower values indicate better performance. While it is reasonable to directly compare models using FID within a single task, averaging performance across multiple tasks becomes problematic when combining metrics with different scales and monotonicities—such as FID (lower is better) and ACC (higher is better).

To address this, we carefully designed a normalization and metric mapping strategy that brings different evaluation metrics into a unified range. We also took special care in choosing appropriate scaling factors to ensure that the transformed scores remain faithful representations of the original quality measurements. For example, an FID of 25 and an FVD of 100 are mapped in a way that preserves their relative performance levels. This normalization process allows us to compute an overall average performance across tasks without introducing unintended biases.

---

Q2. The synergy effect design would benefit from more ablation studies to isolate the contribution of individual components.

A: Thank you for the suggestion. We would like to point out that a detailed analysis and discussion of the synergy effect in our multimodal generalist framework—across skills, comprehension and generation capabilities, and different modalities—is provided in Appendix B.7. We kindly refer the reviewer to that section for a comprehensive breakdown.

---

Q3. While the benchmark provides a broad and impressive evaluation across diverse tasks, it currently lacks in-depth analyses of failure cases, which could limit insights into the specific challenges faced by MLLMs.

A: Thank you for raising this important point. The core motivation behind General-Bench is to provide a comprehensive and systematic evaluation of MLLMs that goes far beyond conventional VQA-style assessments. To this end, we deliberately designed the benchmark to cover a wide range of task formats, modalities, and skills. This enables us to quantitatively and qualitatively measure MLLM performance from multiple perspectives, offering valuable insights into their generalization capabilities and limitations.

That said, we fully agree with the reviewers that in-depth failure case analysis is crucial for understanding model weaknesses and guiding future improvements. While our current focus has been on establishing broad coverage and performance patterns across dimensions, we acknowledge the value of qualitative case studies. In future work, we plan to include more detailed failure analyses and case-based evaluations to better illuminate the specific challenges MLLMs face and foster more targeted research efforts in this direction.

---

Q4. The paper assumes effective cross-modal synergy without adequately addressing integration and robustness issues; I think future work should focus on these aspects for improved model interoperability.

A: Thank for the reviewer’s constructive suggestion. In Appendix B.7, we provide a preliminary analysis of cross-modal synergy, particularly in terms of whether different models have learned synergy effects across modalities. We agree and will conduct a deeper exploration of integration mechanisms and robustness to modality-specific noise or failure for improving model interoperability in the revision.

---

Q5. While the paper posits that non-language modalities can enhance language intelligence, the experimental evidence for this reverse synergy might be largely absent.

A: We appreciate the reviewer’s comment and would like to clarify a possible misunderstanding regarding our claim. Our primary argument is that language serves as a strong prior to enhance other modalities, rather than the reverse. While we do not deny that non-language modalities can, to some extent, aid language understanding, our empirical findings suggest that current multimodal signals are not yet capable of boosting language performance beyond that of SoTA NLP models. In other words, we distinguish between helping language understanding and surpassing NLP SoTA performance through multimodal input—two very different thresholds. Our intention was to highlight this gap and motivate future work toward developing truly synergistic models where multimodal information can meaningfully and reliably elevate language intelligence beyond what is achievable through text alone.