ING-VP: MLLMs Cannot Play Easy Vision-based Games Yet

1. Limited Task Diversity: The six games, while varied, may not fully capture the range of planning and reasoning tasks MLLMs might encounter in real-world applications.

2. Current Model Limitations: The paper notes that current MLLMs struggle even on simple tasks, suggesting that the benchmark might be too challenging without significant advancements in multimodal reasoning models. The paper does not provide or discuss on potential direction for current weakness of MLLM on vision-based games.

3. No Difficulty Grading: The lack of difficulty grading may hinder the benchmark’s ability to track progress across varying model capabilities or subtle improvements in reasoning skills.

Some additional experiments that will improve the quality of the paper:

1. The authors appear to use either image-only or text-only inputs for the model. Conducting experiments with both image and text as inputs could further enrich the analysis and provide insights into the model's ability to handle multimodal information effectively. It would be good to include such experiments.

2. For the text-only setup, did you use a language-only model, or did you use a vision-language model (VLM) without image tokens? It would be valuable to conduct experiments in both scenarios to compare their effectiveness.

3. In Fig. 3, it is unclear whether the planning errors are calculated only in the absence of perceptual errors, or if they include cases both with and without perceptual errors. Could you clarify this distinction?

4. Including experiments to emphasize the following aspects would add valuable insights to the study:

• Prompt sensitivity. Assessing whether the model’s performance on planning tasks is sensitive to variations in prompt wording could reveal if prompt adjustments lead to improved outcomes.
• Prompt non-determinism. Evaluating the model's responses under different hyperparameters, such as temperature, would demonstrate the stability of its outputs.
• Repeatability. Running experiments multiple times and reporting the mean and standard error would provide a clearer picture of performance consistency and reliability.

5. It would be helpful to highlight the model's behavior across all games, from very simple settings to more complex ones, to show how it performs at each level of complexity.

6. Finally it would be very useful if the authors further provide any additional insights into why the model performs so poorly on these planning and reasoning tasks?

1. The paper does not clearly differentiate itself from previous work. For instance, some studies evaluate MLLMs in agent-based environments, such as GUI and robotics, assessing multi-step reasoning and spatial abilities.
2. One of my concerns is the limited scope of the scenarios. The six visual games are pretty specific, and the perceptual or reasoning abilities they test may not adequately represent MLLMs' capabilities in a broader range of real-world scenarios.
3. I noticed the paper does not include advanced prompt-based methods like Chain-of-Thought (CoT) prompting, which is a standard approach in most benchmarks.

1. The motivation of your paper is limited, as it has been proposed by many papers working on MLLMs for visual games and puzzle games, such as [1],[2],[3] (And if you search key words [game, puzzle, VQA, VLM] in some database, you can find there are more works in this topic). I agree your benchmark may have some advantages (like grouping some tasks together) and may be better than these papers in different aspects, you should make a comparison and highlight these differences. And also I don't think grouping them together in a benchmark is a big contribution.

2. It is worth highlighting why these tasks are important for MLLMs. For example, as you showed in section 3.3, all these tasks can be transformed into a deterministic environment with predefined states, actions, so why not use RL in these tasks? Even if your goal is testing the zero-shot performance of MLLMs, it is still weird to us, because MLLMs are not pretrained with these tasks. If you let a human to answer it, humans still need additional training before start doing these tasks.

3. What might be the upper bound performance of these tasks? Human's performance or RL models' performance? I feel that you should present them in your paper.

4. Can you give us an example of the Image-text and text-only testing sample in Table 1? What is the environment prompt you are using to interact with the test-only LLMs? And how about if you apply CoT for these tasks? The current version is quite unclear.

5. If it is possible, maybe you could try the RLHF to finetune some open-sourced MLLM for these tasks. It can be used as the baseline because all of your tasks are a deterministic environment and can be easily adapted into a reward function.

References:
[1] Chia, Yew Ken, et al. "PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns." arXiv preprint arXiv:2403.13315 (2024).

[2] Ghosal, Deepanway, et al. "Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoning." arXiv preprint arXiv:2403.03864 (2024).

[3] Kraaijveld, Koen, et al. "COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes." arXiv preprint arXiv:2409.04053 (2024).