Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?

Thank you for taking the time to review our work and for considering it well written. We hope that the following responses will also convince you of the strength and value of its contributions.

(W1, Q1 - No new method)

We respectfully disagree with the reviewer’s assessment regarding the lack of novelty, and note the other reviewers did not raise any concerns on novelty (with reviewer fqym explicitly stating that "this work appears novel"). While we do not propose a new technique to solve Bongard problems directly (an ambitious and still unsolved challenge) we introduce a detailed typology of model behaviours in such visual tasks, revealing strengths and limitations of current VLMs, helping us to understand their learned knowledge and abilities to reason.

Our contributions include:

• Introducing various prompt configurations (open-ended and multiple-choice) to explore how different prompt strategies affect VLMs’ performance on Bongard problems.
• Presenting two novel problem setups (Task 2 and Task 3) designed to analyze both the perceptual and the robustness aspects of VLMs.
• Conducting a human study involving 20 participants, which allows us to compare human performance with that of VLMs and thereby uncover systematic discrepancies.
• Identifying 10 Bongard problems that no deployed model in our study could solve in any task setup, highlighting clear gaps for future research.
• Demonstrating a substantial discrepancy between correctly identified rules in Task 1 and those that are correctly applied in Task 2, which unveals inconsitant model behaviour as the VLM can corretly find a rule but not use it reliably.

By documenting and analyzing these insights, we aim to inform the broader ML community about current limitations and future directions in solving Bongard problems. We believe this approach is in line with the guidelines’ emphasis on supporting new tasks, metrics, and problem framings, even when no novel algorithmic methods are introduced. We believe this set of contributions constitutes meaningful progress that the ICML community can build upon.

(W2 - Key findings have been discussed in literature already)

Even though there exists other works that investigate the shortcomings in abstract visual reasoning in VLMs, we think that several new insights can be drawn from our work, see the listed contributions above. For example our work identified a gap between problem solving (Task 1) and perception or rather applying rules correctly (Task 2) that is, to our knowledege, not yet discussed in the literature.

(Q2 - Compare to other methods)

We appreciate your interest in seeing broader evaluation results. However, our primary aim is not to identify the single best method for solving Bongard problems, but rather to reveal critical limitations and uncover how current VLMs, which can be opaque in their reasoning, perform on these tasks. We aspire to highlight systematic insights that can guide future improvements in model design and evaluation.

That said, we can discuss other methods that aimed at solving Bongard problems in more detail in related work. However they usually have a very different setup to VLMs and our aim is not to propose a method for solving Bongard problems, instead we use them as a diagnostic dataset to understand and analyse VLMs in more detail.

In that light we also consider the proposed references. Both present interesting approaches; however, they focus on the Bongard-LOGO dataset rather than the original Bongard problems considered in our study. While the approach in [1] could, in principle, be adapted to the original Bongard problems in an open-ended setting, [2] is specifically designed for classification tasks, making it difficult to apply directly to the open-ended nature of the original Bongard problems. Moreover, our primary focus is on evaluating the capabilities of VLMs independently of dedicated reasoning architectures, as their underlying mechanisms differ significantly. Nonetheless, we have included both works in our related work section to acknowledge their contributions.

(Q3 - more theoretical insights)

We are a bit puzzled by this remark. As we have highlighted above the contribution of this work is not to propose a novel AI method, but rather introduce and analyze a valuable dataset and typology of model behaviour for investigating this dataset in the context of VLMs. Can the reviewer clarify what they are referring to with "theoretical insights of the method"?

Thanks for your detailed response and the constructive feedback! Below we address your concerns.

(W1 - Human study for Task 2)

We agree that analyzing human performance in Task 1 alongside Task 2 would be an interesting future direction in a different setting with higher conceptual ambiguity and novel concepts. However, our primary focus in this paper was identifying the VLMs' perception errors in relation to their performance in Task 1.

We hypothesize that humans would excel in Task 2, as previous literature has identified concept recognition, spatial reasoning, and relational abstraction as fundamental aspects of human cognition [1,2]. Given that Bongard problems involve discriminative rules, we argue that human performance should remain robust in this task.

While rule verification tasks can, in principle, be challenging for humans in cases of rule ambiguity or when unfamiliar concepts are introduced, this is unlikely in the case of Bongard problems due to the discriminative nature of the rules. For example, in BP#16, asking a human whether a spiral turns clockwise or counter-clockwise (starting from the center) should be straightforward, as the rule is well-defined.

[1] Lake BM, Ullman TD, Tenenbaum JB, Gershman SJ. Building machines that learn and think like people. Behavioral and Brain Sciences. 2017

[2] Gentner, D. (2003). "Why we’re so smart." Language in Mind: Advances in the Study of Language and Thought

(C1 - Footnote o1 prompting)

The o1 evaluation is quite expensive, therefore we decided to evaluate only once for the concept detection experiment, as it would required 2400 more requests. If the reviewer thinks, it would be valuable to have the additional trials for o1 as well, we can still retrieve these results.

(C2 - BPs in training Set)

Unfortuantly, the opaque nature of the training processes, particularly for models developed by large corporations with proprietary datasets, makes it impossible to determine whether the models have actually been exposed to the BPs during training. However, even if such examples were present, the low overall performance of the models indicates that they have certainly not fully comprehended them. Your suggestion that the models might have learned shortcuts, could be one explanation why many Bongard problems solved in Task 1 remain unsolved in Task 2. It would be interesting future work to investigate this phenomenon on non-public test sets. Overall, this is an interesting hypothesis, and we included it in our discussion.

(Q1 - Representation of BPs (single images))

Interesting suggestion, we have investigated this with some of the models (c.f. table below). We see that the performance in Task 1 is comparable but more interestingly also the behaviour between Task 1 and Task 2 stays similar. This suggests that the image representation alone cannot be the reason for this discrepancy. We included these findings in the final paper.

Results analogously to Figure 5:

Thank you for your valuable feedback. We respond to your points in detail below.

(W1 - Selection of test data)

We chose to work with the original Bongard problems introduced in [1], as they were specifically designed to test pattern recognition capabilities in machines, yet they remain unsolved by current AI systems. These problems were carefully crafted and, in our view, already present a significant challenge on their own. Importantly, we find that they strike a valuable balance between simplicity (in terms of visual size and structure) and conceptual difficulty, a point supported by our accompanying human study.

We agree that datasets such as CVR and SVRT are also interesting, as they share some conceptual elements with Bongard problems, including shape variation, counting, insideness, and contact. Exploring how VLMs perform on these datasets would be an exciting direction for future work.

[1] Bongard, M. (1970). Pattern recognition. New York: Spartan Books

(S1 - Add abstract visual reasoning tasks)

Thank you for the thoughtful suggestion. We agree that Raven’s Progressive Matrices and odd-one-out tasks are important forms of abstract visual reasoning, and we appreciate the idea of transforming them into open-ended formats. However, as motivated in the paper, we believe that Bongard problems already provide a strong and sufficiently challenging foundation for evaluating the kinds of open-ended, concept-based reasoning we are interested in. In fact, many of the underlying visual concepts of Raven and odd-one-out (e.g., sameness, symmetry, numerical relations) are already well-represented in the Bongard problems.

That said, we agree that adapting other reasoning formats into an open-ended, explanation-driven setup as the reviewer suggests could be an exciting direction. While not necessary for the scope of the current study, we see this as promising future work, especially for expanding evaluation diversity and probing generalization across reasoning formats.

(Q1 - same BPs variance)

The "same" category includes only seven Bongard problems (see Table 4), which naturally increases the variance of participants’ results and makes it difficult to draw strong conclusions. It is possible that some individuals find the concept of "sameness" more intuitive than others, though further study would be required to explore this systematically.

In general, high variance across participants may stem from differing levels of task comprehension. For instance, the lowest-scoring participant - who did not solve any Bongard problems in the ‘same’ category - often formulated a rule that applied only to the left side of a problem while dismissing the right side as simply “not following that rule.” In #BP98, they labeled the classes as “triangle shapes” and “non-triangle shapes,” overlooking the crucial contrast between “triangles” and “quadrangles.” Such answers suggest they may not have realized that merely stating the inverse of a rule is insufficient to fully characterize a Bongard problem. Similarly, this participant used relative terms like “more circle shapes” vs. “fewer circle shapes,” highlighting a misunderstanding of the need for clearly defined classification rules independent of context.

(Q2 - black area in legend)

The legend entry for "solved" is meaning filled rectangle (red or blue) rather than black filling. We see that this can be misleading. To improve clarity, we consider updating it to a half-red, half-blue filled rectangle. Does the reviewer think that this would improve clarity?

Thank you for your detailed feedback and questions, we address them below.

(W1 - Differences to Malkinski et al.)

Malkinski et al. (2024) also evaluated VLMs on Bongard problems, concentrating on open-ended and classification-based settings. Our work shares their open-ended focus but goes further by adding two additional tasks (Tasks 2 and 3) to examine specific abilities we hypothesize are vital for solving Bongard problems. While Malkinski et al. compare synthetic Bongard problems to their real-world variant (Bongard-RWR), we target the original Bongard set, pinpointing especially challenging cases and identifying concept-detection inconsistencies. We also compare model output directly with human performance, offering more granular insights into how VLMs reason about these complex tasks. We have added this discussion to the related work section for greater clarity.

(W2 - Takeaway messages)

In the following we outline the key takeaway messages of our work.

• We introduce a typology of model behaviors for visual tasks, revealing both the strengths and weaknesses of current VLMs.
• Models often fail even with multiple-choice answers, indicating challenges not only in discovering correct rules, but also in recognizing when they have the right one (Task 3).
• We uncover a pronounced gap (Fig. 5) between solving Bongard problems (Task 1) and consistently identifying their relevant concepts (Task 2), highlighting an underexplored interplay between abstract reasoning and perception.
• We identify 10 Bongard problems that no model solved under any task condition (Fig. 11), offering a strong basis for future benchmarking.
• Our human study shows key differences between VLMs and people: participants collectively solved 95 Bongard problems (averaging 42 each, with best participant at 68), whereas all models combined solved only 66.

(Q1)

Unfortunately, the training data for both closed-source and open-source VLMs is not publicly available, so we cannot determine this with certainty. However, given that these large models are typically trained on massive corpora that include a substantial portion of publicly available data, it is likely that they have been exposed to Bongard problems during training - especially since the Bongard problems, along with their solutions, are publicly accessible (e.g., https://www.oebp.org/welcome.php). It would be interesting future work to design versions of BPs that are private or could get generated automatically to avoid the public exposure.

(Q2)

Yes, models were sampled three times. Tasks 1 and 2 were considered correct if answers/images were correct in at least 2 of 3 attempts. For Task 3, we sampled once and checked if a correct hypothesis was among the 20 proposed.

(Q3)

Please refer to W1 of reviewer uWew for more details.

(Q4)

Our goal in Task 2 is to examine whether models that successfully identify a discriminative rule (Task 1) also apply that rule consistently to each individual image. We hypothesize that a truly correct solution should translate into accurate classification of all images in a Bongard problem. However, our findings show that a model can solve a Bongard problem in Task 1 yet still classify certain images incorrectly in Task 2. For instance, GPT-4o solves BP#15 - distinguishing open from closed shapes - in all three Task 1 trials but fails to label an open triangle as open in Task 2. This indicates that even when VLMs manage to solve a Bongard problem, they may not reliably apply their “understood” rule at the level of individual images. While asking open-ended questions about each image (rather than providing ground-truth concepts) is an alternative approach, our intention here is to assess the consistency between rule discovery and rule application under controlled conditions.

(Q5)

We agree that a model may sometimes approximate the correct solution without consistently applying it across all images. However, this is not necessarily the behavior we seek in VLMs, particularly in applications that demand reliability and robustness. This highlights the importance of considering the perceptual aspect together with the reasoning abilities when evaluating VLMs.

That said, our comment was pointing to a complementary finding: that some ground-truth concepts underlying Bongard problems are well-detected by the models, even though they do not emerge as the rule for solving the BP in the general task. This may stem from confounding visual factors or the limited prominence of some ground-truth concepts. We find this to be an interesting observation, and think that it opens up promising directions for future work.

(Q6)

The figure is rather illustrative, however we intended to give an impression of different possible diagrams occuring in Bongard problems (displayed on the cards).