We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

• I would like to look at more variants of the neurosymbolic approach proposed in this work. One avenue worth exploring is a line of research that leverages domain knowledge, such as knowledge graphs, to identify pertinent concepts within an input. Active nodes within the graph could then be employed to pinpoint the common concept within the positive set of images.

We thank you for the suggestion! Indeed, leveraging KG is a brilliant idea of enhancing the neurosymbolic approach presented in this paper, which currently still replies on perception from other visual foundation model like BLIP. Please feel free to list a few papers that are relevant to the idea you mentioned above, we would love to cite and discuss them in the final version of our paper and if possible, benchmarking them on our dataset. Again, thanks for the insight.

• The evaluations used in this paper though really comprehensive, miss out on some more ways of evaluation. VLM-based approaches, like GPT4(V), that directly take images as input and can be prompted to obtain the desired input, could be used to identify the relevant concept from a collage of images given together. Since current VLM/LLM approaches are very susceptible to the way they are prompted, it is very important to prompt engineer them in a number of ways and then identify the best working one.

Your insights are notably forward-thinking, and we have already prepared results using GPT-4V. All parameters are aligned with GPT-4, except that the called model is "gpt-4-vision-preview" and the parameter "detail" set to "high" for fine-grained image representations. We splice raw images together in a manner similar to the examples in Appendix H, using only one query image each time, which is then inputted for GPT-4V reasoning. Here are the results (and they have been merged to the updated version as well). It can be seens that Bongard-OpenWorld remains a great challenge even with GPT-4V. We commit to exploring this further as our prompt engineering might not be that comprehensive given the limited time (and the non-stable performance of their API as well!).

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

• The presentation and writing could be clearer, making some parts of the paper difficult to understand, especially around the creation of the dataset in Sec. 2 and the precise problem setting. For example, it was not immediately clear that the labels of the two sets (positive, negative) are given and do not need to be inferred. The query image can belong to either, but the name "positive" suggests that this is the GT set of the query image, which is not the case. I am adding more clarifying questions in the questions section below.

Thanks for highlighting this point, we have revised our main paper for greater clarity. For your specific question here, the labels of the support set are given and without anything else. Each Bongard Problem includes both a positive and a negative query image.

• The problem setting is imbalanced. The positive set corresponds to a single concept while the negative set does not, but instead contains a subset of the complete concept. While this is not necessarily an issue and can be a design choice, there is no justification why this choice has been made. For instance, why not have both sets correspond to a single concept where the contrasting sets are close in semantics to make it a hard problem?

Our problem setting adheres strictly to the original Bongard Problem (a classic puzzle of computer vision and pattern recognition: https://en.wikipedia.org/wiki/Bongard_problem), which include several features:

• The support set comprises only images of positive and negative with their labels (just indicates whether it belongs to the positive or negative set).
• Images from the positive set should depict the same concept, while those belong to the negative set should not depict this concept and the concept depicted by the negative images can be varied -- this could lead to an "imbalance" on the number of concepts but yes, this is a designed choice made by the original Bongard problem.

But Bongard-OpenWorld also introduces some new features:

• First of all, both the positive and negative images includes distractor concepts (as we've demonstrated in Figure 1 of the original paper) -- that is, images from the positive set are in fact could also depict many more concepts besides the ground truth, alleviating the "imbalance" issue.

• Secondly, as you can see from the example problems we showed in the Appendix H, when we picking the negative images, we also seek to make them share some common concepts, this is relevant to the solution you mentioned above and indeed help create a hard problem.

• Finally, we even make positive and negative images could overlap on the concepts they depict. For example, given the ground truth concept is "red tie", both positie and negative concepts can depict "dog", resulting "dog wearing a red tie" (positive) and "dog wearing a blue shirt" (negative).

Again, we thank you for raising this issue, and we believe our design in Bongard-OpenWorld not only respect the original Bongard problem setting, but also mitigate the possible drawbacks, including imblancing and easy problems.

• ...Similarly to how different splits are evaluated in Table 2, it would have helped to show the performance of positive query images vs. negative query images in order to understand if this imbalance makes positive/negative queries easier/harder.
• Is there performance difference between positive and negative queries?

We thank you for your suggestion! Here are some additional results singling out the accuracy on positive and negative queries. As you can see, there might be slight difference for a single model, but we did not observe a pattern or preference towards positive or negative queries globally. Therefore Bongard-OpenWorld does not appear to exhibit a strong imbalance issue.

• Have you thought about not providing the labels "positive" and "negative" for the two sets to the methods? Why have you chosen this setup?

As we mentioned above, not proving the positive or negative labels could make the setting largely deviate from the original Bongard problem setting, which we choose to respect when creating this dataset. Also, we don't see a clear advantages of doing such. That include the "imbalancing" issue in your comment -- our response above has shown that our dataset does not appear to have a strong imbalance issue.

Dear Reviewer CSkd,

We would like to thank you for your prompt reply and we're happy to see some of your concerns have been addressed. Per the remaining issues, we would like to offer some further clarifications:

My point is: It is fine to adhere or deviate from a previously established setting, but there should still be a proper reason behind each choice such that the benchmark targets a clear goal in computer vision research. For some of my original questions above, good reasoning is currently missing.

We thank you for posting the references to the original Bongard problem. Indeed, our implementation slightly deviates from the original Bongard problem, in terms of the following:

• The two sets are not symmetrical -- in our case, only set A depicts a shared concept, while set B does not

• We add a query image and switch the main task from recognizing the concept (we still have it, as the captioning task, but it is not the "main task" that is presented in the "main result" section of the paper) to determine whether the query belongs to set A or set B.

• Some additional features on image selection, as we mentioned in the response above.

Now we dive into some reasoning on the various design choices (why making these modifications, why introducing several additional features, etc), including what we would like to benchmark, and how is this important to computer vision research.

1. both sets with labels, also an additional query

   To be more accurate, this "non-symmetrical" setting is effectively inherited from two seminal works: Bongard-LOGO[1] and Bongard-HOI[2]. We believe the idea behind this is: the original Bongard problem is rather challenging, as it requires the machine to directly produce natural language descriptions of the concept after reading images from both sets. Even with today's powerful VLMs, this is still quite difficult. Therefore, it is natural to simplify the question a bit, let's say, turning the original "concept regression" problem into classification, so more approaches, including many popular few-shot image classification methods (ProtoNet, MetaBaselines, etc) can be benchmarked here. A good idea would be to additionally introduce a query image, and categorize which set it belongs to. This creates a classification problem, while maintaining the core idea of the original Bongard problem -- to identify the concept of a set of images. Evidently, if the query is properly chosen, only when the model is able to identify the true concept and (implicitly) develop an image classifier for this concept, the query can be correctly categorized. From a computer vision research perspective, such modification democratizes the evaluation of the Bongard problem by creating a "simpler" version of it, and makes it possible to examine the progress of current computer vision models in terms of few-shot concept understanding, which is a very interesting yet crucial capability that is originally benchmarked by Bongard problems.

2. two "non-symmetrical" sets

   This is also inherited from Bongard-LOGO[1] and Bongard-HOI[2]. Compared to the original Bongard problems with two symmetrical sets, creating a non-symmetric "negative" set actually streamlines the reasoning process a bit. In the original Bongard problem, the concept induction process is repeated. We likely start with some hypothesis of the concept being depicted by set A, then we use set B images to refine them by ruling out those being depicted by set B. Next, we move to set B and do the same. If there is no clear answer, we may have to repeat the aforementioned steps until a verdict is returned. It can be seen that the induction processes of concepts from set A and set B are symmetrical as well. Therefore, by making the shared concept only exist in set A, we remove the reasoning process that is symmetrical to this in the original Bongard problem and therefore streamline the overall reasoning process. We admit doing such could introduce some "imbalance" issues. But as we already mentioned in the previous response, we've introduced several techniques to mitigate this. Back to what this design choice means to computer vision research in general: the idea is similar to 1., we would like to simplify the problem a bit so examining the progress of existing models, which are not strong enough to solve the original Bongard problem directly.

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

• In the experiments, the authors primarily focus on conducting investigations using real-world datasets, particularly the their self-constructed dataset. However, given the Bongard Problem, it raises concerns about the generalizability of the conclusions/findings obtained from real-world datasets to mathematical datasets.

Could you please provide a few examples of mathematical datasets? This would allow us to elaborate further. The primary focus of our benchmark is on real-world reasoning, or reasoning in a more realistic setting -- real-world images, open-vocabulary, or open-world concepts. Mathematical datasets, which we suppose primarily adopt synthetic images and close-world concepts, are not the main focus of this work.

• The experimental results seems to ignore the traditional models, and it remains a concern.

Our experimental results cover a broad spectrum of approaches, ranging from traditional few-shot learning methods, combinations of Large Language Models and Visual Language Models, as well as a neuro-symbol method, and offering a comprehensive analysis. Could you please specify the traditional model you are referring to? If you give a concrete reference, we would like to try.

We hope the above response can resolve your questions and concerns. Please let us know if there is any further question!

Hi Reviewer yoip,

Pls read and reply to the authors' responses.

Thanks, Cheston

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

• Some related work is missed. [A] (see references below) studies a setting very related to the proposed benchmark (though they didn’t use the terminology Bongard problems). They also created tasks using natural (‘real-world’) images from different datasets, from using computer vision datasets (rather than scraping the web).

We appreciate your suggestion and have incorporated this reference with discussion into our main paper. It is pertinent to note that the primary focus visual concepts of this work comprise various compositional attributes. Our proposed benchmark is characterized by its free-form nature, allowing a wide range of concepts with diverse elements.

• It would be great to add additional few-shot learning baselines to cover families of approaches that are excluded from the current analysis like approaches that perform FiLM-conditioning e.g. [B, C] (see references below) and approaches that train the backbone with gradient descent within each task, like MAML and Proto-MAML (the latter is a version proposed in the Meta-Dataset paper which is cited by this work).

We are grateful for your suggestions. We are currently implementing the FiLM-conditioning approach, this process will take a while. Regarding the MAML and Proto-MAML approaches, we encountered issues where its training process would collapse, leading to NaN. We believed similar issues have been observed in Bongard-HOI(http://arxiv.org/abs/2205.13803), i.e. some FSL approaches cannot produce meaningful results due to collapsing. Our hypothesis is having even fewer training data (in Bongard-OpenWorld) might have worsen this issue. We hence the exclusion of these results from our report.

• The paper has some clarity issues, perhaps owing to the fact that the authors tried to 'squeeze' a lot of content in the required number of pages. It’s hard to fully understand the different methods by reading only the main paper. I found the neuro-symbolic method proposed especially hard to understand (even after looking at the algorithm in the appendix). Please include some higher-level motivation and the intuition for the particular updates that it entails.

We have further clarified approaches in the updated paper. For your convinience, here we offer some higher-level motivation and the intuition for the neuro-symbolic method:

• In contrast to canonical FSL and large model based approaches, this approach begins with not just converting images into captions (ex. a dog is running on the grass), but further into concepts (ex. <dog, running, on grass>). This is done by employing GPT-4, the prompt is detailed in Appendix D. Our goal here is to mitigate the substantial noise present in image captions (both in single-round and multi-round approaches).

• With these concepts, we apply logical operations like AND, OR, and NOT, aiming at mimicking the human logical thinking process when tacking Bongard problems. Specifically (please note that this is just high-level illustration and some details for handling corner cases are omitted. Again, the code includes all the implementation details):

  1. Compute the intersection of the concepts of all positive images, this gives us the initial hypothesis on the true concept of this Bongard problem. If there is no intersection (possibly due to flaw in perception), a majority vote is adopted -- concepts that present in more than 4 positive images will be treated as the "intersection".
  2. For each negative image, we substract its concept from the "intersection" obtained in the previous step, this is to mimick how human rule out the concepts that is depicted by both the positives and the negatives, and finally reach to the concept that is uniquely depicted by the positives.
  3. We compare the updated "intersection" with the concept of the query image. If the "intersection" is a subset of the concepts depicted by the query, the query will be categorized as positive, otherwise deemed negative.
  4. The "intersection" is also the ultimate induced concept of this Bongard problem, exclusively depicted by the positives.

• As a result, such logical operations over concepts can deterministically produce binary predictions on the query image and induced visual concepts depicted by the positives.

Negatives contribute candidates that aid positives in updating their missed atomic concepts (potentially distractors). Conversely, positives provide candidates that help negatives refine their concepts (potentially overlaps).

Hi Reviewer bqT5,

Pls read and reply to the authors' responses.

Thanks, Cheston