Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning

Thanks for the constructive suggestions. The detailed responses to the reviewer's comments are as follows.

Q1: Some statements could be misunderstood and the paper would benefit from a more precise formulation.

Thank you for pointing out the inaccurate statements. We will carefully check the manuscript to modify or remove the statements in the revised version.

Q2: Comparison and discussion to task-specific solvers

We conducted additional experiments on the task-specific solvers and multimodal LLMs GPT-4V and GPT-4o. Since UCGS-T can handle both selective and generative problems, we compare it to generative task-specific solvers PrAE [1], NVSA [2], GCA [3], ALANS [4] and RAISE [5] that support answer selection and generation. The task-specific solvers are trained separately on each dataset without the supervision of rule and attribute annotations to ensure consistency with the experimental setup of UCGS-T.

Experimental results show that UCGS-T outperforms generative task-specific solvers under the same setup. PrAE and ALANS cannot solve PGM problems since they only define visual concepts and abstract rules of RAVEN. Without additional annotations, task-specific solvers suffer significant performance drops, e.g., PrAE’s accuracy drops from 65.0% (reported by [1]) to 13.6% in our experiments. We exhibit the results of GPT-4V and GPT-4o on RAVEN from [6], where both models show lower accuracy than UCGS-T, ALANS and RAISE. The multimodal LLMs are not designed for abstract visual reasoning, therefore solving such tasks may require further prompt engineering and chain-of-thought design.

[1] Abstract spatial-temporal reasoning via probabilistic abduction and execution.

[2] A neuro-vector-symbolic architecture for solving Raven’s progressive matrices.

[3] Generating correct answers for progressive matrices intelligence tests.

[4] Learning algebraic representation for systematic generalization in abstract reasoning.

[5] Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection.

[6] What is the visual cognition gap between humans and multimodal llms?

Q3: Doubts regarding the replacement of the last image in Proposition 3.6

SVRT provides an image set for each rule containing different images generated by computer programs that follow the rule. Each image is assigned to only one panel of Bongard Problems (BPs). If the remaining images of a set are insufficient to form a complete panel, they will be dropped. Therefore, there are no duplicate images between BP panels even if one image is removed from the left and right problem panels.

Q4: The main paper is missing descriptions and references for the G1-set and SVRT datasets, while the supplementary material provides these.

Thank you for the suggestion. We will add the descriptions and references in the first paragraph of Experiments.

Q5: Is the model training of VQVAE and transformer done end-to-end or as usual one after another?

VQVAE is pretrained before training the remaining modules. We leave the image reconstruction loss in Eq. 9 to make it possible to finetune VQVAE end-to-end in the training stage. But we find that setting \lambda to 0 (i.e., freezing the parameters of VQVAE) is the best choice. Please refer to Appendix B.1 for the detailed descriptions.

Q6: Regarding the lack of clarity w.r.t. the used terms

In this paper, tasks refer to different abstract visual reasoning tasks such as RPM and BP. Problems refer to individual problem instances. Definitions describe the form of different tasks. Visual concepts refer to image attributes like object size and color. Rules are changing patterns on attributes (e.g., progressive changes). Thanks for the helpful suggestion to make the terms clear. We will introduce the terms at the beginning of the method section.

Q7: The motivation of including an object-centric baseline

Some task-specific solvers [1, 2] have realized abstract visual reasoning with object-centric representations. Therefore, we adopt it as one of the typical approaches to explore the performance of different visual representations in UCGS.

[1] Learning to reason over visual objects.

[2] Systematic visual reasoning through object-centric relational abstraction.

Q8: Other comments or suggestions about writing

Thanks for the constructive comments. In the revised version, we will clarify that the baselines are ablations on model architecture when introducing the baselines. And we will also remove the repeated parts (Lines 64-70 and the first paragraph in Section 3.1) in the manuscript.

Thanks for the constructive suggestions. The detailed responses to the reviewer's comments are as follows.

Q1: Comparison and discussion to RPM solvers and general-purpose systems

We conducted additional experiments on the general-purpose systems GPT-4V and GPT-4o and classic task-specific RPM solvers. Since UCGS-T can handle both selective and generative problems, we compare it to generative task-specific solvers PrAE [1], NVSA [2], GCA [3], ALANS [4] and RAISE [5] that support answer selection and generation. The task-specific solvers are trained separately on each dataset without the supervision of rule and attribute annotations to ensure consistency with the experimental setup of UCGS-T.

Comparison with task-specific solvers. The experimental results indicate that under the same experimental setup, UCGS-T outperforms the generative task-specific RPM solvers. Since PrAE and ALANS only define visual concepts and abstract rules for RAVEN, they are incapable of solving PGM problems. Our experiments reveal that, without additional annotation information, the performance of the generative task-specific RPM solvers significantly declines. For example, [1] reports that PrAE achieves accuarcy of 65.0% after rule-supervised training, which declines to 13.6% in our experiments.

Comparison with general-purpose systems. We also exhibit the results of the general-purpose systems GPT-4V and GPT-4o on RAVEN. The accuracy scores are reported by [6] where GPT-4V and GPT-4o have lower accuracy than UCGS-T, ALANS and RAISE. GPT-4o shows a notable performance improvement when provided with language descriptions of candidate images (from 19.2% to 38.8%), but still falls short of UCGS-T, ALANS, and RAISE. While the general-purpose systems GPT-4V and GPT-4o are not specifically designed for abstract visual reasoning, solving such tasks may require further prompt engineering and chain-of-thought design.

[1] Abstract spatial-temporal reasoning via probabilistic abduction and execution.

[2] A neuro-vector-symbolic architecture for solving Raven’s progressive matrices.

[3] Generating correct answers for progressive matrices intelligence tests.

[4] Learning algebraic representation for systematic generalization in abstract reasoning.

[5] Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection.

[6] What is the visual cognition gap between humans and multimodal llms?

Q2: Despite claiming to present a highly general model, it is overly focused on a very specific subset of idealized abstract visual reasoning problems.

As stated in Lines 26–29 of the abstract, UCGS is "general" since it aims to solve multiple abstract visual reasoning (AVR) tasks in a unified manner, which are often treated as independent in task-specific solvers. We agree that AVR tasks are widely used not because researchers are interested in achieving high scores in these tasks, but because they can reveal the core reasoning abilities of AI. Our goal is precisely to develop a more general framework for AVR—one that can exhibit multi-task AVR ability as humans, rather than simply “solving these tasks.”

To this end, we use representative AVR tasks as benchmarks. While the matrix reasoning tasks may seem simple in form, building multi-task solvers on them remains challenging. As shown in the table, GPT-4V and GPT-4o can hardly achieve performance comparable to task-specific solvers. The performance of UCGS-T illustrate the effectiveness of the proposed framework in the typical AVR tasks. Importantly, selecting these tasks does not mean our framework is limited to them. UCGS can extend to other tasks, such as visual analogy extrapolation problems [1], which shares a similar structure with matrix reasoning.

[1] A review of emerging research directions in abstract visual reasoning.

Q3: The propositions and proofs in section 3.1 seem somewhat unnecessary …

The significance of the propositions consists of two aspects. On one hand, it demonstrates how different forms of classical reasoning tasks can be described as a conditional generation processes. On the other hand, it explains how generative and selective tasks can be unified into one framework. Specifically, some selective tasks can be reformulated as an implicit answer generation problem, where the goal is to search for an option among predefined candidates that matches the generated result. This design allows a single conditional generative model to solve both types of tasks simultaneously, eliminating the requirement of an additional scoring network for answer selection.

Thanks for the constructive suggestions. The detailed responses to the reviewer's comments are as follows.

Q1: Concerns about the formulation and experiments of Bongard problems

Similar to the task-specific solvers, UCGS is a framework designed to solve abstract visual reasoning (AVR) tasks with only visual input/output. Therefore, we adopt a simplified version of Bongard Problems (BPs) mentioned in [1], where the task of describing rules via natural language is transformed into a classification problem on query images. Since the modified BPs have a similar definition to SVRT, we validate the ability of models to solve the modified BPs based on SVRT in our experiments. We will provide a more detailed explanation in the revised manuscript to clarify the setting of BPs.

[1] A review of emerging research directions in abstract visual reasoning.

Q2: Comparison and discussion to multimodal models and neuro-symbolic models

We conducted additional experiments on multimodal LLMs GPT-4V and GPT-4o and task-specific solvers. Since UCGS-T can handle both selective and generative problems, we compare it to generative task-specific solvers PrAE [1], NVSA [2], GCA [3], ALANS [4] and RAISE [5] that support answer selection and generation, where PrAE and NVSA are neuro-symbolic models. The task-specific solvers are trained separately on each dataset without supervision of rule and attribute annotations to ensure consistency with the experimental setup of UCGS-T.

The experimental results indicate that under the same experimental setup, UCGS-T outperforms the generative task-specific solvers.

Comparison with neuro-symbolic models. Without additional annotations, neuro-symbolic models show a significant performance drop on RPMs. [1] reports PrAE achieves 65.0% accuracy with rule supervision, but it drops to 13.6% in our experiments. PrAE and NVSA rely on predefined rules and explicitly defined representations for specific datasets, making it hard for reasoning with undefined concepts and rules. UCGS-T does not rely on manually designed concepts and rules. It performs independent reasoning process on each visual concept, inferring concept and rule without additional annotations.

Comparison with GPT-4V and GPT-4o. We exhibit GPT-4V and GPT-4o's performance on RAVEN from [6]. Both models have lower accuracy than UCGS-T, ALANS, RAISE and GCA. GPT-4o improves with candidate image descriptions but remains behind UCGS-T, ALANS, and RAISE. Since not designed for abstract visual reasoning, GPT-4o may require further prompt engineering and chain-of-thought design to improve the performance on RPM.

[1] Abstract spatial-temporal reasoning via probabilistic abduction and execution.

[2] A neuro-vector-symbolic architecture for solving Raven’s progressive matrices.

[3] Generating correct answers for progressive matrices intelligence tests.

[4] Learning algebraic representation for systematic generalization in abstract reasoning.

[5] Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection.

[6] What is the visual cognition gap between humans and multimodal llms?

Q3: Missing references to some papers of multimodal models and Bongard problems

Thanks for the suggestions. We will cite them in the revised version.

Q4: Can the authors explain the large improvement in accuracy in the ID tasks with comparatively little improvement compared to baseline in the out of distribution tasks?

The performance of models in ID tasks is influenced by the architecture and composed modules. In OOD tasks, the difference between the training data and unseen visual concepts or abstract rules dominate the performance of models, which may reduce the performance difference between baselines and UCGS-T.

Q5: The proposed architecture is rather intricate and complex. Are there any insights the authors would like to share beyond the performance on the benchmarks?

UCGS-T extracts global visual concepts from patch-based representations for reasoning. RPM tests often involve global rules, e.g., consistent object counts across panels, which patch-based and object-centric representations struggle to capture. UCGS-T addresses this by adding a module to extract and reason independently on visual concepts. Exploring better image tokenizers may help build models with stronger OOD reasoning capabilities.

Q6: In fig. 2 (a) it looks like there is a redundant positional embedding 2 /3.

The positional embedding 2 indicates the target position and is used as the query vector in the panel encoder.