Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection

Thanks for the constructive suggestions. The detailed responses to the reviewer's comments are as follows.

Q1. The authors fail to convince me, why we need to generate answers in arbitrary positions? I am eager to see what is the advantage of generating an arbitrary position over generating the right-bottom answer, not only the final performance, but also the rationale or the motivation behind this design.

We hope that RAISE can understand the overall rule of RPM panels, rather than learning to map given contexts to bottom-right images. The model's understanding of overall rules is illustrated through the ability to generate target images at arbitrary positions. Moreover, the arbitrary-position generation ability can help RAISE solve abstract reasoning tasks besides RPM. The experiment in Section 4.5 shows that RAISE can solve the odd-one-out problems by predicting the image at each position and comparing the prediction results with the realistic images to determine the rule-breaking image in the panel. If the model only predicts the bottom-right images, solving odd-one-out problems can be difficult as the rule-breaking images are not always located at the bottom-right.

Q2. Whether the model shown in App is used in all configurations or not?

Yes, all configurations of RAVEN/I-RAVEN use the same hyperparameters.

Q3. Too many hyper-parameters should be tuned.

In general, neural networks require inductive biases to learn attributes from images. Some generative solvers introduce artificially designed structures in representation learning, e.g., PrAE and ALANS design specific representations to encode attributes like object size, color, etc. While RAISE introduces inductive biases via hyperparameters on network structures and weights in the loss. By adjusting the hyperparameters, RAISE can adapt to different types of data and automatically encode attributes into latent concepts.

Q4. Experiments are not enough. I suggest using PGM-Neutral, PGM-Interpolation and PGM-Extrapolaton to confirm the effectiveness of RAISE.

We choose to evaluate models on RAVEN/I-RAVEN since their samples contain fewer noise attributes, and we can control the noise in RPMs to study the influence of noise in generative solvers by changing the settings of RAVEN/I-RAVEN, while PGM does not provide a similar way to control noise. We also refer to the configurations of PGM to generate two held-out configurations based on Center and O-IC and conduct experiments in Section 4.5 to evaluate the performance of models under OOD samples.

Q5. The authors also should clearly state why to use rule annotations, but not the answer images. Which indeed violates the RPM question.

RAISE uses images of correct answers to supervise the prediction results in training but does not require distractors in candidate sets. We think that there is no conflict between using candidate sets and rule annotations. The generative RPM solvers like PrAE and ALANS use both candidate sets and rule annotations in training. Rule annotations can guide the generative solvers to generate target images with specific rules. We also attempt to reduce the use of rule annotations by supporting semi-supervised training settings. RAISE does not use candidate sets for supervision, mainly because it is trained via arbitrary-position generation, but the datasets do not provide candidate sets for non-bottom-right images. We find that without using candidate sets, RAISE can generate better results than the compared solvers. During the testing phase, we evaluate the generative solvers in the conventional setting of RPM problems, which is to select answers from candidate sets based on the generated target images.

Q4. ... how do you pick the candidate from the given set? PrAE and ALANS actually only generate the hidden latents and compare in the latent space. Do you compare in the pixel space? Do you think comparing in the hidden space would help further improve performance of RAISE?

We select the candidate by comparing the latent concepts of the predicted target and candidate images. We conduct an experiment to illustrate the selection accuracy by comparing images and latent concepts. The experimental results are given in the following table.

The results indicate that RAISE achieves higher accuracy by comparing latent concepts. Due to the existence of noise, there can be multiple solutions to a generative RPM problem. Assuming the answer is the image having two triangles, in this case, models can generate the two triangles in various positions. These two-triangle images may significantly differ from each other in the pixel space. While they have the same concepts "Number=2" and "Shape=Triangle" in the latent space. Therefore, RAISE selects answers by comparing candidates and prediction results in the latent space rather than the pixel space.

Thanks for the constructive suggestions. The detailed responses to the reviewer's comments are as follows.

Q1. In general, I don't think the comparison is completely fair compared to other baselines, as some of the baselines only use the ground truth answers.

The arbitrary-position solvers cannot leverage rule annotations since they do not explicitly model the category of rules. For a more fair comparison, we provide the selection accuracy of unsupervised RAISE in the following table.

We find that RAISE trained without rule annotations still achieves higher average accuracy than baselines.

Q2. While RAISE only uses the rule annotations, grounding of rules to corresponding hidden concepts is also implicitly encoded in the matrix. So supervisory signals could actually be backpropagated to the attribute / concept part.

The rule annotations may indicate the correspondence between attributes and rules, such as "Attribute #1 follows Rule #1". However, they cannot provide the meta information about attributes or rules (e.g., Attribute #1 is the number of objects, or Rule #1 is the progressive change). The annotations may inform RAISE how many independent attributes there are, but RAISE needs to automatically discover the meaning of attributes and encode them into latent concepts correctly.

Q3. In evaluation, I do note that PrAE and ALANS are only trained on a specific split whereas RAISE are trained on more than one, and that is at least twice the data.

We provide an additional experiment to train ALANS and PrAE in the same single-task setting as RAISE (i.e., the models are trained and tested on each configuration). The test results are given as follows.

In the single-task training setting, the accuracy of PrAE increases significantly. However, we found that using the official code, ALANS can hard converge to the accuracy provided in its original paper (the regular I.I.D. evaluation results). The experimental results show that, in the same single-task training setting, RAISE achieves higher accuracies on most configurations and outperforms ALANS and PrAE on average accuracy. We will replace the accuracy of ALANS and PrAE in Table 1 with the single-task results in the revised version.

Thanks for the constructive suggestions. The detailed responses to the reviewer's comments are as follows.

Q1. How to generalize RAISE away from the RPM setting?

We provide three examples to illustrate the way to generalize RAISE away from the RPM setting.

1. RPM can be regarded as an analogical reasoning task. The first two rows are reference examples following specific rules, and models need to apply the rules to the third row to produce predictions (the third row is the problem panel where the first two images are queries, and the third image is the solution). RAISE can solve other analogical reasoning tasks by concatenating reference examples and problem panels into RPM-like matrices. The reasoning process of RAISE supports learning latent concepts and atomic rules on matrices of different shapes and generating solutions for given queries.

2. The odd-one-out experiment is another example. In this experiment, we leverage the arbitrary-position generation ability of RAISE to find the image that breaks the panel rules. We can set each image on the panel as the target image, generate the prediction of targets using RAISE, and find the one having the largest prediction error as the rule-breaking image.

3. Regarding image sequences as one-row matrices, we can use RAISE to generate the missing frames in sequences. For example, we can represent object appearances and positions in latent concepts, and different patterns of movement in atomic rules. RAISE can parse the movement pattern based on the given frames, and then apply it to the latent concept of position to generate the missing frames.

Q2. Any specific insights / technical novelty compared to previous works?

The novelty of this paper lies in the definition of rules and how we realize atomic rules in the latent space via learnable networks. RAISE defines the overall rules on the panel instead of mapping the context images at eight positions to the bottom-right one or predicting the third image from the previous two images in a row. Therefore, RAISE can predict answers at arbitrary and even multiple positions conditioned on the given context and solve non-RPM reasoning tasks like odd-one-out.

Q3. Details about candidate answer selection

RAISE computes distances between the generated target latent concepts and the latent concepts of candidates, and selects the candidate with the smallest distance.

Q4. Choice of $C$ and $K$

How sensitive is the model to varying C or K? How were these chosen? How adapted to the number of attributes and real rule numbers do they have to be? I’m aware they are different, but how sensitive is it e.g. can you use K=100?

To test the sensitivity of RAISE to $C$ and $K$, we train RAISE by varying $C$ ($C=4, 8, 16$) and $K$ ($K=4, 25$) on the original configuration. The results are given as follows.

RAISE is insensitive when increasing $C$ because it automatically generates some redundant latent concepts. When $C$ is too small to encode all changing attributes, the accuracy of RAISE will decline significantly (e.g., the accuracy of L-R, U-D, O-IC, and O-IG when $C=4$). To choose the proper $C$, we can first set a large number for $C$ and reduce it until the number of redundant latent concepts is reasonable. The results show that setting a large $K$ (here we set $K=25$ due to the limitation of devices) has no significant influence on the accuracy. But in general, we choose the hyperparameter $K$ by counting the unique labels in rule annotations.

Q5. Details about the Transformer baseline

1. Which encoder did you use? Was it a discrete representation? Transformers behave much better on VQ-like latents.

2. I somehow expected it to do better, e.g. if attributes were provided instead of pixels, isn’t a Transformer a pretty strong baseline?

The Transformer baseline uses the same encoder and decoder structures as RAISE. We conduct additional experiments to evaluate the Transformer baseline with discrete representations. The results are given as follows.

We take the vector quantization method of VQ-VAE [1] to discretize the representations. We set the size of the discrete latent space as 512, the dimensionality of each latent embedding vector as 32, and the weight of commitment loss as 0.1. The outputs of the encoder are split into 8 vectors of the length 32 and replaced by looking up the nearest embeddings in the latent space. The experimental results show that the discrete representations improve the performance on L-R and U-D but decrease the accuracy on O-IC and O-IG with in-out layouts.

We think the annotation of attribute values can be too strong supervision for model training. Though powerful RPM solvers like ALANS and PrAE leverage meta information about attributes to design visual representations, their training does not require any annotations of attribute values. We consider the Transformer as a baseline rather than a powerful solver because it is a black-box model and can hardly produce explicit representations of underlying rules on RPMs.

[1] Van Den Oord, Aaron, and Oriol Vinyals. "Neural discrete representation learning." Advances in neural information processing systems 30 (2017).

Q6. How needed was the rule supervision? Do you have numbers when you drop this to 0?

Without the supervision of rule annotations, the accuracy of RAISE is

The experimental results show that, the average accuracy of unsupervised RAISE is between the unsupervised arbitrary-generation baselines and the generative RPM solvers trained with full rule annotations.

Q7. Typos

1. It would have helped to explicitly write that r^c takes values between 1 and K.
2. \mu_T^c -> \mu_t^c in (4)
3. z_T^c -> z_t^c in (6)

Thanks for pointing out the typos. We will fix them in the revised version. We would like to clarify that, 2 and 3 are not typos. Here we use $T$ to index all target latent concepts.

Thanks for the constructive suggestions. The detailed responses to the reviewer's comments are as follows.

Q1. Discussion on “Bayesian vs Neural concept learning”

We thank the reviewer for providing the insightful works on systematic generalization. We would like to discuss Bayesian and Neural concept learning in three folds.

1. The learning objective. As mentioned by the reviewer, MLC uses meta-learning objectives to solve systematic generalization problems. In the discussion of MLC, it is mentioned that the hierarchical Bayesian modeling can be explained from the perspective of meta-learning. Grant et al [1] also indicate the connection between meta-learning and hierarchical Bayesian models. From this perspective, the global knowledge of atomic rules in RAISE can be regarded as a kind of global latent variable in hierarchical modeling. Although RAISE and MLC have different thoughts on model design, there may be close connections between their learning objectives.

2. Interpretability of latent variables. Bayesian and Neural methods can define basic modules in the learning processes, e.g., Functions in Neural Interpreters and atomic rules in RAISE. In Bayesian methods, we usually take interpretable latent variables in the learning process, e.g., using categorical random variables to indicate the types of the selected rules. While Neural Interpreters route inputs to different Functions by calculating specific scores. We can define more latent variables with specific meanings in DLVMs, e.g., RAISE defines a set of latent concepts to capture attributes (object size, color, etc.). In this way, visual scenes are decomposed into a simple set of latent variables, which may reduce the complexity of abstract reasoning and enable systematic generalization on attribute-rule combinations.

3. Solving multi-solution problems. In generative reasoning tasks, there can be multiple solutions for the given context of a problem. Bayesian methods can handle multi-solution problems by sampling latent variables from distributions (as shown in Figure 3, RAISE can produce results that are different from the original sample but still follow the correct rules). The experimental results of RAISE indicate that DLVMs can potentially solve generative RPM problems with multiple solutions in less noisy data.

[1] Grant, Erin, et al. "Recasting Gradient-Based Meta-Learning as Hierarchical Bayes." International Conference on Learning Representations. 2018.

Q2. The bottleneck with DLVMs & Solving real-world reasoning problems

If we can design general inference and generative processes, DLVMs can potentially solve different tasks in various scenarios. RAISE achieves the arbitrary-position generation ability by decomposing high-dimensional images into low-dimensional latent concepts and learning atomic rules shared among latent concepts. This design allows RAISE to be potentially applied to different real-world scenarios.

For example, when predicting missing frames in surveillance videos, latent concepts can represent object appearances (e.g., the appearance of a car) and spatial parameters (e.g., the position of the car), while atomic rules can encode different patterns of movement (e.g., turn left, stop, etc.). Through the given frames, RAISE can analyze the most likely movement pattern of the car, and then apply it to the latent concept of position to generate the missing frames. In this framework, latent concepts can also represent objects in scenes, and atomic rules can represent the actions learned from observation. We can apply the atomic rules to latent concepts (objects) parsed from new scenes and predict the results after applying the action.

Q3. What the authors mean by "a problem have a large number of solutions"

"A large number of solutions" here means there may be many solutions to the bottom-right answer. It is mainly caused by the noise in object attributes. Please refer to Figure 6 in the appendix.