GenVP: Generating Visual Puzzles with Contrastive Hierarchical VAEs

Response to Weakness 1

Although the author's experiments are very detailed, the rules of the puzzle generation task in this article are relatively simple, and the task's search space is relatively small.

Thank you for mentioning this.

Existing Methods Struggle in These Tasks: Actually, while the rules of RPM settings might appear simple to humans, it is far from straightforward for machines to process and reason about these rules effectively. Previous studies [2], [3] have shown that AI models often struggle to understand arithmetic or logical rules embedded within RPM tasks, highlighting the challenges posed by even seemingly simple rules.

Large Search Space: Additionally, while the number of rules in datasets like RAVEN may seem small (e.g. four in RAVEN), their combination with other factors, such as multiple objects (up to nine in RAVEN), numerous attributes (more than six in RAVEN), and varied attribute values (ten different colors in RAVEN)—leads to huge puzzle spaces. This leads to a total of 440,000 rule annotations in RAVEN [1]. For PGM, this number is even larger; consider only that each training set (eight in total) alone contains over one million different puzzles.

Compounding Complexity: It is also important to note that RPM problems involve a context matrix and a choice list. Together, these components interact with the rules, attributes, and values, making the problem space significantly larger and more intricate than it might initially seem. This complexity becomes even more apparent in datasets such as PGM and VAD, where only one attribute typically follows a rule, while others act as distractors. This distractor mechanism drastically increases the solution space and often proves detrimental to previous state-of-the-art models like RAISE, which struggle to handle such complexities effectively.

Finally, while exceeding the state-of-the-art performance in in-distribution settings, we also demonstrated that our method surpasses the state-of-the-art RAISE model in out-of-distribution scenarios. By testing our method on large solution space datasets like PGM and VAD, we showed superior performance (Section 4.1.3). We hope this explanation clarifies the concern and highlights our contributions to advancing research on the AVR problem.

[1] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5317–5327, 2019a.

[2] Hersche, M., Camposampiero, G., Wattenhofer, R., Sebastian, A., & Rahimi, A. Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning.

[3] Małkiński, M., & Mańdziuk, J. (2023). A review of emerging research directions in abstract visual reasoning. Information Fusion, 91, 713-736.

Response to Weakness 2

According to Figure 2, it can be seen that the image quality of the puzzles generated by this method is still not good enough; the edges of the puzzle elements are relatively blurry, and there are artifacts.

Thank you for mentioning this.

Firstly, we would like to emphasize that GenVP is the first practical generative model capable of creating complete RPM puzzles, a significant advancement over existing methods that lack such generative capabilities.

Improving the image quality of generated puzzles is an important direction for enhancing GenVP's overall performance. One way to enhance the image quality is by using a more powerful architecture (e.g., increasing the number of layers or parameters), utilizing higher-resolution images, or continuing training to further reduce the image reconstruction loss. In our experiments, we selected the checkpoint with the first best puzzle-solving accuracy for evaluation. However, GenVP rapidly achieves state-of-the-art accuracy, which remains consistent in subsequent training iterations.

To address the reviewer's concern, we repeated our experiments and retained the last checkpoint that achieved the best puzzle-solving accuracy while also minimizing the image reconstruction loss. This checkpoint maintains state-of-the-art performance in puzzle-solving accuracy while producing sharper images with fewer artifacts.

We will update the figures in the main paper with these improved results to better reflect the enhanced image quality.

We are delight to find our response address the reviewer's concern. However, we do wish to ask if there is further concerns and comments that prevent the reviewer from changing their score if they find the concerns addressed. Since there is still time left for the discussion period and the authors will have one more day (Nov. 27) to response after the open discussion phase end, we would really appreciate the reviewer to give us further feedback on any remaining questions that is preventing the reviewer from updating the score and gives us the chance to clarify them again for you. We thank the reviewer again for their work and hope to hear from you soon.

Response to Weakness 1

Limitation of the application scenarios. The design of the generative model seems rather tailored to RPM problems. This focuses on a very specific problem hence its significance might be limited. It is unclear how to apply GenVP in other abstract visual reasoning tasks.

This is a good question. We would like to reiterate that

• The RPM problem is a solid benchmark that examines the machine's abstract visual reasoning capability.
• We have tested our GenVP model across multiple datasets with different settings, therefore demonstrating its generality.
• Our method has been shown to be superior in the in-domain experiments. Our generative approach also helps us to outperform state-of-the-art methods on the out-of-distribution settings (Table 2), e.g., compared to the RAISE model (ICLR 2024).
• We have also provided experiments that showcase how our approach can scale up to puzzles with large solution spaces.
• Adapting the AVR to direct application is indeed a valid and important research topic; yet there is still room to improve the generalizability of these AVR models within the RPM framework. Our research, therefore, leans toward the latter topic, and we have demonstrated through our results that we made a significant contribution with our proposed GenVP structure.

Response to Weakness 2

Requirement of annotation of rules. Can a GenVP be trained without supervision of rules? For human beings, one does not require to learn rules through supervision, the rules can be discovered by the test subject.

Thank you for this insightful question. While humans acquire the skills to discover rules without explicit supervision, learning-based generative models like GenVP face challenges in achieving similar levels of abstraction and generalization without guidance. Rule annotations serve as a crucial component for training AVR models like GenVP by providing the structure needed to effectively learn the generative process.

• Rule Annotations Enable Controlled Generation: Rule annotations prompts GenVP to generate puzzles governed by specific rules, ensuring structured and interpretable puzzle generation. This aids in both the explainability of the model and its ability to reliably adhere to desired rules in generated puzzles.

• Consistency with Prior Methods: As with other state-of-the-art models in this domain (e.g., GCA, ALANS, PRAE, RAISE), GenVP relies on rule annotations for effective training. This reliance aligns with current approaches that use supervision to achieve high performance.

• Challenges of Unsupervised Learning: Training GenVP without rule annotations would require the model to infer rules solely from visual patterns, a highly challenging task for AI models. While humans excel at this due to innate reasoning abilities and prior knowledge, GenVP benefits from the explicit guidance provided by annotations.

Inspired by your comments, we conducted an ablation study where we excluded rule annotations during GenVP training. (Rebuttal) Table 2 below shows the results. While performance decreased, GenVP still outperforms the existing model RAISE in the rule annotation-free scenario.

(Rebuttal) Table 2: GenVP puzzle-solving accuracy when trained without rule annotations.

These findings underscore GenVP's potential to address unsupervised learning challenges in visual reasoning tasks. Despite certain limitations observed on RPM datasets, the substantial improvement achieved by GenVP over previous approaches marks a significant step toward reducing dependence on rule annotations. This progress opens an exciting avenue for future research, focusing on developing models capable of human-like reasoning without requiring explicit supervision.

Dear Reviewer,

Thank you for your response. We are glad that our previous response addressed most of your concerns. Please let us know if there are any additional points we can help clarify.

In fact, the rule matrix variable can be treated as a latent variable. In this case, our training objective is equivalent to the ELBO with a strong prior that forces the variational distribution of the rule latent variable $q_\phi(\mathbf{R}|\mathbf{Z}_o, \mathbf{Z}_r)$ to the ground-truth rules during training.

$L_R = \beta\_2\mathbb{E}\_{q}\left[ \log \frac{p(\mathbf{R})}{q\_{\boldsymbol{\phi}}(\mathbf{R}|\mathbf{Z}\_o,\mathbf{Z}\_{r})}\right] = \beta_2\left( \underbrace{\mathbb{E}\_{q}\left[ \log p(\mathbf{R}) \right]}\_{\text{Generative Graphical Model Term}} - \underbrace{\mathbb{E}\_{q}\left[ \log {q\_{\boldsymbol{\phi}}(\mathbf{R}|\mathbf{Z}\_o,\mathbf{Z}\_{r})}\right]}\_{\text{Inference Graphical Model Term}}\right)$

In practice, during training, the $L_R$ term reduces to a reconstruction loss between the ground truth rule matrix (which is observed in the generative graphical model) and the inference graphical model rule prediction ($q\_{\boldsymbol{\phi}}(\mathbf{R}|\mathbf{Z}\_o,\mathbf{Z}\_{r})$).

After training GenVP, at test/inference time, we can consider separately the generative graphical model (depicted with red arrows in Figure 1) and the inference graphical model (depicted with black arrows in Figure 1).

Inference Graphical Model (Black arrows in Figure 1):

In the inference graphical model, a complete RPM puzzle $\mathbf{X}$ is observed. Following the black arrows in Figure 1, we infer/predict the rule latent variable $\mathbf{R}$. This prediction marks the end of the inference graphical model pipeline.

Generative Graphical Model (Red arrows in Figure 1):

In the generative graphical model, the rule matrix $\mathbf{R}$ serves as the "root" node. A user can select/observe a desired set of rules $\mathbf{R}$, and by following the red arrows in Figure 1, the model generates a complete puzzle $\mathbf{X}$ that adheres to the specified rules.

For an observed/selected $\mathbf{R}$, the user has two options:

1. Direct Rule Specification: The user can design their own rule matrix $\mathbf{R}$ to generate puzzles.
2. Derived Rule Specification: The user can provide an existing puzzle $\mathbf{X}$, first pass it through the inference graphical model to predict $\mathbf{R}$, and then use this $\mathbf{R}$ as input to the generative graphical model to generate a new puzzle $\mathbf{X}'$. In this case, we are using the inference graphical model because we start from a puzzle $\mathbf{X}$, but we need to emphasize that it is not necessary to use a puzzle $\mathbf{X}$ to generate a new puzzle $\mathbf{X}'$; someone can construct/observe a rule matrix $\mathbf{R}$ and construct a complete puzzle by just using the generative graphical model (learned decoders).

Both approaches are supported by GenVP, showcasing its flexibility.

Summary

In the inference graphical model, the rule matrix $\mathbf{R}$ is inferred from the given puzzle $\mathbf{X}$.
In the generative graphical model, the rule matrix $\mathbf{R}$ is observed or specified, and it serves as the foundation for generating a new puzzle $\mathbf{X}$.

Thus, the trained inference graphical model (learned encoders) predicts the rules $\mathbf{R}$ of a given puzzle $\mathbf{X}$, while the trained generative graphical model (learned decoders) generates a puzzle $\mathbf{X}'$ from a given set of rules $\mathbf{R}'$.

Dear Reviewer,

Next, we provide examples outside the standard RPMs (like those found in RAVEN-based or PGM datasets) where GenVP can be employed, showcasing our approach's broader applicability.

• Flexibility with Input Format:

  GenVP is not limited to the $3 \times 3$ input format of RPM puzzles and can be readily generalized to arbitrary $M \times N$ puzzle settings with similarly arbitrary choice list size.

• Application to Question Answering (QA):

  More generally, the $M \times N$ puzzle settings can be interpreted as a form of Question Answering (QA) or dialogue, where the rows correspond to dialogue turns (e.g., $x_{i,1}, \ldots, x_{i,N-1}$ is the $i$-the question, $x_{i,N}$ is the answer to this question or the $i$-th turn dialogue completion). The puzzle can then be viewed as giving examples of $M-1$ QA pairs and seeking the answer $x_{M,N}$ to the last question $x_{M,1}, \ldots, x_{M,N-1}$, from the list of possible answers.

  The key significance of applying our GenVP framework to the above setting is the ability to explicitly learn and interpret the attributes and rules that underpin this QA/dialogue and subsequently use them to generate new instances of QA/dialogue that employ these rules and/or attributes. This is in contrast to many other models today that approach similar QA/dialogue modeling problems in a completely agnostic, uninterpretable manner.

  In visual puzzles, the dialogue elements $x_{i.j}$ are images, but it is not difficult to conceive that this could be generalized to combinations of images and words, leading to Visual Question Answering (VQA), or natural language QA alone.

• Application to Other Puzzle Settings:

  GenVP can also help solve Bongard problems, c.f., [1]. In Bongard-LOGO, we're given two groups of six images. One group follows rules A, and the other follows rules B. Our task is to match each of the two candidate images with the correct group based on shared properties. GenVP can be tailored for Bongard problems by replacing an image from each group with a candidate image. By evaluating which candidate image most consistently follows either set of rules, we identify the proper grouping for each candidate.

[1] Nie, Weili, et al. "Bongard-logo: A new benchmark for human-level concept learning and reasoning." Advances in Neural Information Processing Systems 33 (2020): 16468-16480.

Dear Reviewer GBgR,

Thank you for your review and engagement during the discussion period.

In response to your questions, we have provided explanations in our last two messages, specifically addressing the role of the rule matrix representation in GenVP (Rule Matrix in GenVP message) and presenting examples of applications beyond standard RPMs where GenVP can be utilized (Examples of GenVP Applications message).

With the ICLR Discussion Period concluding soon (Dec. 2nd (AOE) for reviewers and Dec. 3rd (AOE) for authors), we kindly request your feedback on whether our responses address your concerns or if there are any additional questions or suggestions you would like us to address.

Thank you once again for your time!

Best,

The Authors

Dear Reviewer,

We apologize for the confusion; this is an important point.

Firstly, let us consider the standard VAE model,

$\mathbb{E}\_{q(\mathbf{z}|\mathbf{x})}\left[\log p(\mathbf{x}|\mathbf{z})\right] - KL(q(\mathbf{z}|\mathbf{x}) || p(\mathbf{z}))$

in which case we have the KL term:

$KL(q(\mathbf{z}|\mathbf{x}) || p(\mathbf{z}))$

Then, for the prior distribution of $p(\mathbf{z})$, we can assume it follows a prior distribution $\mathcal{N}(0, \epsilon)$, with $\epsilon \to 0$. Consider $\mathbf{z}$ like our rule matrix.

Similarly, in GenVP, we model the rule matrix categorical prior ($p(\mathbf{R})$) parameters with Dirichlet priors with very large values for the concentration hyperparameters $\alpha$.

The above modeling ensures our GenVP's ELBO correctness. We have included this explanation in our manuscript as well. Thank you for pointing out this important point, which helped us improve our paper's clarity.

Dear Reviewer GBgR,

Thank you for your response. We are glad you found the data-dependent prior formulation helpful for illustrating how the rule annotations were used during training to learn robust rule predictions in GenVP. We would also like to thank you for recognizing the importance of our work.

We want to reiterate that our latest response on the data-dependent prior is consistent with our original paper. In Eq. 7 of the original paper, $p(\mathbf{R})$ is the prior distribution of a latent variable $\mathbf{R}$; therefore, the only (minor) change needed is to clarify that “$p(\mathbf{R})$ is data-dependent prior” in the revision. This clarification is certainly important and will be included in the camera-ready version.

Finally, we kindly ask if you might consider adjusting your rating if you agree that our response has effectively addressed your concerns and added value to the understanding of our work.

Thank you for your time and constructive feedback.

Best,

The Authors

Response to Weakness 3

Using VAE-based generative solvers or contrastive loss is not novel in RPMs. Is there any novel technical or insightful design in GenVP compared to previous approaches.

Thank you for your question. Since the same question was posed by Reviewer YBT2 in Weakness 1, to avoid duplicate answers, we kindly ask you to see our response to Reviewer YBT2 (above).

Response to Question 1

Could the authors explain the process of answer selection in detail?

To select the correct or most likely answer in an RPM puzzle, we rely on the properties of the most likely solution, which is the candidate that forms the largest number of rules when placed in the context matrix. Next we describe the steps for finding the puzzle solution from a pre-trained GenVP model.

Context Matrix Completion: For each answer option in the choice list, we create eight (i.e. in RAVEN-based, PGM datasets) complete candidate puzzles by filling the missing bottom-right position in the context matrix with one of the answer choices. (Each complete candidate puzzle now has 9 images.)

Rule Prediction with GenVP: For each complete candidate puzzle, we apply the GenVP inference pipeline to predict the puzzle rules (using the Mixture of Experts (MoE)).

Answer Selection: We then select the candidate that maximizes the number of active rules in the rule matrix prediction, including rules like constant, progression, distribute-three, and arithmetic. This is the same as selecting the candidate minimizing the number of random/no-rule in the rule matrix prediction. In practice, we return the answer choice that minimizes the count of random/no-rule encodings in the matrix (for example, the encoding $10000$ indicates a no-rule or random pattern, so we aim to minimize occurrences of this encoding).

Response to Question 2 (Part A: Optimization Objective Hyperparameters and GenVP Performance)

Will the selection of hyperparameters heavily influence the performance of GenVP? The RPM datasets contain just small sets of object attributes and rules, and the selection of hyperparameters can probably matter a lot in this case. The authors should discuss this as well.

This is a good question. Following the reviewer's suggestion, we conducted additional ablation studies to evaluate the influence of hyperparameters on GenVP's performance. Our results, shown in (Rebuttal) Table 3, indicate that GenVP is not overly sensitive to hyperparameter tuning, as performance remains robust across a range of configurations.

(Rebuttal) Table 3: GenVP Puzzle Solving Accuracy for the RAVEN dataset. We train four different versions, using different hyperparameter values $\beta_i, \beta_G, \beta_L$ for $i=1,\dots,6$.

Below we discuss these new results in detail.

1. Rule Matrix Weight ($\beta_2$):
   In our primary experiments, we assigned a higher weight ($\beta_2 = 250$) to the rule matrix prediction term compared to other ELBO terms ($\beta_i = 1, i \neq 2$). To test sensitivity, we set all $\beta_i, i=1,2,\dots,6$ to 1, effectively removing the emphasis on the rule matrix. This resulted in an improvement in performance (~94.7% to ~95.0%) demonstrating that the model is robust even when the weighting is altered.

2. Contrastive Loss Weights ($\beta_G, \beta_L$):
   In our main experiments, $\beta_G$ and $\beta_L$ were set to 20. In the additional experiments, we evaluated three configurations ($\beta_G, \beta_L = 1, 20, 40$). Results showed that very small weights ($\beta_G, \beta_L = 1$) caused a modest degradation (~92.1% accuracy), but increasing the weights ($\beta_G, \beta_L = 20, 40$) stabilized performance at ~95%. This suggests that GenVP benefits from reasonable tuning of contrastive terms but does not require fine-grained optimization.

3. Performance Consistency Across Hyperparameters:
   Across all tested hyperparameter configurations, GenVP maintained strong puzzle-solving accuracy. The observed variations in performance were within an acceptable range, indicating that the model’s core design, alongside its generative and contrastive training schemes is robust to hyperparameter changes.

In conclusion, while thoughtful adjustment of certain hyperparameters, such as $\beta_2$, $\beta_G$, and $\beta_L$, can optimize performance, GenVP does not rely heavily on precise tuning. Its stability across a wide range of settings underscores its adaptability and reliability in RPM puzzle solving tasks.

(continued to Question 2 Part B...)

Response to Weakness 1

Many important factors in GenVP should be carefully tuned. For example, and in the sampling process should be carefully handcrafted.

We appreciate the reviewer’s concerns regarding factor tuning and the sampling process in GenVP. We would like to clarify that GenVP’s strong performance is not due to excessive fine-tuning but rather to its sophisticated design and integration of generative and contrastive training schemes.

• Robustness Across Datasets and Architectures: GenVP achieves state-of-the-art results across five datasets (RAVEN, I-RAVEN, RAVEN-FAIR, PGM, and VAD), including challenging ones like PGM and VAD, where existing methods, such as RAISE, struggle to generalize. Importantly, these results were achieved without extensive re-tuning across different architectures (CNN, ResNet, ViT) or image resolutions (64x64, 128x128, 224x224), as demonstrated in our ablation study (Table 13, Section D.2).

• Minimal Hyperparameter Adjustments: The only notable adjustment was for the rule matrix ELBO term $L_R$. In particular for $L_R$, $\beta_2$ was increased to balance the dimensions of the rule latent variables and the pixel-space image vectors. This adjustment is straightforward and ensures effective optimization without requiring extensive experimentation.

• Sampling Process: The sampling process in GenVP follows a standard VAE-based framework. Each intermediate latent variable is modeled as a Gaussian random variable, with GenVP predicting its mean and variance for straightforward sampling. Thus, the process is not handcrafted but aligns with established probabilistic modeling practices.

Overall, the consistent results across datasets, architectures, and resolutions underscore GenVP’s robustness and adaptability, which we attribute to its principled design rather than meticulous fine-tuning. We have included the discussion above in our revised paper. We also performed a number of new experiments for different hyperparameter values (See our response to Question 2 ( experiment results in Rebuttal Tables 3 and 4)).

Response to Weakness 2

The proposed model seems to rely heavily on ground truth rule annotations. GenVP leverages rule annotations of each RPM sample in the training process. This limits the applicability of this method to unlabelled visual reasoning datasets. The authors should investigate why the models cannot learn the rules well without annotations.

Thank you for your question. We would like to clarify that all existing models addressing RPM tasks (GCA, ALANS, PRAE, RAISE) also rely on these rule annotations. Therefore our comparison is fair.

We also agree that designing models capable of abstract reasoning without relying on rule annotations is indeed an essential challenge for advancing unsupervised visual reasoning.

GenVP benefits from rule annotations to learn structured, interpretable representations, ensuring that the generated puzzles align with specific rules. This approach provides a level of control and explainability not seen in other methods. However, we acknowledge the desire for methods that minimize supervision and can generalize to unannotated datasets.

New Results without Rule Annotation: Inspired by your comments, we conducted additional experiments to train GenVP without any rule annotations, using only the puzzle images and their correct candidate label. The results in (Rebuttal) Table 2 highlight GenVP's ability to perform reasonably well without explicit annotations, achieving significantly higher accuracy than prior work. In particular, for RAVEN, GenVP improves the performance of the prior work, RAISE, from 54.5 to 70.3 by $\approx 16$ points. This indicates that GenVP captures useful latent representations even without direct supervision of the rules. However, we recognize that there is room for improvement. In future work, we aim to explore self-supervised or weakly supervised approaches for rule discovery, which could significantly broaden the applicability of GenVP to unlabeled datasets. These directions will be highlighted in the revised manuscript.

(Rebuttal) Table 2: GenVP performance when trained without rule annotations.

We appreciate the reviewer’s suggestions and hope that these results demonstrate GenVP's potential to generalize beyond annotated settings and improve on this compared to the previous SOTA model RAISE (ICLR 2024).

I appreciate the authors for the detailed response. Most of my questions have been resolved. The additional results illustrate the efficiency of tuning hyperparameters related to the training ELBO. But I still wonder if GenVP is sensitive to the hyperparameters of the model architecture, e.g., $K$, $K_{Z_o}$, $K_R$, $N_R$? I noticed that these hyperparameters are differently configured across datasets.

Response to Question 2 (Part B: Hyperparameters for Latent Space Dimensionality and GenVP Performance)

Dear Reviewer YfAj,

We conducted additional experiments to explore the impact of latent variable size on GenVP's performance.

(Rebuttal) Table 4: GenVP Puzzle Solving Accuracy for the RAVEN dataset. We trained five different versions of the model, using varying hyperparameter values $K$, $K\_{Z\_o}$, $K\_{Z\_{\bar{o}}}$, $K\_{Z\_r}$, $K_R$, and $N_R$, which control the size of GenVP latent variables.

As observed in (Rebuttal) Table 4, GenVP consistently achieves high accuracy across a range of hyperparameter configurations.

Notably, if the latent variable dimensions and total number of model parameters are too restrictive (do not have enough capacity to capture the puzzle complexity), we expect GenVP's performance to be affected. However, although representing each pixel-level image of RAVEN with a latent vector $\mathbf{z}_{ij}$, $i,j\in\{1,2,3\}$ of dimensionality $K=32$ is restrictive (prior work, i.e. RAISE, uses 64-dimensional image encodings in the latent space), GenVP manages to achieve an accuracy of 91.6% (first row in (Rebuttal) Table 4). As the model total parameters and the capacity of the latent space vectors increase, GenVP's accuracy improves and stabilizes at $\sim95$%. This indicates that GenVP is robust to architectural hyperparameter choices, though models with large enough latent vectors generally yield better results.

Thanks for the additional experiment on tuning architecture-related hyperparameters. The results provided by the authors have addressed my concerns. I will raise score to 6.

Response to Weakness 1

The technical novelty of the proposed method seems to be marginal since the authors directly employ the existing techniques (e.g., contrastive learning scheme and MoE). More detailed discussions and analyses are required to demonstrate the contribution of the proposed method.

Thank you for your question. Below, we would like to clarify the novel technical and insightful design choices in GenVP compared to prior approaches:

A Novel Task - Complete RPM Puzzle Generation:

• GenVP is the first VAE-based model to tackle the complete RPM puzzle generation process. Previous works focused solely on puzzle solution generation, leaving the broader task of complete puzzle modeling unexplored.
• This shift in scope required the development of a novel graphical model enabling controlled and interpretable generation, which the proposed model, GenVP, embodies.

Novel Disentangled Latent Representations:

• GenVP introduces a disentangled latent space design, separating rule-relevant ($\mathbf{Z}\_o$) and rule-irrelevant ($\mathbf{Z}_{\bar{o}}$) factors. This data-driven disentanglement is both novel in RPMs and crucial for achieving robustness in puzzle generation and solving.

Novel Mixture of Experts (MoE) Design:

• The MoE architecture in GenVP is a sophisticated addition that enables precise and modular rule prediction. It avoids shortcut learning by leveraging both fully (representations from all puzzle images, i.e., $\mathbf{Z}_r, \mathbf{Z}_o$) and partially observed (a subset of puzzle images like $\mathbf{Z}^{prow}_o, \mathbf{Z}^{ctx}_o$) information across coarse ($\mathbf{Z}_r$) and fine ($\mathbf{Z}_o$) representations.
• This design is inspired by human reasoning: focusing on various local (e.g., combinations of two rows of a puzzle) and global (complete puzzle) relations to infer rules effectively.

Novel Contrastive Learning Scheme:

• GenVP’s contrastive scheme is novel and different from typical contrastive learning in two key ways:
  • Complete Puzzle Comparison: Instead of comparing single images like typical contrastive learning, GenVP compares entire puzzles, capturing global contextual dependencies.
  • High-Level Rule Representations: Unlike typical contrastive learning's low-level pixel or latent comparisons, GenVP operates on high-level rule representations. This ensures the model focuses on abstract reasoning rather than superficial visual similarities.
• Furthermore, GenVP’s contrastive learning contrasts valid and invalid puzzles, as well as different valid puzzles, offering a richer training signal that improves both generalization and interoperability.

Novelty in Visual Puzzle Reasoning Process:

GenVP focuses independently on different sub-parts of the puzzle to extract information about the relations and rules formed within those subsets. Ultimately, GenVP analyzes puzzles both locally (puzzle subsets, such as two rows) and globally (the complete puzzle). This approach mimics a meticulous reasoning process, carefully examining the given problem (RPM) to uncover the rules governing the visual puzzles.

In summary, GenVP is not merely an incremental improvement but a significant conceptual and technical leap in generative RPM solvers. Its novel disentangled representations, advanced MoE design, and innovative contrastive learning scheme distinguish it clearly from prior approaches, both in generative capabilities and interpretative reasoning. These contributions are directly reflected in GenVP’s superior performance across diverse datasets and its ability to model the complete puzzle generation process.

Response to Weakness 2

More ablation studies are required to demonstrate the contribution of the main component of the proposed method in the main paper. For instance, the mentioned rule estimators for the RPM-Level inference, the performance gain of the introduced MoE mechanism, and the contrastive learning scheme.

We thank the reviewer for their ablation study suggestions. In the Appendix of our submission, we included the following ablation studies to demonstrate the contribution of the main components of the proposed method:

1. Contrastive Learning Effect
   Tables 8 and 9 in the Appendix present GenVP's performance without the contrastive learning scheme, using only ELBO maximization (Equation 7). Our experimental results demonstrate the critical role of contrastive learning in enabling GenVP to detect puzzles with the strongest visual analogies—those satisfying the largest number of rules. This establishes the importance of the contrastive learning scheme for the model's puzzle-solving capabilities.

2. Effect of Different Ensembling Functions in the MoE Module
   We present different types of MoE function in Section C.6. and the corresponding performance in Tables 10 and 11.

Performance of Individual Rule Predictors vs. Mixture of Experts (MoE)
Following our reviewer's suggestions, we compared GenVP’s performance using individual rule predictors against its performance using the MoE (all predictors) for the RAVEN dataset. The results are summarized in Table 1.

(Rebuttal) Table 1: In the MoE column, we present the puzzle-solving accuracy using all rule predictors. In the remaining columns, we show the accuracy when using individual rule predictors $R(Z)$, where $Z$ denotes the input used for the individual rule prediction. For notation simplicity in the Table, for the rule predictions using only two rows, we use the notation $\mathbf{Z}^{prow}\_{ij}$ meaning that we use the two rows $i$ and $j$, for $i,j=1,2,3$. For the context-based rule predictions (one image is missing from the puzzle), we use the notation $\mathbf{Z}^{ctx}\_{-i}$ meaning that the $i$-th image of the puzzle is missing for $i=1,\dots,9$.

(continued in next message)

Response to Weakness 2 (continued)

In (Rebuttal) Table 1, we observe the following:

• State-of-the-Art Performance with MoE: GenVP achieves the best performance (95.0% accuracy) when combining rule predictors through the MoE mechanism, demonstrating its effectiveness.

• Performance of Individual Predictors: Among individual predictors, the puzzle-level representation predictor $R(\mathbf{Z}\_r)$ achieves the highest performance, followed by predictors using two puzzle rows ($R(\mathbf{Z}\_{13}^{prow})$, $R(\mathbf{Z}\_{23}^{prow})$) and context-based predictors ($R(\mathbf{Z}^{ctx}\_{-1})$, $R(\mathbf{Z}^{ctx}\_{-2})$, ..., $R(\mathbf{Z}^{ctx}\_{-6})$). Predictors using image-level representations ($R(\mathbf{Z}\_o)$) show comparatively lower performance.

  The high performance of $R(\mathbf{Z}_r)$ indicates that the puzzle-level latent variables exclude most of the information irrelevant to the rules (noise), making them robust for the puzzle-solving task.

  Additionally, we observe that the rule predictors focusing on only two rows outperform $R(\mathbf{Z}_o)$, which observes all three rows. By restricting these predictors to derive the rules from just two rows instead of three, the problem becomes more challenging, leading to greater robustness. This restriction prevents these predictors from exploiting shortcuts for rule prediction. For example, in the case of $R(\mathbf{Z}_o)$, the model might predict rules by focusing only on the first two rows of the puzzle, which would fail in scenarios where the third row contains an imposter image.

• Challenging Scenarios: The context predictors ($R(\mathbf{Z}^{ctx}\_{-7})$, $R(\mathbf{Z}^{ctx}\_{-8})$) perform poorly when the missing context matrix image lies in the same row as the negative candidate. This occurs because, in such cases, even though the last row contains an imposter image, the absence of the 7th or 8th image allows GenVP to imagine a plausible missing image that aligns with the puzzle rule.

  For example, consider a puzzle governed by an arithmetic rule based on the number of objects. The correct puzzle might have a number of objects (1,2,3; 2,2,4; 3,2,5), where the third image in each row is the sum of the first two. Now, suppose the last row contains an incorrect image, such as (1,2,3; 2,2,4; 3,2,4). This row violates the arithmetic rule, and GenVP should identify this as a puzzle with an incorrect candidate.

  However, if we consider the $R(\mathbf{Z}^{ctx}_{-7})$ predictor and observe a partially complete matrix, such as (1,2,3; 2,2,4; 3,-,4), this puzzle could still be valid under the arithmetic rule if the missing image were (1,2,3; 2,2,4; 3,1,4). Thus, when the 7th or 8th image is removed from a puzzle containing an imposter image in the 9th panel, GenVP can imagine a plausible image that corrects the rule violation, leading to a performance drop for these context predictors.

• Random Performance: Predictors like $R(\mathbf{Z}\_{12}^{prow})$ and $R(\mathbf{Z}^{ctx}\_{-9})$ exhibit random performance because the choice list images are missing, leading to identical images across all candidate puzzles. This result is both expected and desired, as the subsets of the candidate puzzles are identical in the image space and in GenVP's latent space, resulting in random performance.

These ablations provide insights into the contributions of the contrastive learning scheme, MoE mechanism, and rule estimators to GenVP’s superior performance. We have included the MoE vs. individual rule predictors' performance in the main paper.

Dear Reviewer YBT2,

Thank you for your review. We would like to provide a summary of our posted rebuttal:

• In our Official Author Response (Part A) message, we clarified the novel technical contributions and insightful design choices underlying GenVP.
• In Official Author Response (Part B), we referred to the ablation studies provided in our supplementary material and conducted additional ablations as requested. These new ablations investigate the role and performance of each rule predictor in GenVP, offering further insights into their individual contributions.
• In Official Author Response (Part C), we provided a detailed discussion on the MoE components in GenVP, elaborating on their role and effectiveness.

With the ICLR Discussion Period ending soon (Dec. 2nd (AOE) for reviewers and Dec. 3rd for authors (AOE)), we would like to kindly ask for your feedback on whether our responses address your concerns or if you have additional questions or suggestions for clarification.

Thank you again for your time!

Best,

The Authors