DSRF: A Dynamic and Scalable Reasoning Framework for Solving RPMs

W1: While combination is valid, novelty remains relatively limited.

While our framework builds upon certain established deep learning components, its core design is fundamentally different from prior architectures.

• MVRP differs significantly from spatial pyramids.

MVRP diverges from spatial or scale-based architectures prevalent in vision models [a,b]. It adopts a channel-wise hierarchical organization, where each view represents a distinct feature space. This design integrates DDCPs for dynamic reasoning adjustments based on context–target contrasts and MRM for progressive rule integration. Together, they form a hierarchical reasoning framework that explicitly mirrors compositional structure of latent rules. Our DSRF systematically enumerates rule combinations in a principled and scalable manner, enabling comprehensive coverage of complex attribute interactions while maintaining generalizability.

• DDCP enables dynamic reasoning.

DDCP enables dynamic reasoning behaviors by modeling context-target differences via gated attention guided by Gram matrices. This design allows the model to adapt its reasoning pathway based on specific context–target configuration of each sample, rather than relying on static mappings. DDCP learns dynamic mappings that continuously adapt to evolving context-target relationships, enabling robust generalization under distribution shift and novel rule configurations.

• DSRF: Novel dynamic and scalable reasoning framework

The integration of MVRP, DDCP and MRM forms a novel dynamic and scalable framework to reason for complex abstract rules. Moreover, it is adaptive and generalizable, as evidenced in OOD experiments on RAVENs [7, 8, 18], Unicode [9] and PGM [6] in Table 2-4, setting our approach apart from existing static or handcrafted reasoning strategies in abstract visual reasoning.

We believe DSRF provides a principled and extensible framework for scaling up reasoning capacity without overfitting. We will emphasize these distinctions more clearly in the final version.

[a] Lin, Tsung-Yi, et al. "Feature pyramid networks for object detection." CVPR, 2017.

[b] Yang, Guoyu, et al. "Asymptotic feature pyramid network for labeling pixels and regions." TCSVT, 34(9), 2024: 7820-7829.

W2/Q2: Scalability analysis

Thank you for the insightful question.

• Performance saturation in scaling DSRF

The relatively small gain in Table 2, from 97.3% to 97.7%, is primarily due to the ceiling effect, where top-performing models all achieve high accuracy on RAVENs. Notably, DSRF-S significantly boosts the previously best-performing model, PredRNet [13], from 94.3% to 96.7%, while DSRF-M and DSRF-L further improve to 97.3% and 97.7%, all three significantly outperforming baselines despite the ceiling effect and demonstrating monotonically increasing gains by scaling.

• Consistent performance gains by scaling up DSRFs

Table A compares DSRF-S, DSRF-M and DSRF-L on all 6 RPM datasets. We added the held-out settings of PGM, i.e., PGM-H.O.P (Held-out Attribute Pairs), PGM-H.O.TP (Held-out Triple Pairs), PGM-H.O.T (Held-out Triples), PGM-H.O.LT (Held-out Line-Type) and PGM-H.O.SC (Held-out Shape-Color) settings, each testing the model’s ability to generalize to novel combinations of attributes, unseen pairs or individual triples, and new attribute-object pairings.

Scaling up DSRF yields consistent and substantial gains across all scenarios. Notably, under more challenging OOD settings on Unicode [9] and PGM [6], we observe more pronounced improvements, demonstrating that scalability becomes increasingly valuable when the reasoning task is more complex and the domain shift is substantial.

Table A. Comparisons between DSRF-S, DSRF-M, DSRF-L on 6 RPM datasets.

• Performance-complexity tradeoff analysis.

While model scaling yields diminishing returns where near-optimal performance has already been achieved, it remains crucial for tackling harder OOD challenges. As evidenced above, scaled models exhibit enhanced capability in handling more complex cases, advancing toward developing more generalizable AI systems.

• Practical meaning of scalability.

Previous models typically scale reasoning capacity by stacking reasoning blocks, but generalize poorly [13, 17]. Our goal is not merely to increase model size, but more importantly to expand reasoning capacity in a structured way while avoiding overfitting. Specifically, the MVRP increases coverage of compositional rules by distributing reasoning across multiple attribute-based views. The DDCP adapts to varying rule distributions and focuses on relevant context-target differences. These mechanisms systematically scale reasoning capacity and effectively capture the latent rules present in training data while maintaining strong generalization on OOD samples, enabling the model to handle increasingly diverse and difficult reasoning scenarios.

We will add more scalability analysis in the final manuscript.

W3/Q3: Unsubstantiated overfitting claim:

As noted in PredRNet [13] and HPAI [17], stacking reasoning modules often induces overfitting.

Table B. Ablation of the number of reasoning blocks K in PredRNet [13] and HPAI [17].

Both PredRNet [13] and HPAI [17] exhibit performance saturation or marginal degradation beyond three blocks, revealing inherent scalability constraints and depth-induced overfitting tendencies. These observations align with our motivation to pursue horizontal scalability through multi-view decomposition rather than relying on deep stacking.

We will further clarify this in the final manuscript.

W4/Q4: Lack of comparison with LLMs:

In literature, researchers have applied LLMs for AVR tasks [c]. Table C compares DSRF-M with recent LLMs on RAVEN [18], reported by Apple [c]. LLMs exhibit near-chance performance on RAVEN, while dedicated AVR models like DSRF achieve almost perfect performance.

Table C. Comparison to LLMs on RAVEN [18].

[c] Zhang, Yizhe, et al. "How Far Are We from Intelligent Visual Deductive Reasoning?." ICLR 2024 Workshop: How Far Are We From AGI.

W5/Q5: Inconsistent structural presentation**.**

We will revise the manuscript following your comments.

Q1: Each view in MVRP captures distinct attribute combinations**?**

Thank you for the insightful question.

• Channel-wise decomposition facilitates feature disentanglement [b]

Although we do not impose explicit constraints such as orthogonality loss or entropy regularization, our model encourages view specialization implicitly through the multi-head view mapping in MVRP. Each view divides the global feature representation into attribute-relevant subspaces. Prior studies have shown that channel-wise decomposition can facilitate feature disentanglement and attribute grouping [d]. We allocate dedicated reasoning branch to each view, allowing different views to attend to different relational patterns. In addition, the subsequent DDCP modules adaptively extract localized reasoning patterns within each view, further reinforcing specialization across views during training.

Following your comments, we ablate the case that we add an orthogonality loss as shown in Table D. This results in no performance improvement and may even lead to a slight accuracy drop due to minor conflicts between multiple loss terms.

Table D. Comparison to adding orthogonality loss.

• Clustering analysis

Since figures can’t be reported during rebuttal, we compute cosine similarity between the four views' feature maps in DSRF-M's first layer, following [e]. The results in Table E indicate low interview similarity, supporting the emergence of view-wise specialization.

Table E. Feature similarity across views.

We will include t-SNE clustering analysis in the final manuscript to show that each view captures different aspects of the input.

[d] Hu, J., Shen, L., & Sun, G. Squeeze-and-excitation networks. CVPR, 2018, pp. 7132–7141.

[e] Zhao, Wenliang, et al. "Towards interpretable deep metric learning with structural matching." ICCV, 2021.

The author's response has resolved most of my concerns, and I recommend increasing their score.

W1: Performance gain from model design or model growth**?**

Thank you for your insightful comments. The two principal design objectives of this work, as emphasized in the title, are dynamic adaptability and scalability. Specifically, we aim to develop a scalable abstract visual reasoning framework while effectively tackling the challenge of overfitting.

• Simply stacking reasoning modules would lead to overfitting

As noted in the original PredRNet [13] and HPAI [17] studies, stacking deep reasoning modules frequently induces overfitting, despite aiming to enhance reasoning through additional blocks.

Table A. Experimental results of stacking K reasoning blocks in PredRNet [13] and HPAI [17].

As shown above, both PredRNet [13] and HPAI [17] exhibit performance saturation or marginal degradation beyond three blocks, revealing inherent scalability constraints and depth-induced overfitting tendencies. These observations align with our motivation to pursue horizontal scalability through multi-view decomposition rather than relying on deeper stacking.

• Our scaled-up models maintain generalization

As shown in Tables 1–3, scaled-up models (DSRF-M/L) consistently outperform smaller variants (DSRF-S/M) on both RAVEN [7,8,18] and Unicode [9] benchmarks. We acknowledge that DSRF falls within a relatively large parameter range (10M–40M), but our scaled-up architectures successfully learn latent rule representations while preserving robust generalization capabilities across diverse task distributions.

This robustness is attributed to the novel architecture of MVRP, DDCP and MRM. Each view is processed independently through modular DDCP–MRM branches, which mitigates overfitting by encouraging localized, attribute-sensitive reasoning. The DDCP block promotes relational comparisons instead of direct memorization, while the MVRP introduces compositional constraints that prevent the model from relying on shortcut patterns. Together, these mechanisms guide the model to learn transferable rule structures, allowing larger variants to benefit from increased capacity without overfitting.

• Ablation shows the effectiveness of designed modules.

To assess the individual contribution of our core modules, we provide detailed ablation studies in Table 6. The ablations show that removing designed components leads to substantial performance degradation, even under the same model scale. In addition, in Table 7, we show that replacing the reasoning module of state-of-the-art models by DSRF-S and DSRF-M significantly improves the reasoning accuracy. All these ablations support our claim that the improvement of performance comes from the proposed design rather than simply growing models.

We will further clarify this point in the final manuscript and provide more justifications.

W2: Lightweight scheme of DSRF

Thank you for your insightful comments. Although the original manuscript did not specifically investigate lightweight implementations, our framework inherently supports efficient deployment through its modular architecture, enabling straightforward adaptation to resource-constrained scenarios.

• Lightweight scheme by adjusting MVRP hyperparameters.

In particular, the MVRP allows for controlling the channel dimensions of feature mappings within each view. By decreasing the dimensionality of each mapped view, the model can be scaled down to more lightweight variants with lower memory and FLOPs, while still maintaining core reasoning capabilities.

• Results of lightweight DSRFs

By reducing the channel dimensions of each view by half, we can construct a lightweight variant of DSRF-S and DSRF-M, denoted as DSRF-S-Light and DSRF-M-Light. We conduct comparison experiments on three RAVEN datasets and three PGM settings (PGM-N, PGM-I, PGM-E). The lightweight variants achieve substantial reductions in model size and computational cost (GFLOPs) while maintaining reasonable accuracy.

Table B. Comparisons with lightweight DSRFs.

We will supplement the lightweight schemes and related experimental results in the final manuscript and the supplementary materials accompanying the publication.

Q1: In Table 4 and Table 5, how does the DSRF-S model perform?

We supplement the results of DSRF-S as follows.

Table C. Evaluation of DSRF-S and DSRF-M on PGM [6] and RVP [37] datasets.

While DSRF-S demonstrates inferior performance relative to DSRF-M, it maintains competitive advantages over alternative baseline models, consistent with our expectations.

Q2: Performance loss by replacing DDCP Gram matrix with an alternative scheme

Thank you for the question. We conduct additional ablation experiments where the Gram matrix in DDCP is replaced with a covariance matrix. This modification results in a performance drop of approximately 0.4% on three RAVEN datasets and 0.6% on three PGM settings (PGM-N, PGM-I, PGM-E), indicating that the Gram matrix is more effective than the covariance matrix.

Table D Comparisons between Gram matrix and covariance matrix in DDCP.

We will supplement this ablation in the supplementary materials accompanying the publication.

W1: Computational complexity Analysis

The reviewer may overlook the model complexity analysis in the last two columns of Table 2, and the associated discussions at Line 233-236 in Sec. 4.2, where we reported GFLOPs and model size (M) of compared models. One of our key contributions is to introduce a scalable design to boost the reasoning capability of our models. As shown in Table 2, our DSRF-S, DSRF-M and DSRF-L outperform all the compared models. Through systematic model scaling, we demonstrate progressive improvements in reasoning capability across the DSRF architecture variants (S→M→L), validating the effectiveness of our scalable design paradigm.

W2: Data efficiency

Thank you for your comments. Following your comments, we conducted experiments on the SVRT dataset using 500, 1k, 5k, and 10k training samples respectively, following GAMR [b]. This evaluation provides a controlled setting to assess generalization under limited data. The results are summarized below (left: same-different tasks/right: spatial related tasks). DSRF achieves strong performance across various SVRT tasks, even when trained with limited samples such as 500 samples per task. The results demonstrate its ability to learn abstract visual rules efficiently.

Table A. Sample efficiency analysis on the SVRT dataset.

We will add this data efficiency analysis into the in the final manuscript and the supplementary materials accompanying the publication.

[a] Webb, T. W., Sinha, I., and Cohen, J. D. Emergent symbols through binding in external memory. In ICLR, 2021.

[b] Vaishnav, M. and Serre, T. Gamr: A guided attention model for (visual) reasoning. In ICLR, 2023.

[c] He, Kaiming, et al. "Deep residual learning for image recognition." CVPR, 2016.

[d] Vaishnav, Mohit, et al. "Understanding the computational demands underlying visual reasoning." Neural Computation 34.5 (2022): 1075-1099.

W3: Ambiguity in OOD evaluation

Thank you for your comments. We would like to clarify that OOD (Out-of-Distribution) may refer to different levels of distribution shift.

• Distribution shifts due to novel rule-attribute combinations on PGM and Unicode datasets**.**

• We have conducted experiments on OOD settings on Unicode Analogy [9], where all the symbols in the training set do not appear in the test set. The results are reported in Table 3 and the related discussions can be found at Line 237-250.

• In addition, we have conducted experiments on OOD settings on PGM [6], Interpolation that evaluates generalization to intermediate attribute values, and Extrapolation that tests the model’s ability to reason beyond the observed attribute range. The results are reported in Table 4 and the related discussions can be found at Line 251-260.

• Furthermore, we provide the experimental results on held-out settings of PGM in the table below, i.e., PGM-H.O.P (Held-out Attribute Pairs), PGM-H.O.TP (Held-out Triple Pairs), PGM-H.O.T (Held-out Triples), PGM-H.O.LT (Held-out Line-Type) and PGM-H.O.SC (Held-out Shape-Color) settings, each testing the model’s ability to generalize to novel combinations of attributes, unseen pairs or individual triples, and new attribute-object pairings absent during training. Our models perform best across almost all settings.

Table B. Comparisons on PGM OOD regimes.

• Distribution shifts due to novel samples on RAVENs

The OOD settings across I-RVN, O-RVN, and RVN-FAIR involve shifts in image distributions and answer choice generation strategies, testing novel samples while the underlying rule-attribute combinations remain largely consistent. Under these OOD settings, our DSRFs significantly outperforms compared methods, as shown in Table 2.

We acknowledge that the OOD settings in PGM [6] and Unicode [9] are more challenging than OOD settings on RAVENs. Nevertheless, our model demonstrates strong performance under these OOD regimes, as reported in Tables 3 and 4, showing that DSRF is capable of generalizing beyond memorized rule structures and adapting to new reasoning patterns.

We acknowledge the current ambiguity in defining out-of-distribution (OOD) evaluation settings. For the final version, we will rigorously specify the OOD criteria for each benchmark (RAVENs [7,8,18], PGM [6], and Unicode [9]), explicitly delineating their distinct characteristics.

Q1: Has DSRF been evaluated under held-out settings of PGM or RAVEN?

In the original manuscript, we evaluate DSRF under two OOD settings of the PGM dataset: Interpolation and Extrapolation. Following your suggestion, we have additionally provided results on held-out settings of PGM [6] as shown in the reply to W3 above. To our best knowledge, no held-out settings were defined for RAVENs [7, 8, 18].

Q2: Non-matrix abstract visual reasoning such as Bongard problem

Yes, the core components of DSRF are general and can be adapted to other forms of abstract visual reasoning, including non-matrix tasks like the Bongard problems.

• Adaptation to Bongard Problems

Specifically, the DDCP block only requires a set of context-target pairs and does not rely on any fixed spatial layout. Similarly, the MVRP can be applied to grouped features based on visual attributes rather than grid positions. To adapt DSRF to Bongard problems, one may treat the positive or negative example sets as a separate context-to-target reasoning branch and apply DDCP to extract the reasoning patterns.

• Evaluation of Bongard problems

Following your comments, we add the experimental results of DSRF on Bongard-OpenWorld [c].

Table C. Comparisons on Bongard problems**.**

The results of GPT-4 [d] and SNAIL [e] are obtained from [c]. We evaluate DSRFs following the same evaluation protocol [c]. DSRFs outperform the compared methods on Bongard problems, even the large language model like GPT-4 [d].

We will update the results and the related discussions in the final manuscript.

[c] Wu, Rujie, et al. "Bongard-openworld: Few-shot reasoning for free-form visual concepts in the real world." arXiv preprint arXiv:2310.10207 (2023).

[d] Bubeck, Sébastien, et al. "Sparks of artificial general intelligence: Early experiments with gpt-4." arXiv preprint arXiv:2303.12712 (2023).

[e] Mishra, Nikhil, et al. "A Simple Neural Attentive Meta-Learner." ICLR 2018.

W1. Comparison on PGM OOD regimes

Following your comments, we have evaluated DSRF-M and DSRF-L on all PGM tasks, including PGM-H.O.P (Held-out Attribute Pairs), PGM-H.O.TP (Held-out Triple Pairs), PGM-H.O.T (Held-out Triples), PGM-H.O.LT (Held-out Line-Type) and PGM-H.O.SC (Held-out Shape-Color) settings, each testing the model’s ability to generalize to novel combinations of attributes, unseen pairs or individual triples, and new attribute-object pairings.

Table A. Comparisons on PGM OOD regimes.

As noted by the reviewer, DSRF underperforms Slot-Abstractor [b] in Interpolation and Extrapolation tasks. This is mainly because DSRF adopts a more general reasoning paradigm without accessing attribute labels, while Slot-Abstractor [b] utilizes these attribute labels to model continuous attribute variations effectively.

On other OOD settings, DSRF consistently outperforms ARII [a] and Slot-Abstractor [b]. This is mainly because these settings involve more complex rule-attribute compositions, where DSRF's dynamic and modular design enables stronger generalization beyond fixed attribute-slot mappings.

DSRF-L surpasses ARII [a] in all PGM tasks and Slot-Abstractor [b] in 6/8 tasks, while DSRF-M outperforms ARII [a] in 7/8 tasks and Slot-Abstractor [b] in 6/8 tasks. These results demonstrate their superior performance in PGM OOD tasks.

We will include these experimental results in the final manuscript.

W2: Handling column-wise rule

Thank you for your insightful comments. We respond to your comments as follows.

• Intuitions for modelling row-wise rules only.

DSRF focuses on row-wise rules only as most datasets such as RAVEN [18], I-RAVEN [7], RAVEN-FAIR [8] only implement row-wise rules. We prioritized efficiency by focusing on row-wise reasoning, which suffices for most RPM tasks but admittedly limits the model.

• Not explicitly but implicitly model column-wise rules.

• While DSRF does not explicitly model column-wise rules, column-wise reasoning emerges implicitly through the shared weight architecture across rows. As the same DDCP module is applied to all rows, it encourages the model to learn generalizable patterns that may also align along columns, allowing the model to partially capture column-wise regularities.

• In all the datasets we evaluated, only PGM [6] implements column-wise rules. By utilizing row-wise DDCP only, DSRF performs excellently on PGM, as shown in Table 4.

• Incorporating column-wise DDCP.

Following your comments, we extend the framework by adding a column-wise DDCP reasoning branch, whose outputs are fused with the row-wise DDCP to enhance rule representation. The DDCP’s modular design enables easy adaptation for column-wise processing.

We evaluate this extension on PGM as shown below, where DSRF-M-RC/DSRF-L-RC denote the DSRF-M/DSRF-L variants utilizing both row-wise and column-wise DDCPs. By adding column-wise DDCPs, the performance significantly improves.

Table B: Results for DSRF-M-RC and DSRF-L-RC on PGM.

We will revise these results and related discussions in the final manuscript.

[a] Webb, T. W., Sinha, I., and Cohen, J. D. Emergent symbols through binding in external memory. In ICLR, 2021.

[b] Vaishnav, M. and Serre, T. GAMR: A guided attention model for (visual) reasoning. In ICLR, 2023.

Q1: Evaluating sample-efficient of DSRF on SVRT

Following your comments, we conducted experiments on the SVRT dataset using 500, 1k, 5k, and 10k training samples respectively, following GAMR [b]. This evaluation provides a controlled setting to assess generalization under limited data. The results are summarized below (left: same-different tasks/right: spatial related tasks). DSRF achieves strong performance across various SVRT tasks, even when trained with limited samples such as 500 samples per task. The results demonstrate its ability to learn abstract visual rules efficiently.

Table C. Sample efficiency analysis on the SVRT dataset.

We will add this sample efficiency analysis into the in the final manuscript and the supplementary materials accompanying the publication.

[c] He, Kaiming, et al. "Deep residual learning for image recognition." CVPR, 2016.

[d] Vaishnav, Mohit, et al. "Understanding the computational demands underlying visual reasoning." Neural Computation 34.5 (2022): 1075-1099.

Q2: Standard scoring mechanisms**?**

Yes, our model uses the standard RPM scoring mechanism, consistent with PredRNet [13] and many other models [12, 17]. We apply binary cross-entropy loss over the scores of the eight answer choices, which is functionally equivalent to Softmax with cross-entropy loss under the condition of single-choice questions.

Q3: Importance of multi-view reasoning pyramid

Multi-view reasoning pyramid serves as the foundational architecture of our framework, enabling its scalable design and robust performance. By utilizing different number of views, we design three pyramids with small, medium and large configurations, with a pyramid structure of “2-1”, “4-2-1”, and “8-4-1”, where each number represents the number of views in each layer. Combined with the dynamic reasoning capability enabled by DDCP and its core module, GARB, our DSRF demonstrates strong scalability. Notably, larger model variants do not overfit the training data but instead achieve consistent performance improvements across diverse benchmarks, as evidenced by the progressive improvements from DSRF-S to DSRF-M and, and from DSRF-M to DSRF-L in Tables 1–3 of the manuscript. The reviewer may refer to the reply to W2/Q2 of Reviewer Yeks for a more comprehensive comparison of DSRF variants across datasets. Due to space limit, we omit here.

Q4. Ablation of different mixings in MRM and more

Thank you for your thoughtful suggestions. We respond to your comments as follows.

• Ablation of different mixings in MRM

Following your comments, we have added the ablation of different mixings in MRM, a 3-layers MLP (MLP), two channel-wise 2D convolution layers (CNN), and a self-attention block [e] on O-RVN, I-RVN and RVN-F datasets, as summarized below.

Table D. Ablation of different mixings in MRM.

The results show that alternative mixings exhibit inferior accuracy compared to the MRM, confirming that our multi-granularity mixing architecture more effectively captures diverse rule interactions. While the improvement is relatively modest compared to other novel architectural components like MVRP and DDCP, the gains persist across dataset while maintaining practical relevance.

[e] Vaswani, Ashish, et al. "Attention is all you need." NeurIPS, 2017.

• Additional information on mixings in MRM

MRM enhances rule composition by performing feature mixing across multiple levels of granularity, including channel-wise, token-wise, row-wise, and sample-wise projections.

• Specifically, the channel-wise mixing is implemented through a linear projection that is applied along the channel dimension, capturing interactions across feature channels and enabling the model to refine semantic dimensions associated with specific attributes.

• For the token-wise mixing, each spatial token is projected using a shared MLP, modelling intra-view spatial relationships and allowing the network to process localized rule patterns within the view.

• For the sample-wise mixing, all tokens within a view are aggregated into a global representation and passed through a linear projection layer, providing global context for each sample and supporting top-down rule verification.

• For the row-wise mixing, features from different matrix rows within a sample are aligned through a linear projection, capturing structural relations such as progression or repetition along the row axis that commonly exist in RPM problems.

These mixing operations capture interactions across different dimensions of the reasoning space, enabling the model to refine and combine partial rule patterns extracted from individual views.

We will add these results and discussions in the final manuscript and the supplementary materials accompanying the publication.

We trust this rebuttal has comprehensively addressed your concerns. Should any additional questions arise, we would be pleased to provide further clarification.

We hope this message finds you well. As the discussion period for our manuscript (ID: 3366, DSRF: A Dynamic and Scalable Reasoning Framework for Solving RPMs) is drawing to a close, we kindly inquire whether you have any further feedback following your review of our rebuttal. Your insights remain invaluable to us in refining this work, and we deeply appreciate the time and expertise you have contributed to its review process.