Mind the GAP: Glimpse-based Active Perception improves generalization and sample efficiency of visual reasoning

We thank the Reviewer for constructive comments. Below we have prepared detailed responses to the points raised in the Weaknesses and Questions sections.

Weakness: For the discussion of generalization ability, the paper focuses on the OOD data of the specific benchmarks. However, what we hope to achieve is the ability that models can deal with totally different tasks like humans, although it is really though. This paper lacks the analysis of it

&

Question: What concerns me most is the generalization ability of this approach, although it is inspiring and fancy. Could you give me some clues that this approach can deal with tasks across different domains to reach the authentic reasoning ability?

Inspired by cognitive theories such as (Summerfield et al., 2020), our approach assumes the key role of action in information processing. According to those theories, actions not only allow to gather information about task-relevant parts of the environment, but they also provide information about relations between those parts. In other words, actions allow to “connect the dots” where the “dots” are the single pieces of information about the environment obtained after taking each action. We believe that actions have two important properties that make them important for generalization across different sensory domains.
Firstly, actions tend to have low-dimensional representations, be that information about eye movements to process the visual information or information about finger movements for processing tactile information. Secondly, information about actions has no overlap with its sensory counterpart meaning that the same set of actions can describe the same spatial or abstract relation between completely different parts of the sensory environment. Hence, given the fact that actions control not only our visual sensors but also sensors of other senses (e.g. haptic, auditory), these two properties suggest that actions underpin reasoning capabilities in various domains.

References

• Summerfield, C., Luyckx, F., & Sheahan, H. (2020). Structure learning and the posterior parietal cortex. Progress in Neurobiology.

Yes, I understand your motivation. Starting from cognitive theories, this glimpse-based approach has theoretical potential for generalization. However, my concern is whether this method demonstrates generalization capabilities in experimental applications across other domains. Considering time constraints, I'm not requiring complete experimental results. Combining VLM's knowledge and this method should theoretically enable cross-domain generalization. I'm curious if there are any ways to integrate this mechanism into VLM. I suppose this would make this interesting method more applicable in the current VLM era.

Question: I'm not sure about the term "active", since the architecture relies on a fixed (and not adaptive or recurrent) scheme to extract local features?

By the term “active” we mean that our model sequentially picks the most important image regions, thereby mimicking the human vision with active eye movements. Our scheme to select those regions can be viewed as adaptive to some extent. Firstly, it does not attend to some pre-defined set of image locations. Instead, those locations depend on the saliency extraction process. Secondly, the selection of the next glimpse location depends on the previously selected locations that are suppressed by the inhibition-of-return (IoR) mechanism. Still, further extensions are possible, such as modulating the choice of locations via top-down connections from deeper layers of the network. This improvement would also allow to incorporate various additional information making the glimpsing behavior potentially more goal-driven.

Weakness: Given the extensive prior work on foveated vision, how do existing methods perform on these tasks?

&

Question: Some additional related work: "Learning to combine foveal glimpses with a third-order Boltzmann machine", Larochelle et al. 2010 "Multiple Object Recognition with Visual Attention", Ba et al. 2014 "Show, Attend and Tell: Neural Image Caption Generation with Visual Attention", Xu et al. 2015

We thank the Reviewer for hints on the relevant prior work. We included the discussion of this work in the revised “Related work” section of our manuscript as follows:

The approach from (Larochelle & Hinton, 2010) and its ANN-based extension (Mnih et al., 2014) use reinforcement learning (RL) to train a glimpsing policy to determine the next glimpse location either in a continuous 2D space or in a discrete grid-based space of all possible glimpse locations. While this approach was evaluated mainly on simple image classification and object detection tasks, Ba et al. (2014); Xu et al. (2015) extended it to more complex images and image captioning tasks. However, having been evaluated on several visual reasoning tasks by Vaishnav & Serre (2023), the RL-based approaches could not achieve reasonable performance. This is likely caused by learning inefficiency in exploring the large space of all possible glimpse locations.

Weakness: Training sets for these tasks are also extremely small and will thereby favor architectures that are based on hardcoded (not trained) feature extraction

This may not be always true, especially if OOD generalization is evaluated. For example, the state-of-the-art baseline of Slot-Abstractor relies on hardcoded feature extraction stage that was pre-trained on larger datasets. As we show in our results, this model struggles in many OOD settings, see e.g. Table 2 and Figure 7.

Weakness: One would expect any architecture that makes those positions explicit to do exceptionally well on these specific tasks

The spatial information, as the Reviewer points out, is indeed very important. But as shown in our experiments with different downstream architectures (e.g. Table 1-4), it is not sufficient to simply make this information explicit. What our approach provides on top is a way to effectively combine the locally extracted features with their locations for reasoning about relations between different parts of the visual input.

We thank the Reviewer for constructive comments. Below we have prepared detailed responses to the points raised in the Weaknesses and Questions sections.

Weakness: The proposed architecture can be viewed as a local feature extractor that encodes local image regions and their positions. There is very extensive prior work in this area, which makes the novelty somewhat limited.

The motivation for our approach was to take inspiration from the human active vision that does not process the visual input entirely but rather salient/task-relevant parts of it. There is indeed extensive prior work in this area, and we provide a more extensive discussion on that in our response to the last Reviewer’s question. Most of the prior work focused on using reinforcement learning (RL) to train a glimpsing policy. Our approach, in contrast, uses the concept of saliency maps to determine where to look. Lastly, to the best of our knowledge, for the first time such bio-inspired vision approach is extensively evaluated in the context of relational reasoning, sample efficiency and generalization robustness, surpassing baselines operating using more conventional principles.

Question: It would be very beneficial to include a modern vision-language model in the evaluation, as these models have become very good at solving similar reasoning tasks.

Thank you for this suggestion. To address this aspect, we conducted an experiment described in the general comment to all Reviewers “Study on LLMs’ capabilities of solving same-different task”

Question: Is there a reason why the results in Table 3 (on a subset of ART) do not include all models considered previously (e.g. Attn-ResNet?)

For each table, we selected the best performing models reported in prior work. No prior work reported the performance for Attn-ResNet.

Question: Can the SVRT #1-OOD experiment be extended to other subsets than SVRT #1?

This would be an interesting experiment. However, the SVRT #1-OOD dataset introduced by (Puebla and Bowers, 2022) is currently available only for SVRT #1. While the code to generate SVRT #1-OOD is publicly available, it is not compatible with the code for the original SVRT. It is therefore not straightforward to extend the SVRT #1-OOD experiment to other SVRT tasks. However, we performed similar experiments using other more complex datasets such as ART and CLEVR-ART that also evaluate OOD generalization.

Question: Why is the dataset size for results in Table 1 restricted to 500/1000 training examples? This seems arbitrary. Figure 4 seems to suggest that the training set is larger?

The results reported in Table 1 and Figure 4 evaluate the sample efficiency, where the SVRT dataset is deliberately restricted – the smaller, the more difficult the task. The restriction to 500 and 1000 training examples in Table 1 was made to be comparable and consistent with the prior work (Mondal et al., 2024) that demonstrated state-of-the-art results. Still, we also felt that this choice appears arbitrary and, therefore, we provided more extensive results in subsequent Figure 4, including both larger and smaller dataset sizes than reported in the prior art.

[UPDATED]

Firstly, we would like to clarify that Figure 4 has been present in its current form since our initial submission. There was no update or new results related to Table 1 and Figure 4 included during the rebuttal period.

The purpose of Table 1 and Figure 4 is to show the results from evaluating sample efficiency. Although we agree that the sizes of 1000 and 500 samples appear to be arbitrary, they follow the common convention for the SVRT task and immediately enable comparisons with prior papers. To clarify how the table and the figure should be interpreted, we made the following revisions to the current draft:

• We modified the caption of Table 1: Test accuracy for SVRT evaluated for common dataset sizes of 1000 and 500 samples
• We modified the caption of Figure 4: Sample efficiency evaluation for SVRT beyond the common dataset sizes of 1000 and 500 samples

As for conclusions stemming from Table 1 and Figure 4, we would like to point out that:

• Sample efficiency, shown in Table 1 and Figure 4, is an important benefit of our method, which we emphasize in our manuscript.
• In most cases, the accuracy order of the models in Table 1 is retained for other dataset sizes shown in Figure 4. Of course, Table 1 should not be considered in isolation, thus we immediately placed Figure 4 after it.

We hope that our motivations are reasonable and provide an acceptable trade-off between maintaining comparability with the reporting in prior papers and still providing accurate insights. Shall this be insufficient, we are happy to incorporate further suggestions.

Question: What is the impact of using a hard versus a soft mask M(x_t)?

A soft mask was motivated by a hypothesis that in complex scenes (such as CLEVR-ART) it can be beneficial to make several glimpses close to each other. The hard mask does not allow for that, as it masks out the area around the glimpse location in the saliency map, thereby excluding any further glimpsing in this area. We tested the soft mask only on CLEVR-ART tasks and it resulted in accuracy difference within 1-2% for our GAP-Abstractor model, see the table below

Weakness: Why can GAP-Abstractor improve OOD generalization of same-different relation and more abstract relations?

To provide more insights on this aspect, we revised our “Discussion” section (Sec. 6):

Our results suggest that factorizing an image into its complementary “what” and “where” contents plays an essential role. The low-dimensional “where” content (glimpse locations) is crucial since it does not contain any visual information and, in turn, allows to learn abstract relations that are agnostic to specific visual details. To capitalize on that, it is important to process the “where” content explicitly, disentangling it from its “what” complement. We implemented this by using the recently introduced relational cross-attention mechanism (employed by the Abstractor downstream architecture) where the “what” content is used to compute the attention scores for the processing of the “where” content. In contrast, implicitly mixing the “what” and the “where” contents, as in the case of the Transformer downstream architecture, weakens the generalization capabilities. This can be seen by comparing the performance between GAP-Transformer and GAP-Abstractor. To further support this, we provide supplementary results in Appendix D.2 showing the importance of using both the “what” and the “where” contents instead of just one of them for task solving.

Another important aspect of our model is that it processes only the most salient image regions while ignoring the unimportant ones. The inferior performance of models where the downstream archi- tectures receive full information (i.e. all patches that constitute the image) suggests that supplying a superfluous amount of information may distract the models hindering the learning process. Additional results provided in Appendix D.3 elucidate this further. Specifically, we show that it is insufficient to naively reduce the amount of information provided to the downstream architecture, by discarding uninformative image parts. Instead, it has to be ensured that the supplied information is well structured. In the context of GAP, it means that the glimpse contents have to be centered around salient image regions. For example, the image patches of the glimpse contents should contain objects’ edges in their central parts rather than somewhere in the periphery.

Weakness: Given that the GAP mechanism involves multiple steps (e.g., saliency map generation, inhibition of return), have you compared its computational performance with other baseline models?

&

Question: Have you explored alternative methods for computing the saliency map, given that GAP's effectiveness largely depends on the map's accuracy in identifying key image regions?

To address this important aspect, we conducted additional experiments exploring different ways to compute the saliency maps. The description and results can be found in the general comment to all Reviewers: “Experiment with more realistic objects and alternative saliency maps”.

We thank the Reviewer for the constructive comments. Below we have prepared detailed responses to the points raised in the Weaknesses and Questions sections.

Weakness: The images in the four selected datasets seem relatively simple, with very clean backgrounds. Have you considered comparing your proposed model with baseline models on more realistic image datasets, such as the COCO dataset?

While images in the four selected datasets are simple, it has been shown by prior works, such as (Vaishnav et al., 2023, Mondal et al., 2024), that visual reasoning tasks with such images pose a significant challenge to AI models. Nevertheless, to get insights on the potential of our approach to deal with more complex images, we have conducted an experiment using more realistic objects. The description and results can be found in the general comment to all Reviewers: “Experiment with more realistic objects and alternative saliency maps”.

Thank you for the author's response. I would like to maintain my score. My primary concern is that the images in the datasets used, including the newly added Super-CLEVR dataset, lack sufficient realism. This makes me skeptical about the practical applicability of the proposed approach in real-world scenarios.

Weakness: References are missing in Tables and Figures.

&

Weakness: Explain what the tasks RMTS and ID are, at least briefly, in Section 5.4 before explaining the results.

We thank the Reviewer for the suggestions.

We attempted to introduce references in all tables and figures, but due to long reference formatting of ICLR, the material would not fit the page width and the maximum page count. The references to all baseline models are briefly summarized in Sec. 4 in the "Baselines" paragraph.

We provided a brief description of the RMTS and ID tasks at the beginning of Section 5.4 as follows:

We test our approach on two tasks from ART – relational-match-to-sample (RMTS) and identity rules (ID) – but with more realistic images using the CLEVR-ART dataset. In the RMTS task the model has to determine whether the objects in the top row are in the same ”same/different” relation as the objects in the bottom row. In the ID task the model has to determine whether bottom row contains objects that follow the same abstract pattern (ABA, ABB or AAA) as the objects in the top row. Figure B.4 provides examples for the two tasks

Question: Please explain why each sensor works better or worse for each downstream architecture-dataset pair.

&

Weakness: Authors do not make it explicit or clear that different sensors (multi-scale or log-polar) have been used across different datasets for the same downstream architectures (ViT/Abstractor). Ie. For SVRT, GAP-Abstractor used the multi-scale sensor but for SVRT #1-OOD it used the log-polar sensor. This should be made clear in the main paper, instead of referring to the Appendix

We clarified the usage of different sensors in the main text at the beginning of the “Results” section (Sec. 5) as follows:

[…] One important hyper-parameter is the type of glimpse sensor (multi-scale or log-polar) that we select for each of the four considered datasets. We report performance for different glimpse sensors in Appendix D.

In addition, we highlighted the best-performing sensors in tables Table D.1-D.4 and revised the Appendix D to comment on the performance differences for each downstream architecture-dataset pair in Sec. D.1 as follows:

We evaluated two different glimpse sensors for each of the four datasets considered in this work, see Table D.1- D.4 and Figure D.6. For the Abstractor-based downstream architecture, we observe com- parable performance for both sensors in most cases. The most significant performance difference is observed for SVRT #1-OOD and two tasks from the ART dataset (Table D.2-D.3) where the log-polar sensor achieves around 5% better accuracy. For the Transformer-based downstream architecture, the per- formance difference between two glimpse sensors averaged over all tasks is around 5%. However, there are tasks where the multi-scale sensor results in around 10% worse accuracy, see Table D.1, and 10% better accuracy, see Table D.3. Overall, based on the performance data, there is no clear quantitative trend favoring either of the two glimpse sensors. We ascribe this to the qualitative ad- vantages and disadvantages of both sensors. In particular, the log-polar sensor benefits from the properties of the log-polar space where the rotation and scaling in the cartesian space become trans- lations, see e.g. (Javier Traver & Bernardino, 2010). However, the warping into log-polar space may make it difficult to capture information about image parts that are far from the glimpse locations. The multi-scale sensor, in contrast, captures the distant information more faithfully using downsam- pled image patches of different sizes. The disadvantage of this, however, is that the resulting glimpse content contains additional dimensions for the different scales.

Question: Since the authors explained that "glimpsing behavior can be distracted by spurious salient locations such as edges of shades or regions of high reflection", would performing GAP (finding salient regions) on feature maps, instead of images, where these spurious features are probably less influential, increase performance? Can authors provide results?

We thank the reviewer for the suggestion of improving the glimpsing behavior by applying GAP on top of the feature maps instead of the images directly. To address this question, we conducted additional experiments described in the general comment “Experiment with more realistic objects and alternative saliency maps”

We thank the Reviewer for the constructive comments. Below we have prepared detailed responses to the points raised in the Weaknesses and Questions sections.

Question: Can the authors provide an explanation to why GAP-Abstractor perform worse on 'straight lines' and 'rectangles' compared to other classes?

The reason that GAP-Abstractor performs worse on ‘straight_lines’ and ‘rectangles’ is that the dataset contains many images where the straight lines and rectangles are very similar to each other with only slight differences in size. We included Figure B.3 into the revised Appendix that shows a few examples with different shapes where even humans may have difficulties to tell the difference.

Weakness: Synthetic data: Three out of the four datasets used for evaluation are based on synthetic datasets. It would be beneficial to include more real world datasets such as GQA or Super-Clevr.

&

Question: The paper should better motivate the choice of evaluation datasets.

It has been shown by prior works, such as (Vaishnav et al., 2023, Mondal et al., 2024), that visual reasoning tasks with simplistic images can pose significant challenges to AI models. Moreover, as we show in the general comment “Study on LLMs’ capabilities of solving same-different task”, even VLMs appear to encounter difficulties. Therefore, solving these visual reasoning tasks presents an important research step forward. We clarified this motivation in the revised manuscript in Sec. 4 in the paragraph on Datasets with the following revision:

[…] Finally, we test the model’s potential to scale to more complex images by testing it on the CLEVR-ART dataset (Webb et al., 2024a), see Figure 3(d). It contains two tasks from ART but with more realistic images. Each dataset is described in more detail in Appendix B. While all tasks described above consist of simple images, they are still challenging, as illustrated by the limited accuracy of the state-of-the-art models described below.

Despite the importance of the currently selected datasets, we agree with the Reviewer that it is important to explore more realistic images, and we thank the Reviewer for the suggestion on the datasets. Although at this point our model does not include an NLP module, we have conducted an experiment using the more realistic objects from the suggested Super-CLEVR dataset. The description and results can be found in the general comment to all Reviewers: “Experiment with more realistic objects and alternative saliency maps”.

Weakness: Current state of the art vision-language models such as LLaVA (NeurIPS 2024) keep the visual features from the target image in the context window. This means that they can actively attend to the image as many times are necessary to extract visual features. It would be interesting to compare the proposed approach to current SOTA VLMs such as LLaVA.

We thank the Reviewer for the suggestion on the VLM comparison. We evaluated one of the current SOTA VLMs, Pixtral (Agrawal et al., 2024), on the same-different task and provided the results in the general comment to all reviewers: “Study on LLMs’ capabilities of solving same-different task”

References

• Agrawal, Pravesh, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa et al. "Pixtral 12B." arXiv preprint arXiv:2410.07073 (2024).

We thank the Reviewer for the constructive comments. Below we have prepared detailed responses to the points raised in the Weaknesses and Questions sections.

Weakness: While the proposed approach is somewhat novel it is similar to prior work such as "AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale, ECCV 2024" which also focus on what and where to look.

&

Weakness: Prior work such as "Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024" used surrogate tasks to decide what and where to look at. It would be beneficial to include a discussion of this related work.

&

Question: The paper should include a broader discussion of related work (see above).

We thank the Reviewer for the suggestions on related papers. We included them into Section 3, “Related Work”, with the following revisions:

The approach from (Larochelle & Hinton, 2010) and its ANN-based extension (Mnih et al., 2014) use reinforcement learning (RL) to train a glimpsing policy to determine the next glimpse location either in a continuous 2D space or in a discrete grid-based space of all possible glimpse locations. While this approach was evaluated mainly on simple image classification and object detection tasks, Ba et al. (2014); Xu et al. (2015) extended it to more complex images and image captioning tasks. However, having been evaluated on several visual reasoning tasks by Vaishnav & Serre (2023), the RL-based approaches could not achieve reasonable performance. This is likely caused by learning inefficiency in exploring the large space of all possible glimpse locations. Nevertheless, the RL-based approaches that use more efficient RL techniques such as (Pardyl et al., 2025) to learn complex glimpsing policies are relevant to our work as they can be integrated into our model to enhance its capabilities of dealing with real-world images. Our approach, in contrast, leverages the concept of saliency maps to determine the next glimpse location which significantly reduces the space of glimpse locations to the most salient ones. […] Interestingly, in the domain of visually extended large language models (LLMs), Bhattacharyya et al. (2024) showed that forcing these models via surrogate tasks to collect relevant pieces of information before solving the actual task improves the final performance. While using a different, compared to ours, approach of surrogate tasks to achieve that, this work points to a potential of integrating our conceptual approach into LLMs.