Contour Integration Underlies Human-Like Vision

Thank you for your review. We are glad you find our experiments justified and carefully conducted, and that you found the findings interesting. We respond to each of your comments below point-by-point.

It is not clear whether there are actionable insights for improving models based on our results

We believe there are quite a number of actionable insights for improving models based on our results:

First, many attempts in the literature for reproducing alignment to low-level human visual cortex have focused on architectural changes in lieu of other approaches (such as data-based approaches), see e.g. Kubilius, Schrimpf et al. (2019). We show that these approaches do not pay off when trained and evaluated at scale. This gives a direct actionable insight: for modelers seeking to improve models of basic human vision, a more fruitful approach is to focus on making the training diet of the models more human-like (especially in scale) than to improve the model architecture in a specific way.

Second, we demonstrate success in training models for contour integration directly. While this in itself is the less surprising insight, it certainly is valuable for those who simply want models that exhibit human-like contour integration for further experiments. Interestingly, we also find that training for contour integration in models also leads to shape bias.

Third, we find that models with a more human-like integration bias exhibit improved accuracy on a downstream object classification task (Figure 6b, Figure 8) – indicating that selecting models for their ability to integrate contextual information is a useful validation signal when building artificial neural networks.

Taken together, our work substantially advances our understanding of contour integration in vision science, and provides a clear path models that do exhibit human-like contour integration: large training diets, or direct training. This finding contrasts previous work that focus on the architectural role of horizontal connectivity in human visual cortex and models as the mechanistic source of contour integration (e.g. Linsley et al, 2018).

“Very large models nonetheless approach human performance, although they do not show a bias for directional fragments that is shown in humans” [...] “ [the paper demonstrates] that while large models actually approach human performance even in this new task there is still an interesting bias in human vision that differentiates it from the models”

We would like to clarify that the best models do indeed show a human-like bias (Figure 6b), demonstrating that large training diets can yield human-like contour integration behavior in models. Integration bias separates almost all models from humans due to their integration bias that is not human-like. This shows that object recognition itself is possible without contour integration (a surprising finding in itself: e.g. Field 1993 as pointed out by reviewer eQjn; Kovacs & Julesz 1993; Grossberg & Mingolla 1985), but the largest models learn to do contour integration nonetheless.

Thanks again for your review. We believe we have addressed your comments and ask you to consider raising your score?

References from all rebuttals

Cavanaugh, J. R., Bair, W., & Movshon, J. A. (2002). Nature and interaction of signals from the receptive field center and surround in macaque V1 neurons. Journal of Neurophysiology, 88(5), 2530–2546. https://doi.org/10.1152/jn.00692.2001

Zamir, A. R., Sax, A., Shen, W. B., Guibas, L. J., Malik, J., & Savarese, S. (2018). Taskonomy: Disentangling task transfer learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Gatis, D. (n.d.). rembg [Computer software]. GitHub. Retrieved March 31, 2025, from https://github.com/danielgatis/rembg​

Linsley, D., Kim, J., Veerabadran, V., Windolf, C., and Serre, T. Learning long-range spatial dependencies with horizontal gated recurrent units. NeurIPS, volume 31. Curran Associates, Inc., 2018.

Grossberg, S., Mingolla, E. Neural dynamics of perceptual grouping: Textures, boundaries, and emergent segmentations. Perception & Psychophysics 38, 141–171 (1985). https://doi.org/10.3758/BF03198851

Kubilius, J., Schrimpf, M., Kar, K., Rajalingham, R., Hong, H., Majaj, N., ... & Dicarlo, J. (2019). Brain-like object recognition with high-performing shallow recurrent ANNs. NeurIPS (pp. 12785-12796).

Field DJ, Hayes A, Hess RF. Contour integration by the human visual system: evidence for a local "association field". Vision Res. 1993 Jan;33(2):173-93. doi: 10.1016/0042-6989(93)90156-q. PMID: 8447091.

Marques, T., Schrimpf, M., & DiCarlo, J. J. (2021). Multi-scale hierarchical neural network models that bridge from single neurons in the primate primary visual cortex to object recognition behavior. bioRxiv. https://doi.org/10.1101/2021.03.01.433495

We thank the reviewer for their review. We have addressed your comments individually below and point to an external link for additional figures: https://drive.google.com/drive/folders/1M_nUONfTXLmUZCHHL0PfIlIaEhpu0vLo?usp=drive_link

Long-range interactions in humans being normally associated to contour integration, and how that relates to our task.

In our paper, we have intentionally omitted discussion around long-range interactions as our task was not designed to investigate these effects directly:

• Our stimuli spanned only an 8x8 degree window of visual angle.

• Our stimulus presentation time was only 200ms, which was followed by a 1/f noise mask. This ensured that no long-lasting integration or tracing is possible.

The goal of these steps was to ensure that the types of long-range connectivity you refer to are controlled in this setting - see also below for our analysis of the human data.

The difference in human perception between two sets of stimuli should be analyzed. In my opinion, perception of directionless outlines should have a strong relationship with proximity. However, this is not the case for the directional version.

Thank you for this suggestion, which we have now done. In summary, we found little difference in the scaling of the two conditions (phosphenes vs segments) in humans. The difference in performance in these two cases is mostly a shift of intercept instead of a change in slope across the number of elements (figure human_accuracy_split.png), though note that while the effect of difference in slopes is small, it is still technically significant (t=2.87, p<0.01). This means that the directionality of the segments only slightly affects performance positively over the directionless prosphenes as element density increases.

Comparison between CNN and transformer architectures, since CNNs do not have long-range interactions while transformers do

Thank you for this suggestion. We indeed included transformers in our original work (the single largest architecture family in our paper is the vision transformer (ViT), of which we had 257 decoder-fit variants; Table 1, row 1).

We have now conducted several new analyses comparing transformers and CNNs. To make results comparable, we restricted the analysis to only models trained on ImageNet-1k, although we also provide figures for all models in the figures too.

• We found that the overall performance is no different (CNNs: 32.49% on average, ViTs: 31.24% on average, t-test of means p=0.1108: not significant). This is also true even when not controlling for dataset (CNNs: 29.08% on average, ViTs: 29.98% on average, t-test of means p=0.1889: not significant); figure vit_vs_cnn_imagenet.png

• Integration biases between transformers and CNNs are comparable: ViT (7.8%) and CNN (10.1%) are not statistically significantly different (t=2.45, p=0.016) at the 0.01 criterion; figure vit_vs_cnn_imagenet.png

• The way model performance scales across the number of elements is the same across models and across conditions: only one condition across all conditions is different between CNNs and ViTs (the 16-phosphenes condition), while the differences between the rest are statistically non-significant. vit_vs_cnn_imagenet_scaling.png

Taken together, these analyses show that long-range interactions do not play a crucial role in our task either in humans nor in models.

Receptive field sizes and how they relate to task performance should be studied

We thank the reviewer for this nice suggestion. In addition to the experiments where we compared ViTs to CNNs (effectively two extremes of this spectrum), we have now added an explicit test for receptive field size and its relationship to the primate visual system. We evaluated the same subset of models as in Fig 6d and 8 on two Brain-Score benchmarks that test for the similarity of the effective receptive field size of a model to a primate counterpart. These are the Grating summation field (GSF) and surround diameter (Marques 2020, Cavanaugh 2002) benchmarks measured in macaque V1. In short, we do not find a statistically significant relationship between either measure of receptive field size similarity and fragmented object recognition accuracy (GSF r=0.2655, p=0.0653; surround diameter r=0.276, p=0.055); figure receptive_field.png

We believe these additional results further bolster our conclusion that architecture plays a minimal role in contour integration, and that the receptive field size of models is not a crucial component in this study.

We also thank you for pointing us to missing references, which we have included.

Thank you again for your review. We believe we have addressed all of your concerns and would be grateful if you considered raising your score.

Thanks for your review. We are happy to hear that you found our experiments meaningful, and the paper well-written and very interesting. We reply point-by-point to your comments below and point to an external link for additional figures: https://drive.google.com/drive/folders/1M_nUONfTXLmUZCHHL0PfIlIaEhpu0vLo?usp=drive_link

Contour extraction is not explained in the paper

We apologize for not including a detailed description of this step in our paper. We use a phosphene rendering algorithm (Rotermund et al., 2023, as cited in the paper) for extracting contours and rendering stimuli with phosphenes or segments. The contour extraction is simple: images are transformed from RGB to grayscale, and then convolved with a Gabor filter bank of 8 different orientations. Specifically, our contour image is defined by:

see contour_equation.png

For the placement of phosphenes and segments, we place elements on the contours of the object preferentially depending on the strength of the contour and its directionality. We use this algorithm both for our experimental stimuli, as well as ImageNet-1k images. For ImageNet-1k images, we also perform a background removal using rembg (Gatis) before applying this algorithm. We will include this description in the updated manuscript.

The problem of obtaining contours, which here is considered resolved and easy is, in fact, still an open problem even in the case of supervised models

How exactly we obtain contours from RGB images is not central to our argument in any way. This is for two reasons:

1. all models and humans see the same images, regardless of how the contour was extracted.

2. The contour condition is merely a control condition meant to show that the large drop in model performance is not merely due to superficial changes in data distribution (i.e., most of the image being black).

The contour extraction merely serves as an image preprocessing step that we can use to render images which are shown to humans and models to compare their performance and behavioral characteristics.

The paper is well-written and very interesting. However, I found that attempting to train models directly to group elements is a bit naive.

The goal of this experiment was to causally show the primacy of training data in achieving a human-like contour bias, which we successfully demonstrate with the improved performance of the resulting models. Despite the lack of horizontal connectivity or other architectural biases, we were able to train a model simply using segments and phosphenes to exhibit a human-like integration bias. This also resulted in high shape bias, exceeding previous shape biases from other similar direct training approaches (Geirhos et al., 2018). While the approach itself is simple, it serves to strengthen our claim about the importance of dataset.

There are attempts like this in the literature, and obtaining binary contours using Gaussian filters and Otsu threshold is not quite skillful.

We are not exactly sure what is meant by it not being “quite skillful”. The motivation here was to simply turn our non-binary contours extracted using Gaussian filters into binary contours, and for this purpose the methodology works very well.

Perhaps the only theoretical claim concerns the demonstration that contour integration is not necessarily a product of horizontal connectivity in the primary visual cortex (referring to the literature) and that the mechanism is learned from the amount of data inducing learning. I am not sure that the tests effectively support these conclusions.

This is a very interesting thought – which experiments do you have in mind that would better support these conclusions? We believe our evidence is strong. We show that:

• human-like contour integration emerges in models trained on large datasets despite any explicit architectural mechanism being present (Figs. 6b, 7). This provides proof of existence that such a mechanism is not strictly necessary.
• We causally show that human-like contour integration is possible to directly train for without the use of human behavioral response data (Figure 7). This is causal evidence in a controlled setting that such a mechanism is not necessary.

Taken together, we believe these facts strongly support the conclusion that horizontal connectivity is not necessary for contour integration. Of course it is still possible that the human visual system implements contour integration using a mechanism of horizontal connectivity. Our results challenge the prevailing view that this is the only way to implement contour integration, or that the existence of this mechanism is the key reason for why contour integration exists - clearly, it emerges in other settings too.

We thank you again for your review, and believe that we have addressed your comments. In light of this, we hope that you consider raising your score.

Thank you for your favorable review. We are happy to read that you found the paper insightful and our experiments sound, and that you see how it helps guide future research. We address your comments point by point below, and point to an external link for additional figures: https://drive.google.com/drive/folders/1M_nUONfTXLmUZCHHL0PfIlIaEhpu0vLo?usp=drive_link

Concerns about training data, which could be remedied by testing models on the task with e.g. a different background that is more unlikely to be in training datasets

Thank you for the nice suggestion. We followed your proposal and made a new version of our dataset, this time with a red background instead of the standard black background (image examples in the folder red_images/). We tested the subset of pre-trained models also in figures 6d and figure 8. We did a t-test to test for the difference in the group performance of the fragmented conditions and found that there was no difference in model accuracy under these two conditions (t=0.323, p=0.749). This shows that the specific model training distribution is not the reason for model performance here, and that the results are robust against a change of background. See figure normal_vs_red.png.

Concerns about the analysis regarding how important architecture is and how robust the analysis is

We have taken two additional steps to remedy this: We now also run a mixed effects regression, which allows for the architecture type to be treated as a random variable. This analysis also finds that training dataset size is most important (z=14.66, p<0.001), while model compute is less important but significant (z=7.177, p<0.001). Importantly, the random effect for architecture type under this more controlled analysis is dominated by error rather than systematic variance (variance=0.002, standard error = 0.008), suggesting that the random factor of architecture family plays little systematic role in the mixed effects model. We analyzed ViT vs CNN-based architectures in more detail for a fixed training dataset size (ImageNet-1k) -- due to space constraints, we detail this analysis in our response to reviewer eQjn. These further analyses provide additional evidence to our claim that architecture is less important than training dataset size.

Different downstream tasks, such as segmentation

We thank the reviewer for this idea. We analyzed additional models from the taskonomy (Zamir et al. 2018) model set, which provides a great degree of breadth in terms of tasks, with the downside that we only have one model per task. Nevertheless, we report these additional results here (Figure taskonomy_scores.png).

In summary:

• The base object classification and scene classification models reach 31% & 32% accuracy respectively
• Unsupervised segmentation reaches 29%%, surface normal estimation 33%, and depth estimation reach 34% accuracy
• Interestingly, the edge-computing model reaches only 21% accuracy

In general, it seems that while the task itself appears to play a role, object recognition stands out as a good objective, with more specialized objectives often falling slightly short. Nevertheless, the differences are rather small.

Novelty of the work: “Specifically, the authors highlight how integration bias—a phenomenon extensively studied in models by Geirhos et al.—can arise from large-scale training [...]”

We would like to point out that Geirhos et al. (2018) did not study integration bias, but rather shape bias, which is a different metric measured using an entirely different dataset. Specifically, Geirhos et al. (2018) studied the human and model responses to images with a conflicting texture and shape cue. This gives a point estimate of the amount of shape bias a model has.

Our work does not study shape bias, but rather contour integration. Shape bias is a human preference to detect object categories by their shape, rather than their texture, while contour integration is the ability to integrate elements despite their discontinuity in image space. As such, while contour integration allows the recognition of images that would otherwise be unrecognisable, shape bias is a description of choice preferences. Our experimental setup is also different. Instead of several different conditions that are not related to each other on a single continuous scale (like in Geirhos et al.), we varied the image fragmentation. This allowed us to understand where and why neural networks fail (at the fragmentation of the contour), and to pinpoint this specifically to contour integration, a well-established “algorithm” in humans. Taken together, our work is very different from previous work, with a novel experiment, metric, and findings.

We thank you again for your review. We believe these comments address your concerns, but please let us know if additional analyses would be helpful. If all concerns are addressed, we ask the reviewer to consider increasing their score.