We thank reviewer eW1M for their thoughtful comments and questions. Please see our response to your concerns as follows:

"the dataset itself does not seem to offer substantial value to the research community"

We agree that code would be useful for further experimentation and plan to release it alongside the dataset. That said, we believe a dataset is also a valuable contribution because the peripheral vision model we would like to make available to the community takes ~5-6 hours per image to render. The entire dataset took over $6$ months to create. The procedure also depends on choosing a fixation point beforehand. To render multiple fixations at large scale during training is thus intractable. Furthermore, by providing a dataset, we are standardizing evaluation for future work. If we share only code, everyone will generate slightly different versions of the dataset, making it difficult to evaluate progress. We have made these points more clear in the introduction.

"If we interpret peripheral image transformation via TTM as a form of image perturbation, results are unsurprising"

We appreciate the reviewer eW1M’s insight about domain shift, but as noted by other reviewers, our results are interesting in the broader context of human vision. When humans view TTM images, their performance is highly correlated with original images -- despite never having “trained” on this exact image perturbation. If a DNN learned the same representational properties as human vision during training, then arguably they too should be able to handle this transformation. Yet, we observe that DNNs do not perform well under this transformation. As an experiment, we explore simply fine-tuning and training models on COCO-Periph to understand this behavior. We find that these models still exhibit drops in performance greater than humans do, suggesting the behavior is not just about domain shift. In the future, we ideally will have new models that do not require extensive fine-tuning -- just as humans do not require this. With our dataset and human experiment, our long-term goal is to build new methods that do perform like humans. We see COCO-Periph as a stepping-stone toward that goal, informing how to build representations robust to a peripheral transformation. We have added some of these points in the Discussion.

"The paper fails to provide a clear explanation for why the proposed model, even after fine-tuning on the entire dataset, still performs significantly worse than DINO" "Figure 5, which assesses the model's resilience, is presented misleadingly. It only demonstrates the model's performance across various eccentricities, which is expected, given that the model was trained on the same data ..."

First, we would like to emphasize that our main contribution is a dataset and human experiment, not a new proposed model. We include a fine-tuning experiment as an additional evaluation, but our larger goal is to uncover methods that do not require extensive optimization on TTM to perform well on this OOD transformation like humans do.

We appreciate reviewer eW1M’s request for per eccentricity psychophysics performance in Figure 5. We agree that this is useful information, and we have added a subplot of these values in Figure 5. We originally chose to report an across eccentricity value ($\mu$) because it is standard practice in psychophysics studies to fit a curve across stimulus conditions and report a critical value where performance deteriorates to 75% accuracy (Strasburger, 2020, Figure 12), and we aimed to make an apples to apples comparison between humans and models.

Regarding the results themselves, we agree it is puzzling that the fine-tuned model out-performs DINO when it under-performs on AP. We believe this can be explained by the nature of the TTM transform and psychophysics task we propose. By design, TTM perturbs the true object location -- especially for far eccentricities. When we report AP, we do not account for this and rely on the original COCO bounding boxes. We acknowledge that this is a limitation, and so we do account for it in the psychophysics task by padding the ground truth bounding boxes (see Section 6 and Appendix A.4).

In addition, the psychophysics task itself only accounts for object localization and does not account for recognition performance like AP does. To test if this played a role in our results, we evaluated recognition alone for the RCNN in Figure 6b. Here, we see that the fine-tuning has a less noticeable improvement in performance; training from scratch even reduces performance. This indicates that object localization, rather than recognition, drives the performance boost we see in Figure 5. Indeed, peripheral vision is thought to primarily play a role in guiding where to look next rather than in recognizing object identity. For this reason, we evaluate models against humans on this metric and expect to see more improvements in this area when mimicking peripheral vision.

In addition to our earlier remarks, we now include a control human experiment where we measure detection performance directly on uniform TTM images (See Figure 4, and Appendix A.4). Our results show that performance on TTM images is similar to peripherally viewed images. These results validate that TTM adequately models the information loss in the human periphery on our specific psychophysics task. These results demonstrate that TTM is an interesting transformation to explore in DNNs as a validated model of peripheral vision. In addition, we emphasize that humans are able to perform the detection tasks on the transformed images, so it is worthwhile to evaluate DNNs in the same way. We appreciate your consideration.

We are grateful for reviewer xwTZ for their careful review of our paper, appreciation of our contributions, and suggestions for improvements that will strengthen our work.

We thank the reviewer for pointing us to the additional references (Zhang and Han, 2019, and Han 2020), and have added these citations to the manuscript. We appreciate the suggestion to evaluate EEN which captures critical aspects of foveation and visual crowding. We choose to not evaluate this model and many other notable human-inspired DNNs such as VOneNet because they have not been built or trained to perform MS-COCO style object detection. In order to keep our analyses fair across models, we limited our work to models that met this criteria. We hope that our paper will inspire more work on human-inspired models for tasks like object detection so this becomes possible in the future.

"I would like to request further elaboration on the decision not to present the TTM-generated images directly to human participants."

We agree that a strong comparison could have been to measure human performance directly on TTM images. We chose to prioritize original images in order to measure true peripheral performance. We believe this is more useful than TTM performance alone because our overall goal is to model peripheral vision. Although TTM is a strong model, we hope our work will enable better models of peripheral vision in the future. With original image performance, our psychophysics procedures can scale to new implementations of peripheral processing.

That said, we recognize that our choice could have caused discrepancies in our comparisons. As additional controls, we included DNN psychophysics results for uniform, original, and pseudofoveated TTM (See AppendixFigures 14 & 15) to demonstrate that our results are robust to different variants of the transformation.

"it's crucial to note that the stimuli utilized in their study differ in certain aspects from those in previous research"

We thank reviewer xwTZ for bringing up important limitations to consider in our work. On the concern about grayscale vs color, we acknowledge that color has not been widely tested in TTM. However, we chose to evaluate color images because the DNN models we tested were trained on color images. We did not want to disadvantage the models in that regard.

We also agree that the synthesis process produces artifacts at times. We try to limit the impact of this by showing DNNs multiple seeds of the same image per eccentricity. We also argue that, despite this, TTM is still able to do a good job predicting performance and similar models pass metamer evaluations (Freeman & Simoncelli 2011). That said, we agree these limitations are important to consider, and we have added a note on limitations of TTM in the Discussion.

"I am eager to learn about the authors' insights and opinions regarding the fundamental mechanisms absent in CNNs but present in humans, which might account for these observed disparities"

We thank reviewer xwTZ for their thoughtful question. TTM suggests that humans rely on a compressed set of summary statistics to represent much of the visual field. We propose that the problem of aligning DNNs with human perception is in some sense about learning those same statistics or features. Data is a very flexible way to achieve this, but it has limits as you noted. While the structure of the convolution operation has ties to human visual system function, the convolutional architecture of CNNs is distinct from the complex connectivity seen in the human visual cortex. These differences may also limit the ability of CNNs to perform on-par with their human counterparts. A next approach could be to train directly on TTM statistics themselves. Even better would be to optimize DNNs to learn a reduced set of features instead of over-parameterizing visual input as they currently do.

To bridge the gap in object detection specifically, our work suggests that DNNs are trained to favor exact object location and size too much compared to humans. Relaxing this constraint could be key to aligning behavior.

More broadly, we believe that task formulation is a critical area to explore in DNNs. TTM as a model suggests that one general representation can explain behavior on a variety of visual tasks. We believe an important future direction in bridging the gap between humans and DNNs is to optimize for generalization across a variety of tasks -- rather than maximizing for accuracy on a single task. Current benchmarks in computer vision do not encourage this, and we hope that our dataset and experiments can facilitate research in this direction.We have updated the Discussion to include some of these points.

Question #1: "Is there something I have overlooked that rules out the alternative explanation for the results proposed above?"

Please see our response above. In short, we agree that this alternative explanation could impose a ceiling on the machine results as compared to human, but due to the evidence listed this explanation cannot account for the entirety of the results. We appreciate the reviewer offering this insight, and have included this in the discussion.

Question #2 "How often does if pprob = aprob then acc = acc + 0.5 occur?"

We designed this as a catch for a potential unlikely edge-case, and do not expect that is occurs often, but we will investigate this further.

Question #3: "Is it possible that training on detection makes the model worse at the 2IFC task for some images? e.g. if a single confident object detection yields a lower summed "probability" (pprob and aprob) than multiple less-confident detections that overlap the padded box?"

We appreciate the reviewer bringing up this point, as we went to great lengths in designing the machine 2IFC task to evaluate models as fairly as possible. We decided to utilize a summed probability rather than the maximum probability (best prediction) because in our observations, we found that baseline models tended to make few or no predictions when no objects were present, rather than many poor ones (Figure 1, top). In addition, when objects were present, models tended to make multiple overlapping predictions (Figure 1, bottom). Rather than enforce a match on object label and exact location, we wanted to give models the best chance at solving the present/absent tasks, as human participants may have been able to guess by selecting the more ‘object-y’ image. While it is possible that training could have made performance worse in some cases such as that aptly pointed out by the reviewer, we found that evaluating the sum of probabilities rather than the highest probability resulted in better model performance overall. We have included a list of design choices for the machine psychophysics in the Appendix Section A4.

“A more comprehensive description of the TTM should be provided”

We are thankful to the reviewer for asking for clarification regarding the release of our code, and for a more comprehensive description of TTM in our paper. Indeed, as authors we strongly value reproducibility in our work and aim to support this wherever possible. We have updated the manuscript to clarify that we will release both our dataset, as well as the code for the uniform TTM model used to create the dataset, in addition to the machine psychophysics model. Furthermore, we have supplemented the citation to TTM with a citation to the original TTM codebase (https://dspace.mit.edu/handle/1721.1/124819), as we aim to ensure full transparency. We especially appreciate the note on adding the reference to Portilla and Simoncelli, which we have now included in the paper.

“Data analysis and quantification”

We appreciate the reviewer asking for clarification regarding how data is pooled in figures 3c, 5, and 6 - we have added explanations for this in each of these figure captions. To clarify, figure 3 shows data for all human subjects and models for a single image. Figure 5 is not a fit to only the data from figure 3, but averages data over all human subjects over all images. We thank the reviewer for pointing out the absence of error bars for our critical eccentricity (\mu) fits in figures 3c, 5, and 6. For figure 3c which shows results for an individual image, this is indeed possible for the human data, and we have included these, as well as explanation in the figure caption. As figures 5 and 6 show data averaged over all human subjects and images, with different images having different inherent difficulties (object size, amount of crowding, contrast, etc - see Figure 7), error bars in these plots would not reflect the overall variability in performance in a meaningful way.

Regarding reporting uncertainty with error bars for the model data, we note that measuring uncertainty even for a single image (figure 3c) is not possible for the models given the machine psychophysics, which averages the simulated response of a single model over 100 trials (100 possible pairs of 10 present, 10 absent mongrels) at each eccentricity to give each accuracy value. There is no sense of multiple ‘individuals’ of the same model, and models are deterministic, always giving the same response for a given present/absent pair. We previously considered splitting the 100 trials into a simulated 10 accuracy values each with 10 trials to allow for the inclusion of error analysis, but found that calculating an accuracy value over only 10 trials gave too little accuracy granularity in the reported accuracy values (i.e. there would only be 10 possible accuracy values).

In addition, we value the reviewer’s concerns regarding the psychometric function fits to accuracy data. We have added clarification around Figure 5 being an average over fits to all images. Regarding the concern about removing two images from the dataset (Figure 13), we agree that including these images would have slightly raised the estimate of human \mu overall. We note that this would have strengthened our finding that humans outperform models, but nonetheless, being unable to fit a psychometric curve to this data we were forced to remove them in order to fairly compare humans and machines with a psychometric fit. To address all of these concerns, we have included a second plot in Figure 5, which represents accuracy data for both humans and models before any function fits or removal of images for which fitting failed.

Relatedly, we share the reviewer’s desire to evaluate these models at a wider range of eccentricities and with smaller step-sizes. Put simply, the computational requirements of creating these mongrel images (1-2 hours per image for the entire 328,000 MS-COCO dataset) limits the number of eccentricities we can feasibly explore.

To address these above concerns about psychometric function fits specifically, and along the lines of the suggestions by reviewer eW1M, we have included a plot reporting averaged accuracy results for each model, before the psychometric fit (Figure 5). This demonstrates the falloff in performance in machines happening more sharply and at earlier eccentricities as compared to humans.

We thank reviewer ogr4 for the thoughtful and careful review of our paper, appreciation of the strengths and contributions of our work, and for many helpful suggestions that will improve the paper’s impact.

Firstly, we thank the reviewer for pointing us to the Perry & Geisler citations which stitch together blur images for computationally-tractable fixation, similar to what we propose using uniform TTM images. We have included these citations in the manuscript.

“An alternative explanation for the results is that the TTM is an inadequate model of human peripheral vision”

We appreciate reviewer ogr4’s thoughtful alternative explanation for our results due to TTM throwing away too much information. Indeed, like all peripheral vision models, TTM does indeed underpredict human performance for certain stimuli and situations. Among all the peripheral models currently available, we chose TTM to create our dataset because it has been validated on the widest range of tasks compared to any similar texture-based model. It is true that the object detection task tested here differs slightly from the object detection task tested in Ehinger & Rosenholtz 2016, with more hallmarks of a discrimination task. We note that models sharing pooled TTM statistics have been widely tested on discrimination with the metamer paradigm (Freeman & Simoncelli 2011, Wallis 2019). We agree with reviewer ogr4 that there are newer models of peripheral vision that each contribute their own improvements such as style-transfer based models like (Wallis 2019, Deza 2017)). However these models focus primarily on the constraint of metamerism rather than predicting task performance which we believe is important and interesting to consider when making comparisons to DNNs. In our work, although we use object detection as an exemplar, our goal is for COCO-Periph to be applied in many other visual tasks such as search which is labeled in COCO (Chen et al., 2021), as well as scene and image recognition which could be explored using the COCO things and stuff labels (Caesar et al., 2018). We appreciate that our language did not discuss the limitations of TTM and other related models, and we have updated our manuscript to address these points.

In weighing the reviewer’s hypothesis against that presented in the paper, we acknowledge that the limitations of current state of the art peripheral vision models like TTM could contribute in part to our results. We see this as an important point to share with readers, and have incorporated this into the updated manuscript. We argue however, that the limitations of TTM cannot completely explain our results for the following reasons:

1. While TTM throwing away too much information could create a ceiling on model performance as compared to humans, our results also demonstrate the brittle falloff of performance for machine models as compared to the gradual falloff of performance in humans (Figure 3c). TTM uses the same statistical parameters at all eccentricities, modifying only pooling region size, which should not cause a sharp performance drop between low and high eccentricities. Indeed, we show when fine-tuning models on single eccentricities boosts performance at other eccentricities (Appendix Figure 22). Therefore, it is unlikely that this sharp performance drop off seen is attributable to TTM itself, and much more likely to be TTM-transforms being out of distribution to untrained models.

2. The differences in performance between the different model architectures tested (ie DINO, RCNN, DETR etc) cannot be explained by loss of information in TTM, as these models received the same TTM-transformed input images.

3. Relatedly, the improvement in performance seen after training and fine-tuning models on COCO-Periph demonstrates a change in the ability of even the same model architecture to process peripherally-transformed images differently. Again, the baseline, fine-tuned, and trained models receive the same TTM-transformed input images.

4. Also related, the improvement in corruption robustness after fine-tuning and training with COCO-Periph is orthogonal to any limitations of TTM as a peripheral vision model.

5. Furthermore, our results in comparing model performance in detection vs recognition are likewise orthogonal to any limitations of TTM as a peripheral vision model.