The 3D-PC: a benchmark for visual perspective taking in humans and machines

While the paper presents interesting tasks for studying 3D understanding, it could benefit from broader connections to the current 3D computer vision landscape… Adding some discussion about how their findings relate to work in depth estimation and 3D segmentation would help put their cognitive insights into a wider context We appreciate this feedback. Our original manuscript focused more on the growing body of work describing how 3D spatial property perception capabilities have begun to emerge in image analysis models. But we agree that we could expand the context to 3D computer vision more broadly. We have expanded our related work to incorporate this context.

Figure 6 provides an overview of DNN performance but deeper exploration about why models fail on VPT tasks would be better. Including detailed failure cases or decision attribution maps could shed light on where models diverge from human performance, particularly in VPT-perturb where models rely on “shortcut” strategies. We included comparisons of DNN decisions and error patterns (using Cohen’s Kappa) in section 9 of the appendix (also see Fig. 8). These experiments showed strong linear relationships for both depth order and VPT-basic between DNN accuracy and human alignment on these tasks. To better understand DNN failures on VPT-strategy, we compared DNN decision attribution maps to the ground truth location of objects in each scene (Fig. 9). DNNs do not learn to estimate the line-of-sight of target objects to solve VPT; instead, as this experiment suggests, they become progressively more sensitive to object features and their spatial relationships as they improve on VPT-basic (Fig. 9C). To expand on these analyses, we have added example DNN attribution maps for depth order, VPT-basic, and VPT-strategy to the Appendix of our updated submission. Also see our Response to all reviewers for additional thoughts.

I appreciate the authors' responses to my questions and their added details to the paper. My positive assessment remains unchanged.

I think for a benchmark paper, one of the most important things is the correctness of the ”ground truth labels”. Although the method proposed may sound reasonable, it is hard to tell how unlikely the “GTs” are inaccurate. After all, the adopted 3D Gaussian Splatting method is not perfect. Therefore, I expect some experiments or at least some discussions on the correctness of GTs.

Our goal was to systematically evaluate the 3D perception capabilities of DNNs and humans. We were particularly interested in understanding if DNNs could do VPT, since this is an ability that emerges over the first decade of life in humans. We generated images and derived ground truth labels using Gaussian Splatting in 3D space to ensure correctness. Ultimately, the true test for us was how well human participants could solve depth order and VPT in those images. In each case, humans were significantly above chance (Fig. 2, 75% accurate on depth order, 87% on VPT-basic, and 87% on VPT-strategy). Also note that DNNs outperformed humans on depth order after fine-tuning (Fig. 5A and 5C). In summary, humans (and models) were able to solve multiple 3D tasks posed on these images, indicating that our Gaussian Splatting image generation approach was sufficient for generating accurate GT labels. The failure of DNNs to solve VPT is not a byproduct of Gaussian Splatting artifacts.

I think the descriptions for VPT-Strategy are insufficient. I could only understand what this task was about when I read Figure 6. Thanks for this comment. We have revised our explanation in our updated manuscript.

I don’t think we can say that DNNs learn depth from the depth order experiments. It seems to be very likely that the linear-probed DNNs only learned to solve the task based on the relative sizes of the red ball and green camera. Therefore, I don’t see the meaning of the depth order task. There are multiple cues that humans rely on for perceiving object and scene depth, including the relative sizes of objects. As pointed out by the reviewer, relative sizes of objects are also likely a strong cue for solving our depth order task. The fact that DNNs can utilize this (and other) cues after linear probing is evidence that they learn human-like strategies for perceiving 3D spatial properties. As an example of the importance of relative size for 3D vision in humans, consider the Ames illusion, in which a person appears far bigger than they actually are based on the viewer’s expectations about their size. See our **Response to all reviewers ** for more details about monocular depth cues, relative size as a candidate cue for solving our depth order task, and why comparing performance on VPT vs. depth order on the exact same images enables unique and significant insights into differential 3D perception capabilities of humans vs. DNNs.

It seems that the role of the VPT-Strategy task is to test if the observer estimates the line-of-sight of the camera. However, I think it cannot take such a role. I guess if you finetune the DNNs on VPT-Strategy data, they will perform much better. If so, we can’t say they are estimating the line-of-sight because they are likely still using features to perform the task. Therefore, I don’t think the VPT-Strategy experiments are very meaningful.

The purpose of VPT-strategy was to see if DNNs (and humans) could solve VPT as we linearly moved the objects through the environment with a fixed camera. This should be a trivial perturbation for an observer that estimates the line-of-sight of the green camera, and indeed it is for humans. However, DNNs do not use line-of-sight, and hence struggle to generalize on images with object perspectives that rarely or never appear in the training set. To summarize, line-of-sight is not a necessary strategy for learning to solve VPT-basic, but it is necessary to generalize from VPT-basic to VPT-strategy.

It is claimed that AlexNet is included in the model zoo (L250), but I didn’t find it in Appendix Table 1? We apologize for the oversight. We have added AlexNet in Appendix Table 1.

Figure 2: VPT-easy → VPT-basic. Thank you for pointing this out. We have fixed it in our revision.

I think the example of the Ames illusion demonstrates exactly the opposite of what you emphasised, i.e., it shows how relative size is vulnerable for humans to use as a cue for estimating depth. It is because other cues are so powerful over relative size that humans tend to believe a person's real size gets bigger as walking. We agree with the reviewer and appreciate the chance to discuss this point. We only brought up the Ames illusion as an intuitive example of how the relative sizes of objects affect human depth perception. We apologize if this instead added any confusion.

I don't see why line-of-sight is necessary to perform VPT-strategy. As I said, I believe if you finetune the DNNs on VPT-Strategy data, they will perform much better, but they are likely still using features to perform the task. We agree with the reviewer that DNNs could likely learn to solve VPT-strategy with sufficient fine-tuning and data. However, we designed VPT-strategy to be a strong generalization condition for observers originally trained on VPT-basic: it consists of the same objects as in VPT-basic but with different viewpoints and object configurations. VPT-strategy is therefore a hypothesis-driven approach to identify observers (DNNs or humans) that do not use line-of-sight for VPT. DNNs failed to generalize to this condition while humans were successful. We have added another paragraph to the limitations section of our Discussion to address the reviewer’s question.

Finally, we validated developmental psychology work that humans rely on a line-of-sight strategy to solve VPT, and demonstrated in our VPT-Strategy task that DNNs learn to use a different strategy after fine-tuning. We provided evidence through our attribution map experiment (Appendix A.9) that DNN strategies are highly focused on the target objects, which means they may overly rely on the positions and sizes of target objects to solve the task. However, we were not able to characterize exactly how DNNs use object features for VPT, which may be critical for developing the next-generation of vision models that can solve VPT like humans do.

Using only green cameras and red target points in their test images raises a few questions: Would the main results still hold if they used other colors or camera shapes? Is the paper assuming the DNNs can easily spot the camera and determine its viewing angle in the images? If so, how much do the camera's physical properties (like its size, line thickness, or color) affect the results? We decided to use a red ball and green camera as our target objects because they are visually salient and easy to see. Indeed, humans were significantly above chance accuracy on depth order and both VPT tasks (Figs 3, 4, and 6). We also found no evidence that DNNs struggled to see the objects: all DNNs linearly probed on the depth order task performed significantly above chance (all were at least p < 0.05), and the best models were even more accurate than humans (Fig. 4C). In other words, we found no evidence that the colors or shapes of the target objects were problematic.

The paper mentions in its appendix that when testing Visual Language Models (VLMs), they provided in-context examples to help guide the models. However, it's unclear whether they also included step-by-step reasoning examples like chain-of-thought prompting, which can probably help the models better understand how to approach these perspective-taking tasks. Our goal in our original submission was to compare humans and VLMs on depth/VPT as fairly as possible. As discussed in A.6, we gave VLMs the exact same training images and instructions as human participants. Despite this, they struggled immensely on VPT even after trying multiple response temperatures and taking the best possible accuracy.

However, the reviewer raises a great point: if VLMs could achieve good accuracy on VPT after chain-of-thought prompting, it would imply that the challenge is more of a reasoning one than a perceptual one. To test this, we ran an additional experiment with GPT-4-Turbo, in which we cued it for chain-of-thought reasoning (see below for example depth and VPT instructions). However, chain-of-thought did not help the model’s performance (it was near chance accuracy; see Appendix A.7 for more details).

Depth: Here is a sample training image, from the perspective shown, is the red ball closer to the observer or is the green arrow closer to the observer? Answer only 'BALL' if the red ball is closer, or 'ARROW' if the green arrow is closer, nothing else. To answer this, let's think step by step: The red ball is above the suitcase, which is a small distance away from the observer, approximately 3 feet. The green arrow is in front of the suitcase, which is closer to the observer, approximately 1 foot away. Thus the answer is: ARROW

VPT: Here is a sample training image, if viewed from the perspective of the green 3D arrow, in the direction the arrow is pointing, can a human see the red ball? Answer only 'YES' or 'NO', nothing else. Let's think step by step: If a human were looking in the direction the arrow was pointing, they would face a few possible occlusions, such as the suitcase. The red ball is above the suitcase, so the suitcase wouldn't block the view of the red ball. Thus, the answer is: YES