Emergent Geometry in Neural Representations of the Visual World

The main weakness can be summarized as not enough evidence to back the claims. Claims related to the universal properties of processing of the data in intelligent systems (natural and artificial) but both experiments (on oriented objects and oriented textures) were performed on a limited set of conditions, and only a specific layer of a specific architecture was used during experiments. Also, it's not clear if the results are specific to robust (sensitive to low-frequencies, re: texture dataset) vs standard (sensitive to high-frequencies re: object dataset) resnet50.

One could think of an alternative explanation:

** In object set (Experiment 1), the variance primarily exists in high-frequency parts of the images, that's why it persists in the early layers of an untrained resnet50. In texture set (Experiment 2), the variance exists in low-frequency parts of the images, that's why a robust resnet50 picks up on that, it's known that robust nets are sensitive to low-frequencies.**

experiments are insufficient

So here are a few experiments to strengthen the claims ""

1- Claim on 'structured data' preserved even by untrained network representation: only 'layer 3' of untrained resnet50 was tested. What about deeper layers? What about an untrained resnet110? I predict this structure may not survive more aggressive accumulation of non-linearities. Also, It seems like that as a control to digital twin, standard resnet50 was used in experiment 1 and robust resnet50 was used in experiment 2. Why? How does robust resent50 change the manifold where the variance is mostly in high-frequency parts of the image (Object set, experiment 1.) To what extent the result depend on this specific image set? How does the manifold structure change without background or with random distinct backgrounds?

FINDING 1: THE GEOMETRY OF THE DATA MANIFOLD PROPAGATES TO NEURAL SPACE Does the geometry also preserved in standard resnet50 or deeper untrained neural network, or deeper layers of robust neural network?

2- Claim on the structure 'unstructured data' is recovered by a trained model is only partially tested, as it is unclear what the manifold would look like in a standard resent50 which was used in experiment 1, or is the recovery of the manifold only a product of adversarial trained neural networks (or networks trained directly on neural data, aka digital twins which many other works already shown direct correspondence between the two). Also, I wonder if really a massive network trained on massive data is needed to extract the manifold in these simple textures, these simple textures could be processed by a low-freq filter (perhaps with an addition of simple non-linearities). One simple test could be to take the model of early vision proposed in Henaff, Goris & Simoncelli 2019 (code available on github) and run the texture images through it to see if that simple network is enough to capture the hidden manifold in this data. So,

FINDING 2: THE VISUAL CORTEX RECOVERS THE WORLD MANIFOLD FROM AN UNSTRUCTURED DATA MANIFOLD

is really "the network trained on neural data (not visual cortex, because these images were never shown to the animals) recovers the low-dimentional manifold present in the data". What makes a manifold World manifold?

3- Experiment 3 needs to test at least standard resnet50 as well, and on more layers of both types of resnet50 before concluding this generalization "is a universal property of intelligent vision systems".

Recommendation While the work shows promise and moves in an interesting direction, it requires additional control experiments to support its broad claims. Specifically:

Investigation of simpler alternative explanations

Testing across multiple network depths and architectures

Consistent comparison between standard and robust networks

Control image sets (no background or random background for object set)

In summary, although this work seems interesting and in the right direction, at this point it needs more control experiments (see above for a few very simple suggested experiments) or refinement of the claims. At this current format, unfortunately, I can't recommend it for publication. However, I am happy to increase the score if some concerns were addressed.

My first concern is about the novelty of the findings presented in this paper. It is well-known that networks can learn representations of latent factors of variation in the world, such as rotation (Higgins, et al., 2017, Jahanian, et al., 2019). The in-plain rotation equivariance of CNN representation is studied in depth in the work of Lenc et al., 2015, where they find the representation in AlexNet has significant rotation equivariance, and one can utilize this property to better classify images at different rotations and identify the rotation of objects in novel images. While Lenc et al. did not visualize the data using PCA, and there are differences in the specific methods being used, conceptually, there are many overlaps between the results presented here and Lenc et al. It would be helpful if the author could elaborate more on how their results differentiate from these previous results. If the contribution is mainly in showing the rotation equivariance also exists in the representation of a CNN trained to predict neural data, and thus, it is likely that rotation equivariance also exists in the brain, it would be helpful to show how well the "digital twins" model performs at capturing the neural data (and generalize to the images being tested here).

My second concern is about the scope of the claims. The work frames the "mirroring hypothesis" as a general principle that may apply to many latent variables in the world. For example, in the conclusion section, the author concluded, "We demonstrate that neural representations in area V4 of the primate visual cortex, as well as in trained artificial neural networks, reflect geometric structures of latent variables in the visual world.". However, the evidence provided in this paper is only restricted to 2D in-plain rotation. It is unknown whether the claim holds for many other latent variables, such as translation, 3D rotation, object color, weight, etc. It would be helpful to provide more evidence about other latent variables if this paper makes the claim more general. Or the author could restrict the claim to about in-plain rotation only. In addition, to make a general claim about the mirroring hypothesis, it would be helpful to clarify how to define the "geometric structure of the latent world," which is only vaguely defined here in this paper.

References:

1. Higgins, I., Matthey, L., Pal, A., Burgess, C.P., Glorot, X., Botvinick, M.M., Mohamed, S. and Lerchner, A., 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR (Poster), 3.
2. Jahanian, A., Chai, L. and Isola, P., 2019. On the" steerability" of generative adversarial networks. arXiv preprint arXiv:1907.07171.
3. Lenc, K. and Vedaldi, A., 2015. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 991-999).

1. I doubt the very essential novelty for this study.
2. The abstract is obscure and contain little information. There is no real meat there.
3. The descriptions of model training details are unclear.