Flexible Context-Driven Sensory Processing in Dynamical Vision Models

We thank the reviewer for their overall positive comments. We have addressed your comments below including full model and baseline specifications. We hope that we have sufficiently addressed your concerns enough to revise your score.

Capture the gist of interplay between low-order sensory areas and high-level cognitive areas. Top down modulation can be used to build a new class of DNNs

We sincerely thank the reviewer for this positive comment. This was indeed one of our primary motivations to pursue this line of work.

Did not specify basic information on their model

First, we would like to thank the reviewer for checking our source code. We also acknowledge that we could have done a better job of explaining our model components. We provide a full model description in the general response (and will include this information as part of our manuscript). Here, we provide more specific details on exact model parameters.

Layer 1 (Input size 3 x 128 x 128): Excitatory cell types (16), Inhibitory cell types (4). Kernel (5, 5), Padding (2, 2)

Layer 2 (Input size 16 x 64 x 64): Excitatory cell types (32), Inhibitory cell types (8) Kernel (5, 5), Padding (2, 2)

Layer 3 (Input size 32 x 32 x 32): Excitatory cell types (64), Inhibitory cell types (16) Kernel (5, 5), Padding (2, 2)

Layer 4 (Input size 128 x 16 x 16): Excitatory cell types (128), Inhibitory cell types (32) Kernel sizes (3, 3), Padding (1, 1)

Final Readout: Fully connected (1024 inputs, 6 outputs)

All layers have the following convolutional kernels ($**W**$s specified in the governing equations): Input to excitation, excitation to excitation, excitation to inhibition, inhibition to excitation, and input to inhibition. The input and output dimensionalities of these convolutions are listed layer-wise above, along with the kernel and padding shapes for all convolutions in that layer.

More details on the baseline models

Convolution filters are mentioned as: output channels x input channels x spatial size (height, width). Pooling layers are mentioned as: kernel size (height x width), stride. Inputs to every layer were normalized with LayerNorm.

Layer 1 (Input size 3 x 128 x 128): Conv2d (8 x 3 x 5 x 5), ReLU, AvgPool2d (5, 5, stride 2).

Layer 2 (Input size 8 x 64 x 64): Conv2d (16 x 8 x 5 x 5), ReLU, AvgPool2d (5, 5, stride 2).

Layer 3 (Input size 16 x 32 x 32): Conv2d (32 x 16 x 5 x 5), ReLU, AvgPool2d (3, 3, stride 2).

Layer 4 (Input size 32 x 16 x 16): Conv2d (64 x 32 x 3 x 3), ReLU, AvgPool2d (3, 3, stride 2).

Layer 5 (Input size 64 x 8 x 8): Conv2d (128 x 64 x 3 x 3), ReLU, AvgPool2d (2, 2, stride 2).

Layer 6 (Input size 128 x 4 x 4): Conv2d (128 x 128 x 3 x 3), ReLU, AvgPool2d (1, 1, stride 1).

Project convolutional outputs to GRU: Fully connected (2048 inputs, 2048 outputs), ReLU

Recurrence: GRU (2048 inputs, 2048 outputs)

Final Readout: Fully connected (2048 inputs, 6 outputs)

The convolutional backbone serves as a feature extractor for the GRU network. We train this baseline model by passing in the cue for the first $T$ time steps followed by the scene for the next $T$ timesteps. For our experiments we set $T=3$. A CrossEntropy Loss is used on the readout activities at the final timestep.

We will include these details in the Appendix.

Unclear modulation signal is formed from concatenating r1 and r2s

We apologize for this confusion. The modulatory signals are computed in a layer-matched manner and hence there is no issue of concatenation.

However, we include in this rebuttal experimental results (in response to qySN) for a variant of the modulation signal that pools from all layers before computing the modulation factors. For this implementation, we linearly projected excitatory neuron activities from every layer into the dimension of interest (based on which layer is being modulated), followed by a sigmoided linear-nonlinear transformation.

We thank the reviewer for their positive comments. We believe that some of the suggestions raised by the reviewer will increase the quality of the manuscript and we now provide additional analyses and responses to these.

Relatively high level of biological detail. The network is image computable and can be trained for reasonably complex tasks

We thank the reviewer for this note. As we noted in our response to BT2x, one of our goals was to tackle the increasing divergence between successful deep learning approaches that model visual computations and biological visual processing.

Direct reaction time matches to human data

We agree with the reviewer that target-absent trials yield higher reaction times in humans while we don’t see this in our model output entropy. We wish to highlight that model entropy is far from the perfect RT metric that one can extract from neural dynamics. In fact, the subject of computing model RTs is an active research area (Spoerer et al. (2020); Goetschalckx et al. (2024); Subramanian et al. (2022); Graves (2016) to name a few), and exploration of different metrics is beyond the scope of this work. Our goal here was to expose the potential for studying model RTs in the context of cued-paradigms (through DCNet). Cued-contextual paradigms constitute a wealth of data in human psychophysical research but to our knowledge, there is no current modeling approach to study this in a naturalistic manner.

As for Fig. 7c: We limit the number of distractors to 6 to account for the canvas size and placement of the objects within it. The reviewer’s point is well taken. Our response is the same as above – entropy isn’t the perfect metric. We genuinely hope to extend on this in future work.

Human data comparison to 7d: While generating our stimuli, we used perceptually uniform color spaces to stay close to the experimental paradigm discussed in Wolfe and Horowitz (2004) Fig. 5a-e. We agree that this isn’t a direct comparison to human RTs and emphasize that we wished to only show human-like trends (which is strongly present in our analysis). Exact comparisons to human RTs is beyond the scope of this work, but it is something we are really interested in exploring in future work.

We will add the limitations of our chosen RT metric as part of the limitations and outlook for the future in our manuscript.

Customary to look at the accuracy of responses as well somewhere

Thanks for the comment. We do refer to the accuracy of model responses in the manuscript.

L222 “DCnet achieved an overall accuracy of 97% on the test trials”.

L238 “When fine tuned on this task, DCnet achieved an overall accuracy of 95% on the test trials”.

Overall the performance improvement is marginal

As the reviewer points out, the proposed network performs marginally better than ResNet-18 in-distribution (Fig. 2b), the difference is much greater when tested out-of-distribution (Fig. 2c) – our model drops by $\sim 0.2\%$ while ResNet-18 drops by $\sim 0.6\%$. We wish to highlight that performance was not our only criteria of desire. Rather, we wanted to build a competitive and biologically-faithful framework that will allow us to study the dynamics of contextual-cueing phenomenon.

Proper analyses of the dynamics...activity traces is insufficient.

We fully agree with the reviewer and thank them for raising this point. While comparisons to neural data is beyond the scope of this rebuttal timeline, we perform several analyses to better understand and quantify the internal dynamics of our model. We missed the point with Fig. 5, but will now update it to include the analyses we detail below (also in Fig. R2).

As originally presented in Fig.5, we drive DCNet activity by uncorrelated, time-varying Gaussian noise inputs. We then compute:

1. The Dynamic Range (DR) of excitatory cells in each layer of DCNet and compare that to the DR of corresponding excitatory cells from DCNet (Lesioned Inhibition) (Fig. R2c). DR is computed as the Interquartile range (a measure of statistical dispersion) of a neuron’s activity over a time period of $128 ms$ when driven by noise. We take the mean over 64 trials for each neuron and plot this distribution per-layer in Fig. R2c. We find that excitatory neurons in the DCNet model have a significantly higher DR across layers compared to DCNet (Lesioned Inhibition) suggesting the role of inhibitory interactions in expanding the range of computations carried out by each neuron. A Kolmogorov–Smirnov test confirmed these significant differences (Layer 1 (statistic=0.667, p < .001); Layer 2(statistic=0.99, p < .001); Layer 3(statistic=1.0, p < .001); Layer 4(statistic=0.97, p < .001)).

2. The lag 1 autocorrelation as a measure of stability of the excitatory neurons (Fig. R2b). We find that DCNet excitatory neurons are significantly more stable than excitatory neurons in the DCNet (Lesioned Inhibition) model. A Kolmogorov–Smirnov test confirmed the significant difference (statistic=1.0, p < .001).

3. The E-I correlation coefficient as a measure of co-tuning in DCNet. We find that the average (across neurons) E-I correlation is as follows: -0.076 (Layer 1), 0.766 (Layer 2), 0.699 (Layer 3), 0.535 (Layer 4). This confirms the visual intuition provided in Fig. R2a that E-I co-tuning is weaker in early compared to late layers.

Did the authors explore variants of the model...extremely reduced.

The reviewer makes an excellent suggestion. Inspired by this, we implement and test a new form of top-down modulation that pools information from all cortical layers before computing time-delayed top-down modulating factors as a non-linear transformation (one per layer) of this pooled input. This model achieved an overall accuracy of $92.35\%$ on vis-count, which is better than the metrics we report in the manuscript. However, we note that more work needs to be done to understand this better as the biological analogues of this process is not apparent.

We thank the reviewer for their majorly positive comments. We believe that the suggestions raised by the reviewer will increase the quality of the manuscript and we now provide additional analyses and responses to these.

Explicitly highlight DCNet’s anatomical constraints relative to prior work

Thanks for this comment. Also, in line with comments from the other reviewers, we now provide a full model specification in the general response. We will make sure to highlight and signpost anatomical facets of our model that are unique including: distinct E and I subpopulations, lateral recurrent feedback, learnable neuron time constants per cell-type, and long-range top-down feedback conceptualized as multiplicative low-rank perturbations.

Ablation studies to dissect individual components of DCNet architecture

We agree with the reviewer. In this regard, we now perform two ablation studies.

First, we train a version of the model where we lesion the Inhibitory cell populations in every layer. We analyze the internal input-driven dynamics of this model variant in Fig. R2. We train DCNet (Lesioned Inhibition) on vis-count (color condition; for the purposes of time). While DCNet (Lesioned Inhibition) learns the task ($95$% accurate), we find that neurons have a significantly smaller dynamic range and are comparatively less stable (details of these analyses are given in our response to qySN) when driven for longer time periods.

Second, we train a version of the model with lesioned top-down feedback. DCNet (Lesioned Top-down) drops significantly in performance to $38$%, further highlighting the importance of top-down feedback.

Unclear how relevant models [18 - 21] perform on vis-count task and differences between those models and DCNet

Thanks for raising this point. We did not evaluate these models on vis-count as none of these models work within a cueing-paradigm. We will, however, make sure to highlight aspects of these models that are similar to our work. [18, 19, 20] are single-layer convolutional recurrent neural network models with lateral feedback and distinct E/I populations. [21] is an extension of [20] to include a feature processing hierarchy. All these models included ML-esque normalization operations (such as BatchNorm or LayerNorm) to impart stability during training and none of these models included cell-type specific learnable integration constants as well as time-delayed top-down modulation.

Thanks again for your positive evaluation of our manuscript. We hope you had a chance to review our rebuttal and the analyses presented there in. If you believe we had addressed your concerns, we would appreciate it if you can increase your score. Many thanks!

homunculus is misspelled both times

Thank you for pointing this out. We have made the correction.

No explicit testable predictions/potential experiments that come from the model anywhere in the paper (L65)

We believe our framework is among the first to link physiology to behavior (through computations) and is a necessary first step towards hypothesis generation. We will clarify this further in the manuscript.

Lack of clarity in model specification

We apologize for this confusion. We take this comment seriously and we now spell out the entirety of our model formulation in the general response. We are happy to answer any further questions the reviewer might have.

Fig 2b: does not seem to actually outperform ResNet, even though it seems like it might have an advantage over ResNet because of cuing (though this is unclear because they don't explain cuing)

We would like to clarify this point. In Fig. 2b what is shown is the performance of the models when the cues are in-distribution. As we point out in our answer above, the implicitly-cued models are supposed to serve as a potential performance upper-bound. The implicit-cue condition is computationally easier compared to the explicit-cue condition. We point the reviewer to Fig. 2c which shows that on a generalization condition, when the cues are out-of-distribution the same ResNet model falls to chance.

Besides the Conv RNN model (maybe?), are these comparable baselines if they have no temporal dynamics?

Thanks for raising this point. As we discussed above, the baselines were primarily constructed for two reasons. First, we wanted to understand performance bounds and verify that vis-count is a non-trivial computational challenge. Second, we wanted to study the computational implications (and benefits) of two components of our framework: temporal dynamics, and top-down feedback. The Conv RNN model has temporal dynamics, but no top-down modulation. The feedforward baselines have neither.

144: Don't seem to describe the novel cues and scenes anywhere. How different are they from the training data? Is a 55% accuracy impressive?

Thanks for raising this point. The scenes are split into a training and test set. They are i.i.d. samples from our data generation process. None of the scenes used for evaluation were seen by the model during training. We perform two types of tests with the cues. The weak generalization experiment contained cues (colors, shapes, conjunctions) that the model had seen during training (but for different scenes). The stronger generalization experiment used cues never encountered in training. Specifically, for our stronger generalization experiment, we keep the color cues green and blue; the shape cue cube; and conjunctions of these out of the training set.

Chance performance for all of these tasks are $16.67$%. An accuracy of $55$% is well above chance and significantly higher than the performance of models with orders of magnitude more parameters (Fig. 2c; hatched data).

We will add these clarifications to the manuscript.

148: How do they know it included the cyan cylinder and not another object (and therefore conclude it's an illusory conjunction)? Could it also be confused by the occlusion of the blue cube?

The reviewer is correct in pointing this out. Identifying either the blue-cube or the cyan-cylinder would constitute a “binding error”. We highlighted this example since the feature dissimilarity between the target object and distractors (apart from the cyan-cylinder and blue-cube) are very high and is unlikely to be the cause of the error. However, the reviewer’s general point is well taken and we will perform an in-depth model explainability analysis. In the meantime, we will reword this sentence to reflect that this is a possibility and not a certainty.

“When cued to find “blue cylinders", the model mistakenly appears to include either the cyan cylinder or the occluded blue cube in its count; which would count as an illusory conjunction”.

184: Unclear what "cortical gradients" means, I'm assuming trajectory in PCA space? Is it that interesting/surprising that trajectories are different for color and shape given the model presumably would have to disentangle in order to get high accuracy?

We believe there is a misunderstanding here. In Fig. 4 we show the results of a lesion experiment we performed, where we systematically cut-out modulations from each layer of the model. Data presented in Fig. 4 are not trajectories in the PCA space. We find that lesioning later layer modulations in the model significantly impacted accuracy on the “shape” trials but not the “color” trials. Lesioning early layer modulation significantly impacted accuracy on “color” trials. From these we conclude that color selectivity emerges early while shape selectivity emerges late. The selectivity profile across layers of the model is what we refer to as a “cortical gradient”.

169: "We believe"... this whole paragraph seems like speculation based on looking at Fig. 2, but no actual proof/argument?

We believe that the reviewer is referring to Fig. 3 here, and not Fig. 2. In Fig. 3a, we consider model trajectories of the same several hundred scenes (individual trajectories shown in gray) when modulated by four different cues (subpanels). Our point is better illustrated by Fig. R3. When a given scene is cued with different colors, dynamics are driven to context-relevant states though the bottom-up responses from the scene are exactly the same. This is possible only by the inactivation of context-irrelevant subspaces.

We thank the reviewer for their feedback and insightful comments. We provide a point-by-point response below. Your comments have made the manuscript stronger. We hope that we have sufficiently addressed your concerns enough to revise your score.

Model mechanisms are motivated by physiology/behavior and an attempt is made to understand recurrent interactions in deep neural networks which is missing from much of the literature.

We are particularly happy that the reviewer pointed this out. One of our motivations in pursuing this line of work is to tackle the increasing divergence between successful deep learning approaches that model visual computations and biological visual processing. We will make sure to highlight this in our discussion.

What about natural vision? Why focus on a “laboratory task”?

Thanks for raising this point. It is true that feedback processes in the visual system are critical for a variety of functions. Having said that, we wish to highlight that our approach, to the best of our knowledge, is among the first to tackle contextual cueing paradigms in a realistic manner.

Our focus was on the problem of how models can learn sensory representations of “cue”-ed attributes to look for in a “scene”. Please note that these attributes may not directly be present in the visual scene. For example, when cued for a specific color, that color may (or may not) exist in the scene under various lighting conditions, shadows, occlusions, object textures, material properties, and at different scales. To do this, one needs to be able to generate systematic variations on a theme. Although vis-count may not be “naturalistic” in a traditional sense, it contains non-trivial variations that are important to study. We highlight that there is a lack of similar datasets for naturalistic visual search at scale.

We do hope to build towards naturalistic scale, and as a first step, we now try training our sensory backbone on naturalistic object recognition tasks (without standard ML bells and whistles such as model ensembling, test time augmentations, etc.). We report an accuracy of 84.79% on CIFAR-10 (Fig. R4). This is only a preliminary result, but ones which strongly demonstrate the potential of our framework to scale up.

The focus is on phenomenology as opposed to gaining mechanistic insights.

Point well taken. We believe these are non-orthogonal interesting things to study. We have now performed a host of quantitative analyses to gain insights into the role of inhibitory influence in determining the dynamic range and stability of the excitatory neurons and co-tuning (please see our response to qySN). Additionally, we also highlight a macroscopic gradient in learned time constants in Fig. R1.

``Kitchen sink” approach to Figure 1.

We wish to point out that the anatomical constraints that we include are simplified forms of well-studied elements in biophysics. Our ablation studies make it clear that the design constraints here (separate excitatory and inhibitory populations, top-down feedback, learnable dynamics, etc.) are both necessary for model performance and facilitate comparisons to relevant neuroscience literature.

Intuition for how the model is cued and performs the task, or how it learns

We apologize for the lack of clarity in this aspect. With our revised model description (provided in the general response), we have tried to make this point clearer.

Both cues and scenes are visual inputs ($\in \mathbb{R}^{128 \times 128 \times 3}$) provided to Layer 1 of our model. Cues are persistently provided to the network for the first $T$ discrete time steps followed by scenes for the next $T$ time steps. The output activities of the last layer at the last time step ($t = 2T$) is transformed into logits and a supervision signal is provided via a cross entropy loss (ground truth labels are counts from 0 to 5).

Intuitively, our model learns to “up-modulate” features in the scene that resemble features of the cue, and ultimately use these up-modulated features to inform its final output. To do so, it needs to learn a disentangled feature basis (shared between cues and scenes as they are processed by the same backbone) because aspects of the same scene can be up- or down- modulated differentially based on the cue. This is what we try to expose in Figure 3.

Unclear if the comparison to models without time dynamics and explicit cuing is informative

We believe that these comparisons are important particularly because it helps understand the upper bounds of expected performance. Our “implicit” cueing condition is one where the cue and scene are provided to the model at that same time. In practice, for a model without time dynamics, such as the ResNet and the Transformers, this is realized by stacking these two inputs before passing them into the model. This allows for the model to perform direct comparisons between the features of these two inputs, as opposed to time-delayed comparisons that happen in the explicit-cuing condition. Hence, this provides a potential upper-bound on training performance. Interestingly, while some of these implicit models can learn the task well (such as the ResNet in Fig. 2b), they fail to generalize (Fig. 2c).

We hope you had a chance to review our rebuttal. Given your detailed feedback and suggestions, we believe our paper has improved. Please let us know if there is anything else that we can clarify. We kindly ask you to consider revising your score if you agree that the primary concerns raised in the review were addressed.