BrainACTIV: Identifying visuo-semantic properties driving cortical selectivity using diffusion-based image manipulation

We thank Reviewer 5W1K for the constructive review of our work. We address BrainACTIV’s motivation and concerns regarding technical novelty below. Further, we have carefully considered each of the comments and revised our paper accordingly.

W1: Advantages of image manipulation

Our work indeed follows an existing line of research attempting to formulate new hypotheses about stimulus representation in the human visual cortex through data-driven analyses (facilitated by the recent availability of large naturalistic neuroimaging datasets). Existing methods generate images that maximize (predicted) activations of a target brain region, enabling the qualitative interpretation of visual/semantic properties that may be preferred by this region. However, none of them have explicitly enforced structural constraints on the generations; thus, these individual images have no explicit point of comparison. Consequently, their conclusions rely on the joint interpretation of a large number of images through, e.g., human behavioral studies to discern image features that are of relevance from those that were randomly generated by the model (hence irrelevant for driving activations).

The introduction and modification of a reference image provides a direct comparison point for each synthesized stimulus, bringing about several benefits:

• Straightforward interpretation of image features that are relevant to the brain region (because random variability from the diffusion model is minimized).
• Possibility to quantify these relevant features through automated methods from computer vision (because the values for reference and variation images are directly comparable). Hence, there is no longer a need for human subjects to interpret hundreds of images, and the computational costs are vastly reduced.
• The produced image variations represent a hypothesized tuning axis for the target region or group of voxels, with the reference image serving as a control stimulus. Hence, BrainACTIV can be employed by researchers in novel neuroscientific studies to study fine-grained selectivity properties.

To make this motivation clearer in the text, we have rephrased several sentences in the Abstract and Introduction, and we have reordered and expanded previous Results sections 4.3 to 4.5 to better convey these advantages.

W2: Technical novelty

We acknowledge that BrainACTIV primarily implements existing methods for controllable image generation. However, we believe in the importance of bringing tools and ideas from machine learning closer to adjacent scientific fields (particularly, computational neuroscience), which is why we marked ‘applications to neuroscience/cognitive science’ as our primary area The fast-paced development of new models and methods in machine learning leaves many potential applications unexplored, missing out on significant impacts to these fields. We argue that BrainACTIV represents an innovative application of controllable diffusion-based image generation to advance our understanding of the human visual cortex.

BrainACTIV is the first work to explore image variation with respect to a reference image using generative models to (1) formulate novel hypotheses about stimulus representation in the human visual cortex and (2) produce controlled experimental stimuli for novel neuroscientific experiments.

In addition, the incorporation of IP-Adapter and SDEdit significantly reduces the computational costs needed to generate conclusions, relative to BrainDiVE. Our image generation process takes ~80 hours for all subjects on an NVIDIA A100, while Luo et al. (2023) [b] report 1,500 hours on an NVIDIA V100 (reports indicate A100 is only ~2 to 4x faster than V100 [a], while our results take ~9x less time, accounting for the difference in Stable Diffusion versions). We have briefly mentioned this advantage in line 350. Moreover, we remove the need for human behavioral studies to analyze the generated images, further saving time and costs.

---

• [a] https://lambdalabs.com/blog/nvidia-a100-vs-v100-benchmarks
• [b] Luo, A. F., Henderson, M. M., Wehbe, L., and Tarr, M. J. (2023). Brain diffusion for visual exploration: Cortical discovery using large scale generative models. NeurIPS.

We appreciate the positive feedback and helpful suggestions from Reviewer eXwK. We address each of them below.

Q1 and Q2: Clarity in text

Thank you for this valuable feedback. We have updated lines 228 and 345 to make these variables more explicit in the text.

Q3: Additional baseline in Table 1

We agree that the baseline of random images constitutes a lower baseline and that it would be useful to have an upper baseline as well, to quantify the penalty on structural control associated with brain-conditioning . However, we are not entirely sure what insight we would get from comparing to variations without brain conditioning. In our view, this would emphasize that the structural control as implemented by SDEdit is working correctly, but the distance values would be really close to zero since the variation and the target would be almost identical. We welcome further elaboration on the added value of this suggested baseline or other potentially useful baselines.

Q4: Improvement to Figure 6 (now Figure 7)

We agree in adding the original image in old Figure 6 (now Figure 7). We have omitted the row and column corresponding to $\alpha=0$ and $\gamma=0$ from the examples, since these are highly redundant. We have added the reference image in Figure 7(A) within a simple graphic explaining the usefulness of BrainACTIV for producing novel neuroscientific stimuli. This accompanies changes made to Section 4.5 to emphasize this contribution based on reviews.

Q5: Literature review missing references

Thank you for pointing this out. We acknowledge the vast amount of work performed on image reconstruction from brain activity, and its connection to our work as an alternative way to perform brain-guided image synthesis. Accordingly, we have added relevant citations in our Related Work section. Particularly, the work by Ozcelik et al. (2023) [a] is a good example of image reconstruction methods that can be adapted to generate optimal inputs for specific ROIs. We have added citations to this work in lines 60 and 140.

---

• [a] Ozcelik, F. and VanRullen, R. (2023). Natural scene reconstruction from fMRI signals using generative latent diffusion. Nature Scientific Reports, 13.

We thank Reviewer Dc87 for the positive evaluation of BrainACTIV and the insightful questions regarding novel discoveries. We address these below, and incorporate the concrete suggestions into our paper.

W1: Predictive performance

Thank you for this suggestion. We agree that the predictive performance of the encoders provides valuable context to interpret the predictions in Figure 3(B). Because of space constraints, we have included these performance metrics in Appendix subsection A.2. Based on other reviewer comments, we have also updated this subsection to display the predictive performance of our CLIP encoder. Accordingly, we have updated Figure 3 with the corresponding CLIP encoder predictions.

W2: New discoveries

The discovery of novel or finer-grained stimulus representations and organizational principles in the visual cortex is a very important topic in our line of research, and one that must certainly be emphasized in follow-up studies through collection of brain responses to synthesized images in the MRI scanner. We have included an example on the use of BrainACTIV to formulate hypotheses for less well-understood regions (particularly, anterior IT cortex) in Appendix A.7.

BrainACTIV stands out from current approaches to optimal visual stimulus generation in that our synthesized stimuli are directly comparable to a reference image, whose structural properties are preserved. The relevance of this contribution can be understood from different points of view:

• The successful modulation of predicted brain responses through changes in semantic categories in the image (while controlling for low-level structure) enables a strong test (and evidence for) the (well-known) category selectivity theory (Figure 2).
• BrainACTIV can be used by experimenters to produce controlled experimental stimuli, where the hypothesized tuning axis of a group of voxels is derived in a data-driven manner. Additionally we describe how researchers can select the degree of structural control and semantic variation (Section 4.5).
• By manipulating a reference image (instead of generating novel images from randomly sampled noise), the analyses required to formulate new hypotheses become significantly more efficient in terms of compute costs (less images are needed) and automatization (no need for human inspection of hundreds of images).

Additionally, we present a novel way to generate fine-grained hypotheses about ROIs with similar category preferences in Section 4.4, by isolating what distinguishes one ROI from the other and directly accentuating it in an image. We believe this contribution can be used not only to explore selectivity in novel regions of interest, but also to expand our knowledge about well-known category-selective regions.

Q1: Closeness to baseline activation

Thank you for the opportunity to clarify this choice. We select images closest to baseline activations primarily for efficiency purposes: because the ROI-specific modulation embeddings point in the same direction regardless of the image, the semantic categories that appear in the manipulations vary minimally across images. Hence, we avoid the manipulation of the entire test set (1,000 images) and instead focus on a smaller subset. Using images with activation close to baseline probabilistically represents an “average” visual stimulus (i.e. there are significantly more stimuli with baseline activation than with optimal activation) and intuitively provides more room for maximizing/minimizing activity. Because this choice could bias the selection (e.g., towards a single semantic category that always elicits baseline activation), we employed six mutually-exclusive image subsets to enforce the diversity of the 120-image set.

Q2: Structural fidelity

Thank you for pointing out this ambiguity. We have updated Table 1’s caption and line 361 to emphasize our goal: reference and variations should be as similar as possible in spatial structure and color palette. Because relevant works (e.g., NeuroGen [a] and BrainDiVE [b]) do not use a reference image or control the low-level structure of their images, a direct comparison in this table is unfortunately not possible.

---

• [a] Gu, Z. et al. (2022). NeuroGen: Activation optimized image synthesis for discovery neuroscience. NeuroImage, 247:118812.
• [b] Luo, A. F., Henderson, M. M., Wehbe, L., and Tarr, M. J. (2023). Brain diffusion for visual exploration: Cortical discovery using large scale generative models. NeurIPS.

We appreciate the concrete suggestions made by Reviewer 82Gz. We have carefully considered them and adapted our paper accordingly. We address each suggestion below and gladly welcome further questions about them.

W1: Related works on stimulus reconstruction

Thank you for this important suggestion. Due to the wealth of work on both image reconstruction / brain decoding and data-driven exploration of category selectivity (particularly using the Natural Scenes Dataset), we initially decided to narrow the focus of our Related Work section to studies that specifically focus on optimal visual stimulus generation.

For clarification, we would like to emphasize the two different directions regarding brain-conditioned image generation: the first one tackles (exact) reconstruction of observed stimuli based on the brain activity patterns they elicited ([a][b][c]) while the second one concerns the generation of novel stimuli that activates a specific brain region. These two directions share many components (particularly, those using diffusion models and fMRI data), and the methods can often be adapted to perform both of them (see [d][e]). We therefore acknowledge the vast amount of work performed on image reconstruction from brain activity, and its importance to our work. Accordingly, we have adapted the Related Work section by adding the suggested citations. Because our work pertains solely to the second mentioned direction, we preserve the focus of this section on works that deal with optimal visual stimulus generation aiming to provide better context on the evolution of this subfield of research.

W2: Performance of CLIP encoder

We strongly agree that the performance of the CLIP encoder provides important context for our results. Therefore, we have adapted Figure 9 to display the performance of the three brain encoders on the held-out test set for all subjects and ROIs. We have also added the corresponding CLIP encoder predictions to examples in Figure 3(A), as well as predicted modulation results from the CLIP encoder in Figure 3(B).

Due to time constraints, it was not possible to provide a comprehensive analysis on the effect of random initialization of the encoders and generative model on our results. However, to provide further evidence that BrainACTIV minimizes the effect of random variability from the generator model, and better isolates the effect of brain optimality, we have displayed example variations for FFA and PPA over multiple different seeds (for the diffusion model) in Appendix A.10.

---

• [a] Scotti, P. S. et al. (2023). Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors. NeurIPS.
• [b] Chen, Z. et al. (2023). Seeing beyond the brain: Masked modeling conditioned diffusion model for human vision decoding. CVPR.
• [c] Zeng, B. et al. (2023). Controllable mind visual diffusion model. arXiv preprint. arXiv:2305.10135.
• [d] Ozcelik, F. and VanRullen, R. (2023). Natural scene reconstruction from fMRI signals using generative latent diffusion. Nature Scientific Reports, 13.
• [e] Papale, P., De Luca, D., and Roelfsema, P. R. (2024). Deep generative networks reveal the tuning of neurons in IT and predict their influence on visual perception. bioRxiv, pages 2024–10.

We thank Reviewer mur3 for the positive comments and insightful questions. We have revised our paper based on these questions to provide more context into the relevance of our work. We will address your questions and suggestions below.

Quantification / details of the observed effects

How many images are used to compute the selectivity results (e.g. figure 4), why are those chosen?

All analyses in Section 4.3 (specifically, Figure 4 and Figure 5) use the results from our main experiment (Section 4.2). Namely, 20 variations for each of 120 test images in each subject. To aggregate these results into a single figure, we average over all subjects and all images. We make this particular choice because the modulation embeddings z_max and z_min, used as endpoints to generate the subject- and ROI-specific image variations, are highly similar across subjects (please see the newly added Appendix subsection A.8). Hence, the semantic and mid-level features modified in the images vary minimally across subjects. We have updated the first paragraph of Section 4.3 to clarify this information in the main text.

What is the predicted activity associated with the maximizing and minimizing images?

Figure 3 (A) displays example results for our maximization and minimization experiment from Section 4.2. For each of them, we show the activity of the target region of interest as predicted by each of our three brain encoders (based on the other reviews, we have updated Figure 3 to also display predictions from our CLIP encoder). We further show the average of these values over all images and subjects (with a reasoning similar as mentioned above) in Figure 3 (B) to evidence BrainACTIV’s potential to modulate activity in these regions.

Could it be used to make predictions about how such images would activate or deactivate brain regions?

While the effect of our synthesized images must be verified in future work by presenting them to new subjects in an MRI scanner, the observation of a clear pattern across ROIs in Figure 3 (B), as well as the use of two different encoders (namely, DINO-ViT and fwRF) that are not based on CLIP, provides robust support for BrainACTIV’s applicability in such online experimental settings. We have also updated Figure 9 to display the predictive performance of these three encoders, in order to provide further context to the predictions.

Discussion about minimizing categories

How do we understand the minimizing categories? Is it just a factor of having to step away from the preferred categories or is there truly a reduction in activity from baseline?

Here, minimization should be understood as the ROI’s level of activity relative to the reference image. Compared to the no-stimulation baseline, category-selective ROIs typically respond positively to all images (including non-preferred categories; see Fig 3 and 4 in [a] for examples), but much more strongly to the preferred category. In our analysis, we purposely chose reference images from NSD that yielded average activations for each ROI, so lying somewhere between the minimum and maximum for that ROI. BrainACTIV then exploits this range in activity to push the reference towards the tails of the distribution of (positive) activations for that ROI . Hence, our minimization should not be interpreted as ‘deactivating’ or suppressing fMRI activations, but rather as leveraging the full range of activation rather than only considering the top-activating images, as is typically done in MEI studies. We have updated our phrasing in the Abstract, and main text to clarify this.

Is this supported in the literature?

In terms of the observed results with minimization, these are broadly consistent with the known literature (e.g. face-selective regions respond the least to places and vice-versa), but some of the minimization effects we observe are, to our knowledge, not yet documented in the literature and could yield new hypotheses about these regions that can be tested in future studies. We have updated the first paragraph of Section 4.3 to explicitly mention this in the main text.

---

• [a] Groen IIA, Silson EH, Pitcher D, Baker CI. (2021). Theta-burst TMS of lateral occipital cortex reduces BOLD responses across category-selective areas in ventral temporal cortex. NeuroImage (230) 117790 https://doi.org/10.1016/j.neuroimage.2021.117790