SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning

W1. Thank you for the thoughtful suggestion. We conducted 1-hour novel-subject adaptation experiments using multi-subject pretraining (e.g., Sub125→Sub7, Sub257→Sub1) and observed modest performance gains compared to single-subject pretraining (e.g., Sub1→Sub7, Sub2→Sub1), as shown below.

However, these improvements required deeper Transformer architectures (8→12 layers) and extended training (50k→200k steps), substantially increasing computational cost. In contrast, single-subject pretraining followed by 1-hour adaptation offers a more efficient trade-off between performance and training resources.

W2. Thank you for the suggestion. Taking MindSimulator as a cross-subject adaptation baseline is limited by two practical constraints. i) MindSimulator is not open-sourced and incorporates several hand-crafted components (e.g., resting-state initialization, custom noise injection), making it difficult to reproduce; ii) MindSimulator’s Autoencoder is subject-specific and tightly coupled to fixed voxel dimensions, rendering it incompatible with cross-subject scenarios where ROI sizes vary. These limitations prevent a direct and meaningful comparison between SynBrain and MindSimulator in the context of cross-subject adaptation.

To establish a transparent and reliable baseline, we implemented Linear Regression, the most classical encoding model, using a simple two-layer linear architecture. The first layer maps CLIP features to a 4,096-dimensional hidden space, and the second layer projects to subject-specific voxel responses. During few-shot adaptation, we freeze the first layer and train only the second layer using 1-hour data from the target subject. As shown in the results below, SynBrain consistently outperforms Linear Regression across voxel-wise correlation, semantic alignment, and image retrieval accuracy, demonstrating strong generalization and cross-subject transferability under data-limited scenarios.

W3. Thank you for the insightful question. We conducted additional experiments with extended synthetic data durations (up to 32 hours) and observed that the optimal performance is achieved with just 1 hour of data augmentation. As shown below, adding more synthetic data beyond this point leads to diminishing or even negative returns.

This suggests a trade-off between real and generated data: while a small amount of high-quality synthetic data is beneficial, excessive augmentation may introduce distributional mismatch or redundancy, limiting further gains.

Q1. We thank the reviewer for this insightful question. While the S2N Mapper adopts a deterministic mapping from CLIP features to BrainVAE’s latent space, this design does not contradict the biological one-to-many nature of visual-to-neural mappings. In the first stage, BrainVAE is trained to capture trial-level variability by associating each visual input with a latent distribution, whose center captures the most salient and semantically consistent neural features across trials. By conditioning on this center and performing stochastic sampling, BrainVAE can generate multiple plausible fMRI responses that are semantically aligned yet voxel-wise diverse, faithfully reflecting the one-to-many nature of neural encoding.

In the second stage, the S2N Mapper is optimized to align each CLIP embedding with this semantic anchor—the center of the latent distribution—rather than with any specific trial or empirical mean. Its objective is to map visual features to the most representative position in latent space, while delegating the modeling of trial-level variability to the stochastic sampling process. This design ensures that variability is not averaged out, but reintroduced in a controlled, biologically grounded manner.

As shown in Supplementary Figure 2, repeated synthesis from the same visual input yields fMRI signals that are voxel-wise diverse but semantically stable. This confirms that our framework supports semantic-consistent sampling and generation, effectively balancing biological realism with representational precision.

We thank the reviewer again for the valuable feedback. We hope our response adequately addresses your comments. If so, we hope you could consider a more positive evaluation of our work. If there are any remaining concerns, we would be happy to further address them during the discussion phase.

W-Clarity1. Thanks for the feedback. We will clarify the architectural details of BrainVAE in the revised manuscript. While the code cannot be shared during rebuttal for anonymity, it will be released upon acceptance to ensure reproducibility.

W-Quality2. Thank you for the suggestion. Note that Predicting Human Brain States with Transformer (Sun et al., 2024) focuses on resting-state fMRI and aims to predict future brain states based on past activity. As such, it does not fall within the scope of visual-to-fMRI encoding and is not directly comparable to our setting. Hence, we have included additional comparisons with two representative baselines: (1) GNet (St-Yves et al., 2023), and (2) Linear Regression, a classical deterministic voxel-wise visual encoding model, using a simple two-layer linear architecture with a 4,096-dimensional hidden space.

As shown below, SynBrain achieves consistently superior results across multiple levels, highlighting its effectiveness in fMRI synthesis with both voxel-level structure and semantic consistency.

Q1.

Training Efficiency. The faster training of BrainVAE is mainly due to its rapid convergence within fewer epochs. Under the same total batch size of 24, BrainVAE completes 15 epochs in ~2 hours (4×A100-40G), while MLP-AE requires 50 epochs and ~5 hours (1×A100-40G), yet still underperforms BrainVAE across key performance metrics. Note that under the same total batch size, multi-GPU training introduces extra overhead due to gradient synchronization across devices, resulting in a slightly longer per-epoch time compared to single-GPU training. Nonetheless, BrainVAE significantly shortens the overall training time by converging within far fewer epochs.

Hardware Flexibility. While the results reported in the main paper use 4×A100-40G GPUs (with downsampling blocks = 1, latent dim = 4096), we also experimented with lighter variants of BrainVAE, as reported in Supplementary Table 1. For example, setting the number of downsampling blocks to 3 (latent dim = 1024) enables faster training on a single A100-40G GPU, with only minor performance degradation (-1.5% in Incep and -1.4% in CLIP). Even so, it still significantly outperforms MindSimulator (5-Trial) by 2.4% (Incep) and 4.7% (CLIP), demonstrating the scalability of SynBrain to more constrained computational environments.

Inference Efficiency. The observed difference in inference efficiency is not inherently due to the variational approach itself, but rather stems from the design of architecture used in the Visual-to-fMRI synthesis process. Specifically, MindSimulator employs a multi-step diffusion-based method (Diffusion Transformer, DiT), while SynBrain uses a one-step S2N mapper for direct sampling. This leads to a substantial difference in inference time as shown below.

Q2. Semantic Consistency vs. Voxel-level Fidelity

Thanks for your comment. While BrainVAE is capable of modeling the noise characteristics inherent in neural signals through its latent space, it is not intended to faithfully reconstruct the full voxel-level complexity of raw fMRI signals. Instead, by integrating visual semantic constraints during training, the model prioritizes semantic consistency over voxel-wise fidelity, resulting in synthesized signals that retain relevant variability while suppressing irrelevant noise.

The improved retrieval accuracy reflects this design choice. Rather than contradicting the variational framework, it highlights the capability of BrainVAE to balance biologically plausible variability with task-relevant constraints, enabling effective sampling and generation aligned with perceptual semantics. As shown in Supplementary Figure 2, fMRI synthesis via multiple stochastic passes of BrainVAE exhibit voxel-level variability, yet remain semantically consistent across samples.

Q3. Vision Model Comparison:

Thanks for your thoughtful question. We evaluated the impact of visual encoder choice by replacing CLIP in SynBrain with two advance visual foundation models, DINOv2 and MAE. As shown in the table below, CLIP achieves the best overall performance across reconstruction, voxel-level correlation, and semantic alignment. We attribute this to its vision-language contrastive objective, which provides stronger semantic supervision compared to purely visual models like DINOv2 and MAE. Nonetheless, the relatively strong performance of these alternatives indicates that SynBrain remains robust across different encoder choices.

We also include a comparison with MindSimulator (5-Trial), which likewise uses CLIP as its visual encoder. Notably, MindSimulator averages results over five generated fMRI trials to improve performance, whereas SynBrain achieves superior results with just a single trial. This suggests that SynBrain benefits not only from a strong encoder but also from more effective model design and training strategy.

W1&Q4. Cross-dataset Adaptation

Thanks for the suggestions. To tackle this concern, we conducted extra experiments on the GOD [1] dataset, which differs from NSD in both scanning protocol and data quality. Each GOD subject contains only 1,200 training images collected in a single session and 50 out-of-class test images, offering a challenging setting for cross-dataset adaptation.

GOD subjects used the Visual ROI defined by MindVis (4,623 voxels), and for consistency, we employed a SynBrain variant trained on the NSD Visual ROI (4,657 voxels), instead of the standard NSDGeneral ROI (15,724 voxels, from MindEye). This variant features 3 downsampling blocks ($dim=512$) that enable single-GPU training.

We compared two conditions on GOD: (1) training SynBrain from scratch, and (2) adapting a SynBrain model pretrained on NSD (Sub1). As shown in the table below, the pretrained model substantially outperforms the from-scratch baseline across all metrics.

Note that, in this case, voxel-level correlations remain relatively low—primarily due to the poor SNR (Signal-Noise Ratio) of GOD (~0.1, as reported in MindVis [2]). However, the model achieves substantial gains at the semantic level. This suggests that under such noisy conditions, high voxel-wise correlation may reflect overfitting to noise rather than capturing meaningful neural structure. SynBrain, by contrast, prioritizes semantic alignment and achieves $68.0$% 50-way Top-1 retrieval accuracy using synthesized fMRI, outperforming Mind-Vis (pretrained on HCP), which reaches 23.9% with raw fMRI. These results highlight SynBrain’s effectiveness in transferring to small-scale, low-SNR datasets.

[1] Horikawa, T. & Kamitani, Y. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications 2017.

[2] Chen Z, et al. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. CVPR 2023.

Q5. Thank you for the valuable question. BrainVAE is designed to capture stimulus-driven, semantically relevant variability by modeling trial-level one-to-many mappings—that is, the distribution of neural responses elicited by repeated or similar visual stimuli. This allows the model to learn not just a single canonical response per stimulus, but a functionally consistent response space shaped by perceptual semantics. While the model can, in principle, capture broader sources of variability, subject-specific idiosyncrasies and spontaneous background activity are typically down-weighted, as they are less aligned with visual semantics.

Q6. Thank you for pointing this out. We acknowledge the terminology difference and will revise the description accordingly in the camera-ready version.

We thank the reviewer again for the valuable feedback. We hope our response adequately addresses your comments. If so, we hope you could consider a more positive evaluation of our work. If there are any remaining concerns, we would be happy to further address them during the discussion phase.

Q1. Yes, we used the “nsdgeneral” cortical region as fMRI input, following previous works like MindEye.

Q2. Thank you for the insightful question. In our experiments, the fMRI data generated by SynBrain was used as an additional augmentation strategy to further enhance the performance of MindEye2 in the 1-hour few-shot adaptation setting. It’s worth noting that MindEye2 already incorporates standard data augmentation techniques (i.e., image distortions and MixUp) during training. Our intention was not to compare SynBrain with traditional augmentation methods, but rather to build upon them. Specifically, we aimed to validate that SynBrain-generated fMRI can effectively complement real fMRI data, helping to alleviate the challenge of data scarcity and improve model generalization in low-resource scenarios.

Q3. Thank you for the thoughtful questions, and we apologize for the lack of clarity in the current version of Figure 3. To clarify, Figure 3 corresponds exclusively to Stage-1 (BrainVAE) in Figure 2, which focuses on training the BrainVAE module. The experiments in Figure 3 investigate the effect of replacing the encoder and decoder architecture within Stage-1. Specifically, we compare our proposed BrainVAE with two alternative designs—MLP-AE and MLP-VAE—while keeping the loss functions and training setup identical. As defined in Equation (4), all models are optimized using KL loss, MSE loss, and CLIP loss. The only difference lies in the encoder-decoder architecture.

Regarding the second question, all loss terms are used to evaluate the Stage-1 (BrainVAE) only. The MSE loss measures voxel-level reconstruction quality, while the CLIP loss evaluates the semantic consistency between the brain latent $z$ and the image embedding $z_{CLIP}$. Figure 3 (Right) shows the validation loss curves (MSE and CLIP) during Stage-1 training for each architecture. To further assess the quality of the brain latent space, we also conduct Brain-Image Retrieval between $z$ and $z_{CLIP}$ in Stage-1. Higher retrieval accuracy reflects stronger semantic alignment.

Finally, we emphasize that Figure 3 is entirely unrelated to Stage-2 (S2N Mapper) in Figure 2; the experiments and comparisons are solely focused on Stage-1 (BrainVAE).

Q4. Thank you for your thoughtful and detailed question. In our design, the KL loss and SoftCLIP loss are applied to the latent variable z with complementary purposes: the KL loss encourages smoothness and sampleability in the latent space, while the SoftCLIP loss promotes semantic alignment between brain and image modalities.

Importantly, we intentionally downweight the KL loss, allowing it to function primarily as a soft regularizer rather than strictly enforcing a standard Gaussian prior. This relaxed constraint gives the latent space flexibility to encode semantically aligned structure, while still preserving the stochastic sampling capability that underlies variational generation.

As shown in Table 3, removing either KL (Variation Sampling) or CLIP (Contrastive Learning) objective significantly degrades performance, underscoring their necessity. Furthermore, Supplementary Figure 2 demonstrates that their combination enables semantically consistent sampling from the latent space—achieving the one-to-many generation behavior that BrainVAE is designed to support.

This design choice is also supported by prior work such as VAVAE [1], which demonstrated the effectiveness of combining semantic constraints from visual encoders (e.g., CLIP, DINOv2) with variational modeling.

[1] Yao, J., Yang, B., & Wang, X. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. CVPR 2025 Oral.

Q5. Thank you for the insightful follow-up. As shown in Figure 3, both the MSE loss and CLIP loss decrease steadily over the first 15 epochs and then stabilize, indicating that the model converges in terms of both voxel-level reconstruction and semantic alignment. The KL loss follows a similar trajectory, converging within 15 epochs.

To balance the three objectives, we apply appropriate weighting: $\lambda_{KL}=0.001$, $\lambda_{CLIP} = 1000$. After weighting, the KL loss remains at the order of $10^1$, MSE loss decreases from $10^3$ to $10^2$ over training, and CLIP loss stays around the order of $10^3$. This setup ensures that each loss term contributes in a controlled and complementary manner throughout training. In particular, the CLIP loss plays a more influential role in the early stages, helping guide the latent space toward semantic alignment, while the KL loss provides soft regularization to support sampling.

Importantly, our goal is not for the latent vectors to strictly follow a standard Gaussian distribution, but to remain sufficiently regularized to support stochastic sampling. As further illustrated in Supplementary Figure 2, SynBrain can perform stochastic sampling in the latent space (via noise injection) and still generate semantically consistent fMRI signals. This capability reflects the core design principle of BrainVAE: to balance variational sampling with meaningful semantic structure, rather than enforce a rigid Gaussian prior.

We thank the reviewer again for the valuable feedback. We hope our response adequately addresses your comments. If so, we hope you could consider a more positive evaluation of our work. If there are any remaining concerns, we would be happy to further address them during the discussion phase.

W1. Thanks for your correction.

W2 & Q1. Thank you for the suggestion. While simulation-based evaluation is a promising direction, we chose to leave it to future work due to concerns about its biological validity. Most simulation methods rely on injecting noise or applying spatial transformations to existing fMRI data, but it remains unclear whether such approaches can faithfully replicate the complex, subject-specific variability observed in real neural responses.

As shown in Supplementary Figure 3, even when presented with the same visual stimulus, different subjects exhibit substantial differences in activation patterns. This raises concerns about whether current simulation techniques can meaningfully benefit cross-subject modeling without introducing unrealistic assumptions.

In line with the reviewer’s suggestion to evaluate on a more diverse population, we expanded our experiments to include all eight real subjects from the NSD dataset. Four subjects (Sub-1,2,5,7) were used in the main experiments, and the remaining four (Sub-3,4,6,8), who completed fewer sessions, were incorporated as supplementary evaluations to further assess SynBrain’s cross-subject few-shot adaptation capability.

As shown below, SynBrain achieves strong performance across these unseen individuals with only 1 hour of adaptation data, demonstrating robust generalization across real, subject-specific neural variability.

W3 & Q2. Thanks for your comment. We agree that some images generated from the synthesized fMRI exhibit noticeable distortions, particularly in background details. These artifacts primarily stem from the limited capacity of the fMRI-to-image decoder used in our pipeline—namely, an unrefined version of MindEye2 [1]. As shown in the original MindEye2 paper, even with ground-truth fMRI input, the reconstructed images can retain overall semantic content while exhibiting unnatural elements. This issue could be mitigated by replacing the decoder with a refined version of MindEye2 (i.e., by guiding the unrefined reconstruction through base SDXL with image-to-image generation), which offers improved perceptual quality and semantic alignment.

In our work, however, the decoder is used solely as a proxy to assess whether the synthesized fMRI preserves meaningful high-level representations. In fact, due to its own semantic decoding limitations, the reported semantic-level scores may underestimate the true alignment between the synthesized fMRI and the corresponding visual content.

[1] Scotti P S, Tripathy M, Villanueva C K T, et al. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. ICML 2024.

We thank the reviewer again for the valuable feedback. We hope our response adequately addresses your comments. If so, we hope you could consider a more positive evaluation of our work. If there are any remaining concerns, we would be happy to further address them during the discussion phase.