Rethinking Brain-to-Image Reconstruction: What Should We Decode from fMRI Signals?

Given the substantial question of information limits in fMRI recordings, there are several critical components to consider: 1) Since subjects are center-fixated, the early stages of visual processing likely represent a pseudo-foveated version of the image—an incomplete view of the full stimulus; 2) The BOLD signal is an indirect measure of neural activity, contributing to further information loss; 3) The spatial resolution of fMRI (~1,000 neurons per voxel at 7T) imposes additional constraints on the reconstruction ceiling.

Ideally, each of these limitations would be independently assessed, with primary attention to factors like signal granularity and voxel resolution (2-3), which likely dominate information loss. Instead, the authors focus on center-fixation, the factor that may be least influential here, without robustly measuring this impact given current recording limitations. This raises concerns about whether the presented findings are genuinely due to the “limited capacity of perceptual experience and cognitive mechanism” mentioned as a key premise.

A second major issue is the choice of saliency-based foveation. Since NSD subjects were reportedly center-fixating, any resulting neural representation should align with a center-foveated version of the stimulus. The use of artificial predictors to estimate general saliency—creating varying foveated versions—seems scientifically unsupported unless subjects were viewing freely. This is a fundamental issue, as it challenges the entire premise of the foveated reconstruction approach in this context. Whether or not this potentially confers some gain to reconstruction quality by a mysteriously emerging optimization/regularization tweak, is another topic, which I address below.

Additional Points for Improvement:

• The paper places excessive emphasis on generating foveated images, with minimal grounding in the literature on this approach’s scientific merit for fMRI reconstruction.
• The review of prior work is sparse, missing both past and recent studies while presenting weakly related works in detail.
• Evaluations are based solely on a single dataset, limiting the generalizability of findings.
• [67-86] Unclear.
• [86-91] BrainScore, typically used for model evaluation, seems unrelated to image reconstruction—its relevance here is unclear.
• [94-97] “Rethinking” without elaboration lacks contribution.
• [101-103] The reference to gaze behavior and time intervals appears abruptly, lacking context.
• [105-108] Unclear use of BrainScore and for a single fMRI dataset, neither fully justified and not comprehensive.
• The brain-score metric, although marked as the fourth contribution, is disproportionately highlighted across figures and tables.
• Table 5, intended as a comparison with prior works, shows only marginal support for the proposed approach, weakening the overall case.

1. The motivation for using TempSAL to generate fixation points is unclear, especially considering that eye-tracking information is already available in the NSD dataset. If one of the goals of this approach is for temporal saliency prediction to serve as a substitute for eye-tracking data, it would be valuable to include an analysis comparing TempSAL’s predictions with the actual gaze positions of subjects. This comparison could help clarify the accuracy and relevance of using TempSAL in place of direct eye-tracking data.

2. The authors' focus on a single-subject pipeline would benefit from a comparison with state-of-the-art methods. Notably, MindEye2 (Scotti et al., 2024 Table 6) has demonstrated superior performance over MindEye1 in single-subject scenarios. A comparative analysis with MindEye2 is essential to validate the proposed method's effectiveness. Additionally, exploring the application of this approach within multi-subject pipelines could provide valuable insights into its broader applicability.

3. The brain has evolved to operate in a ‘foveated’ manner, and the authors have effectively demonstrated a higher similarity between pseudo-foveated image embeddings and fMRI signals. However, since we cannot be certain that the generated fixation point for a given image matches the subject's actual focus during the scanning session, how can we be sure that the increased correlation is not simply due to the inherently higher SNR of foveated image embeddings compared to full-image embeddings? Concretely, foveated images naturally contain fewer peripheral details, which reduces noise in the embeddings. This leads us to expect that the model would produce cleaner, less noisy embeddings, as it does not need to encode as much extraneous peripheral information. As a result, the model’s latent representations are more concentrated on relevant features, which could increase the overall signal-to-noise ratio in the embeddings.

4. Line 101, 421: typo

1. Why choose ridge regression and MLP to encode fMRI data instead of CNN? Convolution of images is a state-of-the-art method.
2. What is the low level in Table 3 and Table 4?
3. What is the meaning of ‘two-way identification based on the output embeddings of AlexNet’?
4. Although the article shows some details of the experiments, there are still some that have been ignored, such as the optimizer and the encoder structure when calculating the brain score.
5. It would be better to show on which datasets the pre-trained methods such as the diffusion model were pre-trained.
6. What is the meaning of '-' in Table 5? Can't it be calculated? Some words are written inconsistently, such as trained and pre-trained.

• My major concern is about whether decoding success with foveated images could be attributed to their reduced information content, making it inherently easier to match patterns in brain signals. To address this, a useful control would be to compare foveated images with a set of matched, low-pass filtered versions of the natural images that have similar information content but lack the specific spatial emphasis of foveation.
• Further, the difference between foveated and normal images appears only marginal based on the proposed brain score metric. Error bars would be helpful here. Why not use standard encoders commonly used in brainscore style studies to compare natural vs foveated images?
• Also, the rationale behind the third step in the pipeline (the stimulus generation phase) is unclear - is it used only to compare against existing methods? The original idea in the paper that brain signals would not be able to reconstruct inputs at high fidelity due to the foveation conflicts with this phase