NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction

We sincerely appreciate your careful review. In particular, you have provided us with more than twenty constructive suggestions and insightful questions, which is extremely precious at a time when the quality of reviews in the community is deteriorating. We value each of your suggestions and provide the following responses:

Weakness 1: Details of Multi-fMRI Fusion

We acknowledge and accept your suggestion. Considering that Multi-fMRI Fusion is merely an extra merit of NeuroClips and the page limitations, we were unable to present more details regarding multi-fMRI fusion. Thanks to the design of keyframes in the Semantic Reconstructor (SR), Multi-fMRI Fusion can be effectively achieved. In this study, we obtain the CLIP representations of reconstructed neighboring keyframes and train a shallow MLP based on the representations to distinguish whether two frames share the same class. The exact process is also shown as the GIF in the code repository. This training process operates at the image level. Despite our efforts to assess whether neighboring fMRI frames belong to the same scene class at the fMRI frame level using an MLP with fMRI frames as inputs, aiming for fMRI anomaly detection, our results were suboptimal, as illustrated in the following table. As can be seen, the fMRI level analysis frequently led to excessive false fusions. Conversely, established techniques for categorizing images perform reliably, particularly within the proficient CLIP space we employ. We promise to include more technical details in the final version. Thank you for your suggestion.

Weakness 2: Rapid Scene Change

We will present four perspectives on the inability of NeuroClips to perceive rapid changes in the scene

• Regarding Dataset. The rapid scene change depicted in the code repository is a distinct alteration influenced by human intervention, diverging from authentic real-world visuals. In the CC2017 dataset, video sequences were randomly divided into discrete segments, then amalgamated and presented to subjects. Such spliced videos are seldom encountered in everyday visual encounters, as scenes typically unfold continuously. This is what we have mentioned in Limitation about cross-scene fMRI.

• From the fMRI Data. We must acknowledge that continuous scene transitions occur in actual visual perception, such as when a person turns their head. However, it remains unknown whether and how rapid scene changes occurring within an fMRI frame (e.g., 2 seconds) are reflected in the fMRI signal due to its intricate nature. Consequently, decoding scene changes from fMRI data poses a significant challenge.

• From Keyframe Design. When scene changes occur within one fMRI frame (e.g., 2s), the design of the keyframes may adequately capture it, although the blurred video would be continuous. However, the keyframe-based method to guide the generation of continuous video can serve as a foundation for the subsequent design of decoding two consecutive scenes separated rapid scene change.

• From Cross-scene Diffusion Models. Presently, most of the current video generation diffusion models are not capable of very silky smooth scene switching, like SORA, to the best of our knowledge. We believe that, from a technical point of view, the issue can be better alleviated if consecutive semantics can be decoded from a single fMRI frame, or if the temporal resolution of fMRI can be refined.

Thanks again for your suggestions. We will provide the results of some of our exploratory experiments and a detailed discussion in the final version.

Weakness 3: Generalization Capability

Diverse real-world video content. The videos in the CC2017 dataset used are, somehow, sufficiently reflective of real-world visual experiences. As the original dataset paper states 'All video clips were chosen from Videoblocks and YouTube to be diverse yet. For example, individual video clips showed people in action, moving animals, nature scenes, outdoor or indoor scenes, etc' [1].

Diversity of fMRI recordings. To ensure consistency with the baselines [2] and to make a fair comparison, we experimented on the dataset, which unfortunately has only 3 subjects. We appreciate you pointing out that this should at least be mentioned in Limitations, and we will add it in the final version. Publicly available datasets for fMRI-to-video reconstruction are valuable and not easy to find. Until now, we discovered that the Algonauts 2021 [3] dataset can also be used for video reconstruction, but it is a great pity that this dataset is currently in an unpublished stage. To show the generalization capability of our approach, we finally chose to perform fMRI-to-image reconstruction on the Natural Scenes Dataset (NSD) [4] to assess the keyframe effect of our Semantic Reconstructor (SR). The visual results of the reconstructed image and groundtruth images are shown in the PDF appendix. Notably, our method exhibited satisfactory reconstruction outcomes even when applied to fMRI data with a distinct distribution, signifying the generalization capabilities of NeuroClips.

Reference

[1] Neural encoding and decoding with deep learning for dynamic natural vision, Cerebral cortex 2018

[2] Cinematic mindscapes: High-quality video reconstruction from brain activity, NeurIPS 2023

[3] The algonauts project 2021 challenge: How the human brain makes sense of a world in motion, 2021

[4] A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence, Nature neuroscience 2022

Weakness 4: Neurobiological Justification

Thank you for sharing your expert advice from neuroscience.

In numerous studies within cognitive neuroscience and computational neuroscience, researchers have delved into the mechanisms related to the brain's visual information processing and memory functions. Building on these studies, we innovatively propose using keyframes to guide our research, addressing the issue of frame rate mismatch between visual stimuli and fMRI signals, and enhancing the model's accuracy in fMRI-to-Video Reconstruction.

Specifically, [1] demonstrate that 'key-frames' play a crucial role in how the human brain recalls and connects relevant memories with unfolding events. In [2], a novel video abstraction paradigm which use the brain response reflected by fMRI to guide the extraction of visually informative segments from videos was proposed to quantitatively reveal the attentional engagement of human brain in the comprehension of video. In [3], the key frames in a video clip were used to extract these features, with the combined features from all keyframes representing the entire video clip. [4] provided a framework and explanation for video summarization.

We will revise and update our text clearance and references in the final version.

Reference

[1] Brain mechanisms underlying cue-based memorizing during free viewing of movie Memento, NeuroImage 2018

[2] Video abstraction based on fMRI-driven visual attention model, Information sciences 2014

[3] Bridging the semantic gap via functional brain imaging, IEEE Transactions on Multimedia 2011

[4] A comprehensive survey and mathematical insights towards video summarization, Journal of Visual Communication and Image Representation 2022

Question 8: More Metrics

The two new metrics ST-SSIM and ST-PSNR you mentioned are very interesting and enlightening, and we have carried out comparative experiments with MinD-Video in terms of the two metrics. We also evaluated NeuroClips with Visual Information Fidelity (VIF) metrics, and the results are shown in the table below (All results are averages of 3 subjects). Notably, NeuroClips' performance at the ST-level far exceeds that of MinD-Video, and even more than pixel-level, which further indicates that NeuroClips has a smoother video reconstruction capability.

As commonly used evaluation metrics in the field of video generation, Fréchet Video Distance (FVD) is more often used to assess the performance of video diffusion generation models. However, since we freeze the pre-trained parameters of the advanced generation model Animatediff [1], the effect of our video generation model must be better than the previous video models. Therefore, the evaluations on these video metrics may not be a fair one. Instead, we consider the CLIP rerepsentations for evaluation. Note that the consistency in CLIP representation space is more revealing of the degree of semantic consistency, reflecting the superiority of Semantic Reconstructor (SR) in NeuroClips. In the table above, we conducted an evaluation using the Visual Information Fidelity metric (VIF), instead of the FVD metric. When assessing FVD, all of Pytorch's open-source methods necessitate a video frame rate exceeding 10fps due to their need for some level of downsampling. Despite NeuroClips making significant advancements in frame rate, it falls to meet this requirement.

[1] Animatediff: Animate your personalized text-to-image diffusion models without specific tuning, ICLR 2024

Question 9: Difference Voxel Weight Visualization

We will add a third row with the difference between the weights of subjects as described in Question 3, thanks again!

Question 10: Voxel Selcetion

In fact, we did not select voxels specifically for the visual cortex. We calculated the voxel-wise correlation between the fMRI voxel signals of each training movie repetition for each subject. The significant voxels(Bonferroni correction, P < 0.05) were considered to be stimulus-activated voxels and used for subsequent analysis, as described in the pre-processing paragraphs of Section 4.1. We agree with you that other brain regions may also contribute to video decoding as well, so we expanded the range of significance (MinD-Video using P < 0.01), and the voxels used in NeuroClips are more than MinD-Video. The following table shows the number of voxels selected by the two methods.

We believe that our voxel-selective paradigm is more easily migrated to other fMRI decoding tasks and provides a more comprehensive neurobiological perspective.

Question 11: Trade-off Between PR & SR

Due to the parallel design of our Semantic Reconstructor (SR) and Perception Reconstructor (PR), SR focuses more on the decoding of video semantics, while PR is more geared towards the reconstruction of pixel-level information. SR can significantly improve semantic-related metrics, and PR can improve pixel-level related metrics. So in the end, when the two are combined together, there is a trade-off between semantic and perception reconstruction. During training, the Video Diffusion model will achieve a compromise effect between semantic and perception reconstruction. We will provide more discussions and deeper insighs on the compromise effect.

Question 12: Low-level Visual Details

Thanks for your suggestion. We willl modify it to 'MinD-Video lacks design of low-level visual detailing, so it significantly diverges from the brain’s visual system, exhibiting limitations in perceiving continuous low-level visual details.'

Question 13: Persistence of Vision

Thanks for your suggestion. Regarding the persistence of vision, we will careful check the references and discussions and supplement more supporting references outlined below.

Reference

[1] Ultra-High Temporal Resolution Visual Reconstruction From a Fovea-Like Spike Camera via Spiking Neuron Model, TPAMI 2023

[2] CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process, Arxiv 2024

[3] Persistence Of Vision Display-A Review, IOSR-JEEE e-ISSN 2015

[3] POV: Persistence of Vision: International Journal of Ethics in Engineering & Management Education

[4] Persistence of vision: the interplay of vision, Vision, Memory and Media 2010

Minor 1

We would modify this sentence to "The alleviation of these limitations will require joint advances in multiple areas and significant further effort will be required." Thank you.

Minor 2

fMRI measures brain activity by detecting changes in blood flow, and can thus reflect and quantify brain activity evoked by visual stimuli which has been widely applied in studies. The following are relevant references:

Reference

[1] Reconstructing perceived images from human brain activities with Bayesian deep multiview learning, TNNLS 2019

[2] Survey of encoding and decoding of visual stimulus via FMRI: an image analysis perspective, Brain imaging and behavior 2023

[3] Compressive spatial summation in human visual cortex, Journal of Neurophysiology 2013

[4] fMRI evidence for areas that process surface gloss in the human visual cortex, Vision research 2015

[5] A comparison of fMRI adaptation and multivariate pattern classification analysis in visual cortex, Neuroimage 2010

[6] Spontaneous activity associated with primary visual cortex: a resting-state FMRI study, Cerebral cortex 2008

[7] The human visual cortex, Annual review of neuroscience 2004

Minor 3

Thanks for your advice! We will replace ‘donated’ with ‘described’ in the final version.

Minor 4

In the context of the current diffusion model and attention-based transformer model as the dominant models for image generation, the computational overhead required for image generation models is sufficiently large. The content of video grows linearly with the number of frames, so the technical field of long video generation is still immature.

We value your opinion and also believe that a clearer explanation is needed here. Therefore, we will revise the above statement to make it more understandable.

Minor 5

Thank you for your professional suggestion. We decide to change the font in the Figure 4 to 'Times New Roman' in the final version.

Minor 6

Thanks for your kind reminder. Upon careful consideration, we acknowledge that using 'prove' is indeed inappropriate since the paragraph is an exposition of the results of the experiment. We agree with your suggestion and will change the sentence to "This may indicate that there were differences in the understanding of the video between subjects."

Minor 7

The classifiers we use are frozen pre-trained classifiers, so the total number of categories is fixed independent of the CC2017 dataset. The image classifier is an ImageNet classifier, pre-trained on ImageNet-1K[1] hence 1000 image classes. The video classifier based on VideoMAE [2] is trained on Kinetics-400 [3], an annotated video dataset with 400 classes, including motions, human interactions, etc.

Reference

[1] Imagenet: A large-scale hierarchical image database, CVPR 2009

[2] Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training, NeurIPS 2022

[3] The kinetics human action video dataset, ArXiv 2017

Minor 8

Thank you for your valuable addition. We will carefully review this literature and include it as a justification for using the keyframe approach in the final version.

 

---

We would like to extend our heartfelt gratitude to you! Our sincerest thanks! Your insights and suggestions from a neuroscience perspective have provided us with numerous inspiring ideas, greatly benefiting not only our current work but also our future research endeavors. Additionally, your detailed feedback has made our submission more solid and complete. We hope our response adequately addresses your questions, and we eagerly look forward to further communication and discussions.

Once again, we sincerely express our deepest gratitude and highest respect for your effort and time!

Best wishes,

All authors of Submission 351

Question 1: Hemodynamic Delay

Indeed, a considerable number of exploratory experiments have been conducted on the topic of hemodynamic delay, which has also been considered in Mind-Video. In the current version of NeuroClips, which employs a fixed 4-second delay, it was observed that the semantics of some of the generated keyframes exhibited latency, particularly in instances where the video clips of the scene were of a greater duration. Accordingly, a sliding window comprising two or three fMRI frames was devised to actively learn the aforementioned delay. However, it was discovered that employing a sliding window resulted in a notable reduction in the final evaluation metrics, with a more pronounced negative impact, particularly in the case of shorter video clips. It may be the case that longer videos have a more enduring effect on human brain fMRI signals. In light of the experimental outcomes, we ultimately opted to discard this methodology and instead fix the delay.

Question 2: Ridge Regression

As you mentioned, the human brain processes information in a highly complex and non-linear way. However, empirical evidence [1, 2, 3] underscores the effectiveness and sufficiency of linear mapping for achieving desirable reconstruction outcomes. Notably, complex nonlinear models will easily overfit to fMRI noise, leading to poor performance in the test set [4]. We will add more discussion in the Method Section.

Reference

[1] Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors, NeurIPS 2023

[2] High-resolution image reconstruction with latent diffusion models from human brain activity, CVPR 2023

[3] MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data, ICML 2024

[4] Through their eyes: multi-subject Brain Decoding with simple alignment techniques, Imaging Neuroscience 2024

Question 3: Voxel Weight Visualization

In Figure 12, each column represents the PR and SR visualization weights for the same subject. The voxel distribution is the same in each column since the voxels initially selected by each subject are fixed. For each subgraph, the weights of the voxels were visualized after normalization, with brighter regions representing higher weights and darker regions representing lower weights. For subject 3, the distribution of voxel weights in PR and SR is quite different. For example, in the right region of V4 of the right brain, PR is brighter and SR is darker. We appreciate your suggestion to add another line to visualize and highlight the difference between PR and SR. In this way, we can better show the difference in voxel weights between the two modules, and we will add it to the final Supplement.

Question 4: Larger Example Images

We will modify it to a vector figure and adjust the number of images to make it clearer.

Question 5: Major Limitations

Since NeurIPS allows an additional page to be added to the final version, we will move Limitation from the appendix to the main body. Thanks for your suggestion.

Question 6: Visualization Software

We use Connectome Workbench from Human Connectome Project (HCP), and flatmap templates were selected as 'Q1-Q6_R440.L.flat.32k_fs_LR.surf.gii' and Q1-Q6_R440.R.flat.32k_fs_LR.surf.gii' . Additionally, the cortical parcellation was manually delineated by neuroimaging specialists and neurologists, and aligned with the public templates in FreeSurfer software with verification. We normalised the voxel weights, scaling them to between 0 and 1. Finally, to show a better comparison, the Colorbar was chosen to be 0.25-0.75. We will add this note to the final Supplement section.

Question 7: Metric Improvement

We'll add extral analysis to indicate the improvement values on the metrics.

We would like to thank you for taking the time to review our work. We value each of your suggestions and provide the following responses:

Weakness 1: Light Variations

We highlight that light variations are also a feature that distinguishes video from images. Our initial hypothesis was that this phenomenon was caused by the presence of partial light variations in the blurred video of the Perception Reconstructor (PR). However, subsequent experiments involving the removal of the blurred video revealed that the effect persisted. This may be caused by the pre-training of AnimateDiff, where a corresponding discussion will be further explored in the final version of the paper. It is still important to emphasise that although the phenomenon exists, it is only observed in a limited number of videos.

Weakness 2: Longer and High-frame Videos

• From the Dataset. After counting the video clips in our dataset, we discovered that the longest video spanned merely around 10 seconds. Hence, there is no need to reconstruct videos that extend up to a minute in duration.

• From an availability standpoint. Currently, the most cutting-edge video generation technology has now advanced to create longer videos, like Sora. However, the quality of the generated content still raises concerns due to the current limitations of the technology. Additionally, it's important to highlight that the majority of these technologies are not open source.

• From a Research Perspective. fMRI-to-video reconstruction is an emerging field. At this critical stage, we believe that the applicability and scalability of innovative methods are of greater consequence. Empirically, once longer blurred videos or video generation tools are available, NeuroClips can generate 24-frame and longer videos. However, this will remarkably increase the GPU usage. From a research standpoint, we believe that the current reconstructed video is impressive and sufficient.

Question 1: Hemodynamic Delay

Indeed, a considerable number of exploratory experiments have been conducted to study hemodynamic delay, a topic also considered in Mind-Video. In the current version of NeuroClips, with a fixed 4-second delay, it was observed that the semantics of some of the generated keyframes exhibited latency, particularly in instances where the video clips of the scene were of a greater duration. To address this, a sliding window comprising two or three fMRI frames was devised for the purpose of actively learning the aforementioned delay. It was discovered that the application of a sliding window resulted in a notable reduction in the final evaluation metrics, with a more pronounced negative impact, particularly in the case of shorter video clips. It may be the case that longer videos have a more enduring effect on human brain fMRI signals. In light of the experimental outcomes, we ultimately opted to discard this approach and instead maintain a fixed delay.

Question 2: Cross-Subject

Yes, it is necessary to train a distinct model for each subject when using NeuroClips. As cross-subject approaches in this area still fail to achieve satisfactory results, we finally choose to utilise and explore a single-subject model in this paper. However, in light of the recent advancements in fMRI-to-image reconstruction, a series of cross-subject models have emerged [1, 2, 3]. We believe that extending NeuroClips to encompass cross-subject content will be a promising avenue for future research.

Reference

[1] Mindbridge: A cross-subject brain decoding framework, CVPR 2024.

[2] Psychometry: An omnifit model for image reconstruction from human brain activity, CVPR 2024.

[3] MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data, ICML 2024

Question 3: Low Quality of Dataset

Since both NeuroClips and the previous baselines are founded upon the same dataset, the ensuing comparison is deemed to be equitable. The assertion that the CC2017 dataset is of poor quality is intended to highlight the difficulty in achieving impressive fMRI-to-video reconstruction results in comparison to fMRI-to-image reconstruction. The capacity to produce superior outcomes on datasets of inferior quality serves to illustrate the resilience of the NeuroClips' method. It would be of interest to ascertain whether NeuroClips could achieve even more impressive results with a higher-quality dataset.

Question 4: Enhancement from Text Modality

Considering that the diffusion prior loss is rooted in MSE at the representation level, this prior is inherently unstable and the semantic information is susceptible to bias. It is important to recognise that text modality possesses its own distinctive characteristics and robust semantic support, which serve to complement the image representation space. To enrich the semantic depth of the representation, we strategically place text assistance subsequent to the prior.

 

---

You have offered many constructive and valuable suggestions, making our submission more solid and complete. Once again, we sincerely express our best gratitude for your effort and time!

Best wishes,

All authors of Submission 351

We would like to thank you for taking the time to review our work. Your effort has ensured that our submission received adequate attention and review. We address the questions and clarify the issues accordingly as described below.

Weakness 1: NeuroClips Fails to Accurately Follow the Ground Truth Movement

• Thanks for pointing out your concern. The fMRI-to-video decoding task presents significant challenges related to both high-level semantic comprehension and low-level motion perception. On the one hand, fMRI data is characterized by its high dimensionality, with hundreds of thousands of voxel signals captured across the entire brain. Even after rigorous voxel selection processes, approximately 10,000 voxels remain. On the other hand, fMRI data is heavily influenced by human behavior in response to visual stimuli, resulting in an extremely low signal-to-noise ratio.

• The aforementioned hurdles complicate the decoding of precise and comprehensive ground truth movement from fMRI data in fMRI-to-video reconstruction. Nevertheless, through this endeavor, we present elaborated designed components and loss functions tailored for motion perception. These refinements enable NeuroClips to efficiently capture motion details and show impressive video reconstruction results. It is a valuable and motivating contribution to the fMRI decoding community. Further elaboration on these advancements is presented in the following responses.

Question 1: No Model or Loss Function for Movement

NeuroClips is equipped with a specific model design and a tailored loss function, which guarantees its ability in capturing motion information.

• Regarding Model Design. The Perception Reconstructor (PR) module we designed can perceive movement (motion) information. In this work, we focus on capturing generic motion within videos by modeling the motion information from two perspectives: perceiving the structured information within frames (pertaining to objects' shape, position, and orientation) and grasping the dynamic information across frame sequences (related to the movements of an object or the dynamics of the scene). To accomplish this, we introduce a spatio-temporal attention module within Temporal Upsampling, which is explicitly designed to decode spatiotemporal (structural-dynamic) details, i.e., motion cues, from fMRI data. We utilize the cues to guide the Video Diffusion model toward a more nuanced perception of motion dynamics, ultimately guaranteeing the motion-awareness of the video generation of NeuroClips. This is our meticulous design from the model perspective for capturing movement (motion) information.

• Regarding Loss Function. The Mean Absolute Error (MAE) part within the loss function (Eq. 1) quantifies the disparity between the generated video and the groundtruth video frame-by-frame, facilitating the perception of generic motion within videos. Note that the perception embeddings of video frames, denoted as $\mathbf{E}_\mathcal{X}$, are aligned to the latent space of the Stable Diffusion Variational Autoencoder, which can be the equivalent of the pixel space at the frame level. Numerous recent video generation models have shown that the effective perception of generic motion can be achieved through two fundamental yet straightforward designs: temporal attention mechanisms and frame-level loss functions [1, 2]. Consequently, the design inspiration is embraced in this study.

• Regarding Experimental Evidences. In Figure 3, the turtle's swimming direction, the airplane's orientation and flight direction, and the motorcycle rider's posture, along with their corresponding motion details, are accurately reproduced. If NeuroClips be exclusively crafted at a semantic level, it becomes evident that semantics alone are inadequate for capturing these motions. In addition, the exceptional accuracy of NeuroClips, as evidenced by the pixel-level and ST-level metrics in Table 1, further corroborates these visualization outcomes.

• Regarding Fine-grained Motion Decoding. Currently, decoding fine-grained movements from fMRI data poses a significant challenge due to its intricate and noisy nature. Unlike text, fMRI data lacks straightforward cues such as motion-descriptive words, making the motion perception more complex. NeuroClips stands apart from previous text-to-video models that are capable of generating motions closely aligning with textual motion semantics. Note that NeuroClips has achieved significant progress in fMRI-to-video decoding tasks. Certainly, NeuroClips is by no means the final solution for fMRI-to-video decoding tasks; undoubtedly, superior solutions will emerge in the future. Looking ahead, there are promising prospects for enhancing the accuracy of decoding fine-grained movements from fMRI data in the future. This will be the focal point of our upcoming efforts.

Question 2: Code Release

We promise to release the code at the earliest opportunity, following the acceptance of the paper. We appreciate and support open source because it helps more researchers contribute to the field and advances its development. This is an effective way to further enhance the significance and value of our research.

Reference

[1] Stable video diffusion: Scaling latent video diffusion models to large datasets, Stability AI

[2] Align your latents: High-resolution video synthesis with latent diffusion models, CVPR 2023

 

---

You have offered many constructive and valuable suggestions, making our submission more solid and complete. Once again, we sincerely express our best gratitude for your effort and time!

Best wishes,

All authors of Submission 351