Brain-to-4D: 4D Generation from fMRI

1. Writing issues.

The authors exhibit significant issues in writing and expression, which considerably hinder the reading experience and further comprehension of their intentions.

（1）The authors did not provide a detailed explanation of the architecture shown in Figure 2, which makes it difficult to understand the model. The authors are requested to separately describe the training and inference processes of WSf4D. Additionally, they should clarify the origin of the “Fg” in Figure 2(a)—whether it was pre-trained by themselves or derived from an existing pre-trained model. Lastly, the authors are asked to explain the meaning of the dashed line above “Supervision” in Figure 2(a).

(2) From Section 4.2, it is evident that both Foreground 3D-aware diffusion and Background 3D-aware diffusion are pre-trained models from other works and are not contributions of this paper. Therefore, it raises the question of why the authors dedicate such extensive space from lines 239 to 269 to describe their loss function.

2. Experiment issues: （1）The authors have included too few baselines for comparison in Table 1, and under the front view setting, WSf4D does not demonstrate significant superiority over MinD-Video.

（2）The ablation experiments presented in this paper consist only of qualitative comparisons and lack quantitative analysis, which may hinder the validation of the effectiveness of the proposed components. For instance, in Figure 7, it is challenging to discern the improvements brought by the refinement stage through visual inspection alone. The authors are requested to include quantitative ablation experiments.

1. The lack of comparison with other baseline models is a limitation. Since your model encodes and generates the foreground and background separately, it would be possible to compare the generation quality of the foreground and background independently rather than combining them in a single comparison. Moreover, having only Mind-Video (Chen et al., 2024) as a comparison seems insufficient; additional fMRI-to-video models, such as Mind-Animator (Lu et al., 2024), could be included. Although these models generate only 2D videos, comparing the images from the frontal view could provide a partial assessment of generation quality, as you have done in Fig. 4 and Table 1.

2. Most of the components in this method have already been explored in previous works, and this paper primarily integrates them to generate 4D representations. For example, the concept of VQ-fMRI (Chen, Qi, & Pan, 2023) is no longer novel, and the authors have subsequently extended its application in MindArtist (Chen et al., 2024). Another key component is the representation-to-4D generation part, where the paper also mentions in lines 890-891 that it uses the framework of DreamGaussian4D (Ren et al., 2023).

3. I noticed that you directly use the original image to supervise background generation (Fig. 2), which leads to parts of the foreground being mistakenly recognized as background, resulting in an impure background generation (Fig. 6, right). It may be beneficial to preprocess the supervision data to remove foreground interference, which could better leverage the advantages of foreground-background decoupling.

1. Although the authors propose a new decoding method for brain-to-4D, the experimental validation appears to be weak. First, the fMRI dataset used in the experiments consists of 2D videos watched by the subjects, which do not effectively stimulate the brain's 3D representation information. Why did the authors choose to use fMRI evoked by 2D stimuli to reconstruct 3D information? Furthermore, since there is no ground truth for the 3D information, how can the quality of the 3D reconstruction be effectively evaluated?

2. From the reconstruction results in Figures 4 and 7, the reconstruction effect of WSf4D does not seem promising. For instance, in Figure 7, the color, posture, and size of the dog are poorly reconstructed. Under such circumstances, what is the significance of pursuing consistency across multiple viewpoints? Even if a high degree of consistency is presented across viewpoints, is this information decoded from the brain signals or is it prior information from the diffusion model?

3. The authors only used one fMRI dataset and only compared it with one method, MinD-Video, which is insufficient. In the field of video reconstruction, there are already multiple datasets and reconstruction methods available for comparison.

4. Since this paper is about 4D reconstruction, it is difficult to appraise the temporal coherence and multi-view consistency solely through image visualization. Is there an anonymous project homepage or link to display the relevant experimental results?

1 - The problem formulation lacks clarity, and the flow of the abstract could be improved.

2 - To clarify how the VQ-fMRI encoders address the challenge of distinguishing meaningful brain dynamics from noise, could the authors provide more details on the design or training methods used to differentiate signal from noise in the fMRI data? Additionally, it would be helpful for the authors to discuss any limitations or assumptions related to noise suppression in their approach.

3 - Could the authors elaborate on how temporal information is represented and processed within the model architecture? Specifically, it would be useful to clarify whether the model assigns a latent vector to each fMRI time point, and to explain the significance of $K$ in the equation. Does $K$ represent the number of time points in $g_{Fg} \in \mathbb{R}^{K_{Fg} \times D_{Fg}}, \quad g_{Bg} \in \mathbb{R}^{K_{Bg} \times D_{Bg}}$ ? This additional information would help assess the model's suitability for capturing temporal dynamics in fMRI data.

4 - Could the authors discuss the potential limitations of mapping fMRI data into a text embedding $Z$ and how they address any associated information loss? It would also be helpful to explain how their approach compares to a more direct mapping from fMRI data to the target, if feasible, and the potential impact on model performance.

5 - To facilitate evaluation, could the authors include experimental comparisons with relevant baseline methods, if available? For computational complexity, it would be useful if the authors could provide details on runtime and hardware requirements, and discuss how their pipeline compares to other methods in terms of computational efficiency as well.

6 - The paper requires revision to address minor typographical errors, such as changing "use experience" to "user experience" in the introduction and correcting "an shared" to "a shared" in the methods section.