Machine Learning Benchmark on Dynamic Functional Connectivity: Promise, Pitfalls, and Interpretations

1. While it is valuable to combine task-fMRI and rs-fMRI datasets as a benchmark to gain insights into deep learning model design in neuroimaging, the benchmark dataset lacks some novelty, as it primarily consists of datasets previously introduced in studies such as BrainGB (Ref1) and NeuroGraph (Ref2).

• Therefore, it would be more differentiated if the models of each group were evaluated in each scenario and with each setting through a table.
• It would be helpful to include a detailed explanation of the criteria for the configuration of the input data of the deep learning model in using the benchmark, such as performance comparison of various input feature types in FC network data or network topology configuration.
• An additional explanation on how dynamic functional connectivity is configured would be beneficial. For example, this could involve the window size (which determines the number of dynamic FC matrices).

2. The models are referred to as SoTA deep learning models, but they may not demonstrate optimal performance, as it lacks graph models (such as BrainGNN-Ref3) and transformer models (Ref4) that have shown strong performance on specific datasets. Additionally, computational details such as model parameters, memory usage, and training time are challenging to compare across models.

• It would be helpful to provide an explanation of the selection criteria, as the model structures used appear relatively simple
• Including an additional table to show the memory usage, training time, and parameters of spatial-based and sequential-based methods would facilitate an intuitive understanding of the differences between models.

3. Although model global attention weights were used to assess explainability through brain activity maps, attention maps specific to neurodegenerative or neuropsychiatric disorders were not provided separately, and a standardized attention module was lacking. BrainGNN [ref3] evaluated model explainability in disease diagnosis based on rs-fMRI data. If a similar approach was applied in this study, additional analysis on why it may not have performed as expected would be require.
4. The authors performed statistical analysis between groups with significant test in figure3, it is difficult to identify intuitive individual performance comparison of the models in the experimental table. it seems that further performance comparison of the models is needed for other datasets like figure4.

1. The paper presents some confusion regarding the concept of dynamic functional connectivity.
2. This paper summarizes existing datasets. However, further depth is needed regarding the related methods and quality control of the datasets.
3. The authors summarized seven datasets. However, some other important and commonly used datasets, such as ABIDE II, Cleveland Neuropsychiatry Study (CNP), The Pediatric Obsessive-Compulsive Disorder Treatment Study (POND), and NKI-Rockland Sample, seem to be omitted.
4. From a methodological perspective, the authors only considered deep learning methods, overlooking traditional machine learning approaches and other methods, such as deep forest.

The first claimed contributions is a collection of preprocessed datasets, however this work uses existing image data and existing preprocessing algorithms. Further, some of the datasets have already been preprocessed, and there is little to no description of how the data will be released. All of limits the value of their data contribution.

The authors present a dichotomy between "spatial" and "sequential" data. However, unless I am misunderstanding their setup, there is a key relationship between these two forms in that the spatial data can be written as a function of the sequential data i.e. the sequential data has at least as much information as the spatial data. In theory, a good sequential classifier could always outperform a spatial model. I believe that the main results in this paper are not biological but a form of the classic bias-variance tradeoff.

The authors aim to validate models by comparing a set of attention maps to the "brain activation patterns established by neuroscience," but there is little description of what activation patterns they are referencing. Figure 5 shows a set of outlines that vaguely identify "Motor and Language-related brain areas" but these outlines vary from panel to panel, and it seems that their black outline follows the postcentral gyrus, which is generally associated with somatosensory, not motor activity.

One of their central conclusions is that spatial models are better for disease classification and sequential models are better for task classification. There are already ample results on these setups (e.g. Wang et al. 2020 for task classification with fMRI time-series or Dadi et al., 2019 for disease prediction with functional connectomes), so I am not sure their conclusion is surprising. They also state that disease "forecasting" is promising, but it seems all their experiments involve classifying patients with an existing disease, not forecasting future disease.

Misc.

• There are several typos or grammatical errors: "Since connectome is often encoded...", "As shown in Table 1, we uses a diverse...", "Human Connectome Project in Aing"
• The equation W=XX^T only works if X is normalized appropriately.
• The explainability section mentions both interpretability and explainability while there is a difference between these concepts, see Rudin, 2019.

In implementing multiple models, key hyperparameters that can significantly impact model performance are not properly optimized. For example, in graph-based models, performance can vary greatly depending on how the graph topology is consturcted for each study, but the authors adopted the simplest method of remaining top 10% of values in FC matrix. In addition, other model hyperparameters, such as the depth of the layers and the number of nodes within the layer, were also applied to the default settings provided in each original paper with fixed learning rate of 0.0001. Since these hyperparameters can greatly affect performance depending on the size of the data and the difficulty of the task, it is common to optimize them based on validation data for a fair comparison. The reviewer believes that this limitation is closely related to the fact that even the simplest models, such as MLP, result in high performance with little influence on hyperparameter settings.

The authors stated in their paper that they compared SOTA methods, but this claim is difficult to accept since most implementation models are traditional and basic. In particular, the authors implemented spatial and sequential models separately for fMRI analysis, but many recent studies are simultaneously considering spatial-temporal features of fMRI.