Multi-Atlas Brain Network Classification through Consistency Distillation and Complementary Information Fusion

[W1. Too many hyperparameters to tune.]

Thank you for pointing this out. We agree that our model includes a range of hyperparameters, but we want to clarify that not all of them require meticulous tuning. Here are the details:

• The keeping ratio (sparsify threshold, related to adjacency matrix construction) is applied for GNN-based models and is fixed at 20% based on the dataset’s recommended setting in [1].
• The trade-off hyperparameters $\lambda_4$ (orthogonal loss) and the temperature parameter $\tau$ (subject-level consistency loss) are quite stable across datasets. We found that setting $\lambda_4 = 1.0$ and $\tau = 0.75$ works consistently well, allowing us to use these default values without further tuning when adapting to new datasets.
• The key hyperparameters that require tuning are $\lambda_1$, $\lambda_2$, $\lambda_3$ (for the different loss components), and the $k$ value in kNN. We performed a grid search for these parameters.

[W2. Numerous networks to learn complicates the training process that potentially leading to instability.]

We appreciate this feedback. While AIDFusion involves multiple components, it actually has fewer model parameters compared to several state-of-the-art multi-atlas methods. Table 5 in our manuscript shows that our model requires fewer parameters. Additionally, our experiments demonstrate that AIDFusion converges significantly faster especially on large datasets, requiring fewer epochs compared to existing methods, as shown in Table 5. Besides, regarding model stability, we can observe that the standard deviation of AIDFusion is at the low end among all models (as shown in Table 2). Even when compared with LR and SVM (which much fewer model parameters), the std of AIDFusion still much lower. This demonstrates the stability of our proposed method.

[Q1. How to overcome the risk of overfitting.] Overfitting is indeed a common challenge in brain network analysis due to the limited sample size. To prevent our model from overfitting, we use the early stopping criterion, i.e., we halve the learning rate when there is no further improvement on the validation loss during 25 epochs and stop the training once the learning rate is smaller than the minimum rate we set. And we also include dropout layers and L2 regularization in our model to prevent overfitting. This description is included in Appendix F.

[W3. Justification of the effect of incompatible nodes and orthogonal loss.] Please refer to the general response.

[W4. The comparisons are insufficient.] Please refer to the general response.

[1] Data-driven network neuroscience: On data collection and benchmark. NeurIPS 2023

[W1, Q2, Q4. Disentangle transformer may discard atlas-specific information that are useful for classification.]

Thank you for pointing this out. We aim to filter out atlas-specific information that is harmful to the downstream task. This is done in a soft way by using incompatible nodes in disentangle transformer. The atlas-specific information is not entirely filtered out as we can observe from Fig. 2 that different atlas still focuses on different ROI connections. In our design, we have applied the consistency constraints primarily at a higher representation level rather than directly at the ROI or feature level to balance this trade-off. The orthogonal loss is only an auxiliary objective, ensuring that the disentangled representations are distinct but not dominant in driving the training process. Additionally, our case studies in Section 5.4 and Appendix H illustrate that AIDFusion focuses on diverse connections from different brain networks, demonstrating its capability to capture atlas-specific information.

Regarding Q4, the learned consistent representations from brain networks with different coverages (e.g., AAL covering cerebellum regions, while Schaefer focuses on cerebral cortex) are obtained at a high-level feature space. This consistency aims to extract information that is disease-specific, even if the coverage differs. In our case studies, the attention weights show that the cerebellum ROIs receive lower attention, which aligns with existing neuroscience findings suggesting a lesser relevance of cerebellum regions in neurological disorder classification.

[W2, Q5. It is not clear what atlases should be fused.] Thank you for this observation. We agree that determining the optimal set of atlases for fusion is an open research question. However, the main focus of our work is to propose a robust method for integrating multiple atlases rather than conducting an exhaustive search for the best atlas combination. Our experimental results show that in most cases using two atlases achieves superior results to using a single atlas with the same model. Base on this observation, we further explore combining AAL116 with Schaefer at different number of ROIs, reported in Table 11 of Appendix G . Based on the results, we observe that combining two atlases with similar numbers of ROIs (e.g., Schaefer100 and AAL116) tends to yield better performance, which may be due to the balanced information content from both anatomical and functional perspectives. We acknowledge that this is a preliminary finding, and more extensive exploration is needed to determine the most effective atlas combinations. This is something we plan to investigate in future work.

[Q1. Not clear how the identity embedding was implemented.] We apologize for the confusion. Yes, one node corresponds to one brain ROI. Please refer to our general response for a detailed explanation. In brief, the identity embedding is implemented using a parameter matrix $W_{ID}$ that encodes the identity information of each ROI in different subjects with the same atlas. Each node is assigned an embedding vector based on its ROI identity, helping to differentiate nodes that belong to the same ROI across different subjects.

[Q3. Typo in Eq. (9).] Thank you for raising this issue. We have modified $i$ to $j$ in Eq. (9) of our revision.

[Q6. Loss function when extended to more than 2 atlases.] Yes, when AIDFusion is extended to handle more than two atlases, it computes the orthogonal loss, inter-atlas message passing, and both subject-level and population-level consistency for each pair of atlases. This pairwise computation ensures that the model can fully leverage the complementary information across all available atlases while maintaining consistency. We have clarified this in Appendix G of our revision.

Dear Reviewer 3zUh,

Thank you again for your time and valuable suggestions on our work! We understand you are busy. As the discussion period is closing soon, could you please take a look of our response above and let us know if our explanation addresses your concerns? We are more than happy to provide more details if it does not. And we would sincerely appreciate it if you could jointly consider our responses above when making the final evaluation of our work.

Sincerely,

Authors

[W1, Q1.C larification of identity embedding.] Please refer to the general response.

[W2, Q2. The efficacy of the disentangle Transformer is not fully supported.] Please refer to the general response.

[W3, Q3. The actual efficacy of the proposed architecture and losses is not described.]

We appreciate the reviewer’s insightful feedback on the subject- and population-level consistency constraints in our AIDFusion model. To address this, we conducted an in-depth analysis and visualized the impact of these losses.

1. Subject-level Consistency Loss: We visualized the difference matrices of hidden feature representations $\hat{\boldsymbol{H}}$ for two different brain atlases (Schaefer100 and AAL116). The results demonstrate that the subject-level consistency loss significantly reduces the discrepancies between the hidden features learned from different atlases, thereby enhancing the stability of the model's representations. This alignment indicates that AIDFusion captures shared patterns across atlases effectively, contributing to improved generalization.

2. Population-level Consistency Loss: We analyzed the difference in similarity matrices $\boldsymbol{G}$, which reflects pairwise similarities of subjects across atlases. Without the population-level consistency loss, substantial discrepancies were observed, potentially confusing the model. In contrast, applying this loss resulted in better-aligned similarity matrices, ensuring consistency in graph-level representations across different views. This leads to more robust predictions, even when faced with variations in brain network atlases.

Overall, these findings support the importance of both consistency constraints in aligning multi-atlas representations, improving model robustness, and enhancing performance in downstream tasks. We have added these visualizations and detailed explanations to Appendix I.1 of our revision for clarity.

[W4. Some figure and table captions are too terse.] Thank you for pointing this out. We have added more details on the captions of Figure 1, Table 1 and Table 3 to make them more informative in the revision.

[W5. Report testing time for each subject.] Thank you for this valuable feedback. We understand that reporting separate training, validation, and testing times is more conventional. However, due to the small size of the datasets and the relatively small number of parameters in our model, the testing time for each subject is extremely short and does not significantly impact the overall computational budget. Additionally, our method requires computing population-level consistency, which scales with the number of subjects rather than being a linear per-subject cost. For this reason, we reported the overall time cost rather than averaging it by number of subjects.

[W4.1, Q4. Conventional baseline clarifications.]

We appreciate your detailed feedback. We selected SVM and LR as conventional baselines because they have been shown to perform robustly in prior studies and the original dataset papers [2]. We also followed the same hyperparameter-tuning protocol as in the original work. Specifically, we tuned LASSO, Ridge, and their combinations for LR, and applied L2 regularization for SVM. The full list of tuned parameters is provided in Appendix F of the revised manuscript.

Additionally, differences in reported accuracies may arise due to variations in data splits, preprocessing, and cross-validation strategies. Note that using static splits may also incur higher accuracy because the hyperparameter-tuning could overfit on the specific split. Our brain network construction follows a standard pipeline and we implement all baselines under the same framework for fair comparison. Our baseline performance is comparable with the results reported in the original paper of the datasets [2].

[W4.2. Transformer baselines.] Please refer to the general response.

[W5, Q5. More ablation studies.] Please refer to the general response.

[W6.1. Coverage of brain regions by atlases.] While many atlases provide broad coverage of brain regions, not all voxels are assigned to regions across different atlases. As noted in Appendix A.2, up to 33.3% of voxels may differ between atlases like AAL, Schaefer, and HO. Even for the Craddock atlas mentioned in the comment, 16.1% voxels in Craddock atlas are not included AAL atlas, while 16.0% voxels in AAL atlas are not included in Craddock atlas. This variability supports our motivation to integrate multiple atlases for a more comprehensive representation of the brain.

[W6.2. Difference with atlas harmonization methods.] We appreciate your question. While CAROT and other harmonization methods primarily focus on aligning data across different atlases to ensure consistency, our approach simultaneously emphasizes both consistency and complementary information extraction. Specifically, our method incorporates the Subject- and Population-level Consistency Constraint at a higher representation level, an auxiliary orthogonal loss, and the Disentangle Transformer to capture consistency across data while also retaining atlas-specific information. Furthermore, through Inter-Atlas Message-Passing, we enhance the extraction of complementary features. This dual focus enables our model to retain task-specific variations across atlases that are often overlooked by traditional harmonization methods. Harmonization methods typically concentrate on achieving consistency across datasets, yet this often neglects the task-specific discriminative information that is crucial for improving performance in downstream tasks. In contrast, our framework is designed as an end-to-end classification system, which not only ensures consistency but also maximizes task relevance by allowing complementary features from different atlases to contribute to the final classification task.

[1] Modern network science of neurological disorders. Nature Reviews Neuroscience 2014

[2] Data-driven network neuroscience: On data collection and benchmark. NeurIPS 2023

[W1.1, Q1. Transformer trained on small datasets without any self-supervised pretraining.]

We appreciate your comment. As you noted, sample sizes for brain network analysis are indeed limited. We used the largest publicly available datasets for this domain. In fact, previous multi-atlas works such as MGRL (184 subjects), LeeNet (470 subjects), and METAFormer (884 subjects) utilized even smaller datasets than ours. For single-atlas methods like BNT, experiments were conducted on ABIDE with 1009 subjects, still fewer than in ours. We are also trying to collect and prepare more data, such as the mentioned UKBiobank. However, it involves different acquisition protocols and extremely time-consuming preprocessing (even 6-7 hours per subject). So we cannot add further data in this stage. We will try to include more data in our further work.

We acknowledge the potential of self-supervised pretraining to address the challenges of limited training data. However, building a brain-specific pretraining model leveraging the unique characteristics of brain data is part of our ongoing research and lies outside the primary scope of this work. Our current focus is on architecture design for multi-atlas information fusion rather than pretraining strategies. Notably, our baseline METAFormer employs self-supervised pretraining but does not exhibit significant performance gains compared to our approach. This can be attributed to METAFormer’s reliance on late fusion, which lacks intermediate interaction across atlases. The observed results underscore the importance of a tailored architecture, such as AIDFusion, to effectively extract and integrate information from multiple atlases.

[W1.2, Q2. Use processed connectomes instead of raw data?] Thank you for the insightful comment. We chose to work with processed connectomes because they effectively capture functional co-activation patterns between brain regions. These functional connectivity metrics have been well-established in neuroscience as reliable markers for neurological disorder classification [1]. Raw fMRI data often contain substantial noise, including motion artifacts and physiological fluctuations, which can obscure critical features. Besides, in the dataset paper we reference [2], the authors compare graph-based methods with time-series-based methods. Their results show that graph-based approaches, utilizing the connectivity matrix, consistently outperform time-series-based methods. This justifies our decision to use processed connectomes, aligning with the existing body of literature.

[W2. Use of only two atlases in experiments.] We selected AAL and Schaefer atlases because they are among the most widely adopted in the field, representing anatomical and functional parcellation approaches, respectively. Their detailed information of these two atlases are discussed in Appendix A.2. Additionally, we have tested a three-atlases case in Appendix G, on which AIDFusion also outperforms all baselines. We observed that all models tested on three atlases achieved poorer performance than the ones tested on Schaefer100 and AAL116. Therefore, we chose to conduct experiments on these two atlases in the main paper.

[W3, Q3. Unclear significance of results.] Thank you for pointing this out. We performed statistical tests to assess the significance of our results. Specifically, one-sided paired t-tests between AIDFusion and the best multi-atlas baselines yielded p-values of 0.0116, 0.0380, 0.0830, and 0.0886 respectively on the four datasets. This indicates that our model significantly outperforms existing methods on the two large datasets ABIDE and ADNI. However, on small datasets of PPMI and Matai with large standard deviations in 10 folds, t-tests are highly sensitive to outliers existing in the 10-fold results, thus decreasing the t-statistic calculated and lowering the chance of rejecting the null hypothesis. We have included these p-values in the caption of Table 2 in the revised manuscript for transparency.

[W2. Three Atlases Experiments]

We acknowledge that adding more atlases does not always improve performance, which is similar to the observations in multimodal learning where incorporating additional modalities may not consistently yield better results. Specifically, adding a third atlas might introduce noise or conflicting information that outweighs its potential benefits.

Besides, to further verify our finding that our method will benefit more from atlases with a similar number of ROIs, we conduct additional experiments on the ABIDE dataset using three atlases (Schaefer100, AAL116, and BASC122 [3]). Results shown in the following table demonstrate that for five multi-atlas methods, three of them (MGRL, METAFormer, and AIDFusion) achieve better performance with three atlases compared to any two-atlas combinations. Importantly, our proposed AIDFusion still outperforms all baselines in these settings. This reinforces our claim that AIDFusion effectively integrates multi-atlas information while remaining robust to the inclusion of additional atlases. Detailed results and discussion have been added to Appendix G in the revised manuscript. The experiments for ADNI under the same setting are still running and we will update the result to the next version of our paper.

[W1.1, Q1. Data Efficiency] We appreciate the reviewer’s concerns about training transformers on limited datasets. fMRI data are inherently high-dimensional and suffer from a low signal-to-noise ratio due to factors such as cardiac and respiratory processes or scanner instability [4]. This poses challenges for training machine learning models, particularly on small datasets. Transformers, while not immune to these issues, are better equipped to model the highly nonlinear nature of functional interactions in brain networks. However, for larger-scale datasets that are collected from multiple sites, such as ABIDE (17 sites) and ADNI (89 sites), site-specific noise can complicate training, leading to potential overfitting. This explains the observed trends: transformers may generalize better in smaller datasets such as Matai, where cross-site variability is less pronounced, while non-DL methods struggle with the data’s high dimensionality and complexity.

[W3, Q3. Significance Tests] We focused on comparing AIDFusion with the best multi-atlas baselines due to their alignment with our problem definition. Following the reviewer’s suggestion, we conducted one-sided paired t-tests between AIDFusion and the overall best baselines for each dataset. The revised tests yielded p-values of 0.0775 and 0.0380 for ABIDE and ADNI datasets, respectively.

[3] Multi-level bootstrap analysis of stable clusters in resting-state fmri. NeuroImage 2010

[4] Structural deep brain network mining. KDD 2018

Thanks again for the detailed response. As some of my initial concerns (e.g., adding relevant ablations) have been addressed, I am raising the score to 5. It is primarily not higher as the reported gains over baselines do not seem to be robust on the largest reported datasets, please see the discussion on significance testing.

[ Regarding the rebuttal discussion around data efficiency, for clarity, I was referring to transformers (by design) not having inductive biases (beyond permutation equivariance) and therefore having to learn them from large datasets, not the inter-site variability of neuroimaging datasets. It is not clear how this is learnable from datasets with N=60 from random initialization without pretraining. ]