Learning High-Order Relationships of Brain Regions

We would like to address your concerns and questions below

Weakness

C1

A transpose operator should be inserted.

RC1

Thanks for pointing this out and sorry for the confusion caused by the notation. We have inserted the transpose operator and gone through all the notations to make them more consistent and clear.

Questions

Q1

Can you briefly explain how to choose the number of hyperedges?

RQ1

Sure! As stated in Section 4.1 (hyperedges construction), the number of hyperedges $K$ is a pre-defined hyperparameter. We choose $K=32$ based on the discussion and ablation studies that have been included in Appendix E in the submission. Basically, the performances improve as the $K$ goes larger, and becomes saturated after $K=32$.

We would like to address your minor concerns in this thread.

C4

Add a README file to explain how to replicate the results.

RC4

Thanks for the suggestion! We created an anonymous Github link https://anonymous.4open.science/r/HyBRiD-A7CC. Note that the code may be a little bit messy because we haven’t had time to organize and refactor it. We will do this as soon as possible in the future.

C5

Notations in Section 5.1 are not very clear.

RC5

i) I agree that CPM(E, Y) is more accurate and Eq. 13 has been updated.

ii) $E$ denotes traditional pairwise edge weights, and it is a vector of length $K_p$, which is the total number of pairwise edges. We made it clearer by changing the notation $E$ to $\boldsymbol{W}_p$.

iii) We made it clearer by changing $\boldsymbol{w}$ to $\boldsymbol{w}_h$ for Eq.13. And made $\boldsymbol{w}_h = \boldsymbol{w}$ as a special case when evaluating on our model.

You can check Eq.12 and Eq.13 in the updated version.

C6

i. Typos: Section 2.1 "Inupt" should be "Input"; Section 5.2 "conducte" should be "conduct"; Figure 7: "grpahical" should be "graphical".

ii. CPM should be defined in the abstract and introduction upon its first occurrence.

iii. In Section 5.1 Dataset, RS (resting state) should be defined.

iv. In Table 5, $\beta=0.2$ instead of 0.3 to be consistent with Section E.3?

RC6

i) Thanks for pointing out. Typo fixed.

ii) Thanks for the suggestion. We acknowledge the importance of defining 'CPM' upon its first appearance in the text. However, due to the constraints of the abstract and introduction sections, providing a full definition is challenging. We have, therefore, incorporated a concise explanation, stating that CPM is the evaluation protocol for our method, which gives readers an adequate initial background. In addition, we also refer to the appendix. We have updated:

In abstract

Our model outperforms the state-of-the-art predictive model by an average of 12.1%, regarding the quality of hyperedges measured by CPM, a standard protocol for studying brain connections.

In introduction

We quantitatively evaluate our approach by a commonly used protocol

for studying brain connections, CPM (Shen et al., 2017) (Appendix A), and show that…

iii) Sorry for the confusion. We have modified it to

the resting state where the brain is not engaged in any activity (Rest)

iv) Thanks for pointing out. Typo fixed.

We would like to address your major concerns in this thread.

C1

The reviewer wondered if the model could be applied to a dataset with fewer samples.

RC1

While we do think training on a small dataset is an interesting question, we argue that both our model and the state-of-the-art one are unable to solve it, and finding out the solution to such a problem is out of the scope.

To show it, we conducted experiments of training on individual datasets for both our model and BrainNetTF. The $r$ values:

The $P$-values corresponds to these $r$ values:

From this table, we can see all of the $P$-values> 0.05. Therefore, none of the existing models obtain meaningful results in this situation.

However, our work implies that if you have small fMRI datasets, you can usually combine them for a better result. We hope this could be a good solution to the small datasets issue.

C2

Should discuss potential reasons why HyBRiD failed at Rest 1 dataset.

RC2

Thanks for your suggestion. We inspect the performance on both the training, validation and test datasets. We summarize the performances of our model and the SOTA one in the table below

We find our model outperforms BrainNetTF on both training and validation datasets but fails on test datasets. Therefore, given that Rest 1 is the most noisy dataset, we argue that the failure is because

1. Our model is more expressive than BrainNetTF, so it overfits the training dataset more than BrainNetTF.
2. The validation score is higher, indicating the failure reason is the discrepancy between the training dataset and the test dataset.

Additionally, Note that although HyBRiD is not the best on Rest 1 dataset, it is still the second-best one.

C3

The reviewer would like to see region importance for resting state and individual hyperedges.

RC3

We did not visualize it in the original submission because different from task states, where specific brain regions are activated in response to particular tasks, brain activities during resting states, are not driven by external tasks, leading to more diffuse and less predictable patterns of activation. This makes it harder to interpret. However, we appreciate the reviewer’s interest and we have included the visualizations and interpretations of resting-state region importance and individual hyperedges in Appendix G in the updated version. We would like to do more analysis and interpretations in the future.

Q1

Does the model generalize well across conditions or out-of-sample data? For example, if a model is trained on resting state data, can it be applied to predict task data?

RQ1

Thanks for your interest in the model's generalizability. In our experience, training on resting-state and generalizing it on task-state is difficult because resting-state data is more noisy. However, we do think it is a good idea so instead, we train our model on task-state data and generalize it one resting-state data.

The tables illustrate that our model demonstrates a certain level of generalization. For Rest 1, it is even better than training on Rest 1. We argue that it is because the model easily overfits on Rest 1 dataset, which is consistent with our response for RC2.

We believe there is a fundamental misunderstanding between the reviewer and us, which will be described below. In addition, we have also updated the paper to avoid other potential confusion. We would like to resolve the misunderstanding in this thread.

C2

The proposed method learns a partition, rather than a cover.

RC2

We would like to politely point out that the claim is incorrect. Our method indeed learns a cover, rather than a partition.

We guess the reviewer obtained such a conclusion because the statement in the submission

The equality holds if and only if nodes are independent and hyperedges do not overlap

However, this is a fundamental misunderstanding of our optimization objective. This sentence just stated the condition in which equality holds in this inequality.

Specifically, Eq.10 shows

$$I(H; X) \leq I_{\text{upper}}(H;X)$$

where $I_\text{upper}(H;X)$ corresponds the right-hand-side of Eq.10. We optimize $I_\text{upper}(H;X)$ as a surrogate objective of $I(H;X)$. In other words, we minimize $I(H;X)$ through minimizing $I_\text{upper}(H;X)$. This is a common technique in an abundance of works (e.g. in ELBO [1][2], in IB [3][4], and others [5][6]). Hence, the objective does not indicate partition.

On the contrary, if one wants to force the partition, he/she should optimize the tightness instead

$$T(H, X) = I_{\text{upper}}(H;X) - I(H; X)$$

where $T(H,X)$ denotes the tightness and it is minimized when hyperedges do not overlap. And minimizing the tightness is not what we are doing in this work.

Furthermore, the MSE term is not for regularization, it is the lower bound of informativeness in Eq.9, where $Y$ is the cognition score (defined in Section 2), which ensures the predictive performance of our high-order relations. The $I_{\text{upper}}$ actually corresponds to $\sum_{k=1}^K \sum_{i=1}^N \mathbb{H}[X_i](1-p_{\theta, i}^k)$ in Eq,10 & Eq.11, where one can find there is no force to partition at all.

The property that a node can be connected by any number (0 to $K$) of hyperedges is attributed to our masking mechanism. Each hyperedge is generated by a learnable mask independently, which is free to connect to the same node.

[1] Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.

[2] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems, 33, 6840-6851.

[3] Kim, J., Kim, M., Woo, D., & Kim, G. (2021). Drop-bottleneck: Learning discrete compressed representation for noise-robust exploration. arXiv preprint arXiv:2103.12300

[4] Wu, T., Ren, H., Li, P., & Leskovec, J. (2020). Graph information bottleneck. Advances in Neural Information Processing Systems, 33, 20437-20448.

[5] Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., & Bengio, Y. (2018). Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670.

[6] Veličković, P., Fedus, W., Hamilton, W. L., Liò, P., Bengio, Y., & Hjelm, R. D. (2018). Deep graph infomax. arXiv preprint arXiv:1809.10341.

C4.2

The proposed method should be compared with Deep Clustering if it will be rewritten to focus on feature grouping.

RC4.2

This concern is also a result of the misunderstanding of the model objective. Even if our task might be related to Deep Clustering, it differs a lot from the following aspects.

1. Our methods can learn a cover, which means a node can be connected to multiple hyperedges at the same time. However, most clustering methods usually assume each node only belongs to one cluster.
2. Most Deep Clustering methods usually need the supervision of cluster labels or semi-supervision signals of “must-links” and “cannot-links” [7]. However, we don’t have such labels besides the prediction labels.
3. Deep Clustering methods don’t characterize the clusters with a succinct summary. However, our methods compute a weight for each hyperedge, which is informative toward cognition.
4. In contrast to clustering methods, which usually assign each node to a specific cluster, our approach selectively prunes nodes that are not crucial for cognition. For example, for the EN-back condition, only 64 out of 164 regions are connected by at least one hyperedge.

[7] Ren, Y., Pu, J., Yang, Z., Xu, J., Li, G., Pu, X., ... & He, L. (2022). Deep clustering: A comprehensive survey. arXiv preprint arXiv:2210.04142

C5.2

Clusters overlap with each other, which contradicts the regularization

RC5.2

This is also a result of the misunderstanding in Q2. Our approach does not incorporate any regularization term to enforce the partition according to Eq.11 in the submission, so hyperedges are free to overlap with each other as long as the overlap enhances the predictive performance.

We would like to address the reviewer's concerns in this thread.

Weakness

C1

The author needs to clearly and formally define what type of high-order relations the proposed model captures.

RC1

We would like to politely point out that the high-order relations have been intuitively defined in the second paragraph of the Introduction (maximally informative and minimally redundant, MIMR) and formally defined in Equation 10 (IB principle).

Specifically, our high-order relations are those that are most predictive, yet contain the least redundant information towards cognition. We have made the definition more clear and explicit in the updated version.

The definition of ours is different from that in the three papers, where they consider nodes in a high-order relation should be collectively correlated in terms of co-fluctuations or some metrics like O-information.

However, we appreciate the references you provided. We have included them and discussed the difference between their methods and ours in Related Work (high-order relationships in fMRI) in the updated version: Some of them are not scalable to high-order relations of a large degree, while some of them are inconsistent with our MIMR objective.

C3

The approach feels like an ad hoc method for grouping the feature vectors.

RC3

We would like to point out that our method is beyond feature grouping because

1. A node is free to be connected by different hyperedges at the same time, which makes it different from grouping or clustering.
2. Our method summarizes each hyperedge into a scalar, which is itself informative without the original features.
3. Our method also shows consistent predictive performance improvements other than selecting significant feature groups.
4. Our objective (MIMR) is based on a widely used principle and can be generalized on various tasks and various inputs.

Although in practice we choose the Pearson correlation as our input features, our method can generalize on other features (e.g. time signals). Our choice of Pearson correlation coefficients as input features is intended to enable a direct comparison with the state-of-the-art method [9], and they use Pearson correlations as their node features.

[9] Kan, Xuan, et al. "Brain network transformer." Advances in Neural Information Processing Systems 35 (2022): 25586-25599

C4.1

Predictive quality has nothing to do with discovering high-order relations.

RC4.1

This concern is a result of the misunderstanding of the model objective, as explained in our responses to Q1 and Q2, the high-order relations we want to discover in this work are the most predictive ones with minimal redundancy.

C5.1

The author does not show individual hyperedges.

R5.1

We focus on the region importance instead of individual hyperedges because we believe statistics make it clearer to present information. However, we appreciate your interest in individual hyperedges so we have included visualization and analysis of the most significant hyperedge under the EN-back condition in Appendix G.

The reviewer stated

The effectiveness of the inference method was evaluated based on an fMRI condition classification task with comparison…

We would like to politely point out that our method is not evaluated based on the fMRI condition classification task. Instead, as mentioned in Section 2 and Section 5.1 (Dataset) in the submission, our method is evaluated based on cognition score prediction, which is a regression task.

Weaknesses

C1

It is not valid to infer functional connectivity according to the predictive performance.

RC1

We believe that there is a misunderstanding on the motivation. Traditional methods [1][2] typically define functional connectivity based on similarity or correlation (like Pearson correlation, and mutual information). However, this kind of connectivity is not informative toward a neurological outcome (see the performance of standard pairwise baseline in Table 1 as evidence) and might fall short in providing insights relevant to the outcome. As a result, it is crucial to learn these connections taking the outcome into account like what has been done in [3][4]. Therefore, our work does not seek to identify the traditional connections.

Instead, as mentioned in 3rd paragraph of the submission, the objective of our method is to discover high-order connections that are predictive toward a cognition score (a part of our MIMR objective). And for regions that are connected by a predictive high-order relation, the relations between these regions can be highly non-linear and complex, and far beyond any existing manually-designed metrics (e.g. Pearson correlation, mutual information).

[1] Shen, X., Finn, E. S., Scheinost, D., Rosenberg, M. D., Chun, M. M., Papademetris, X., & Constable, R. T. (2017). Using connectome-based predictive modeling to predict individual behavior from brain connectivity. nature protocols, 12(3), 506-518.

[2] Boyle, R., Connaughton, M., McGlinchey, E., Knight, S. P., De Looze, C., Carey, D., ... & Whelan, R. (2023). Connectome‐based predictive modeling of cognitive reserve using task‐based functional connectivity. European Journal of Neuroscience, 57(3), 490-510.

[3] Kawahara, J., Brown, C. J., Miller, S. P., Booth, B. G., Chau, V., Grunau, R. E., ... & Hamarneh, G. (2017). BrainNetCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment. NeuroImage, 146, 1038-1049.

[4] Mahmood, U., Fu, Z., Calhoun, V. D., & Plis, S. (2021). A deep learning model for data-driven discovery of functional connectivity. Algorithms, 14(3), 75.

Questions

Q1

How is the p-value calculated for the hyperedge in Fig. 4?

RQ1

As mentioned in Section 5.3

CPM conducts a significance test on pairwise edges and hyperedges internally based on a linear regression model, and thus we can obtain a P-value for each hyperedge from the significance test.

At a high level, CPM internally calculates a correlation coefficient for each hyperedge with the cognition score. Under a classical statistical hypothesis test framework, the coefficient follows a distribution. The p-value is obtained from the distribution. For details, we have added a paragraph about it in Appendix B.

Q2

How the DimReduction MLP could be trained with a large number of nodes.

RQ2

We apply DimReduction MLP independently on each hyperedge. As specified in Eq. 6 in the submission, for a hyperedge we first average features of nodes in that hyperedge by ${\boldsymbol{m}^k}^T \boldsymbol{h}^k$ and obtain the hyperedge features in $\mathbb{R}^{d}$. After that, the DimReduction MLP compresses the dimensionality from $d$ to $1$ as a scalar weight for the hyperedge. The parameters of the DimReduction MLP are shared for every hyperedge and every dataset.

Q3

Are the linear-head Fl shared across hyperedges or trained separately for each hyperedge?

RQ3

As specified in Eq.7, the inputs of the linear-head $\mathcal{F}_l$ are scalar weights of all hyperedges. Consequently, the notion of shared vs. separate linear-head functions in terms of hyperedges does not apply in this context regarding hyperedges. However, $\mathcal{F}_l$ is shared across all datasets.