Deep Independent Vector Analysis

1. Thank you for the suggestion on the sketch of proof. We have restated key assumptions and Theorems in the identifiability theory (Khemakhem et al. 2020), and provided a conceptual sketch of proof showing that the generative model assumed in DeepIVA is identifiable up to a permutation and component-wise transformation in Appendix A. We have also updated Section 2.1 iVAE and DeepIVA parts accordingly in the revised manuscript.

2. We are not clear about what the reviewer is suggesting we could do with the ROIs specifically. We aim to identify latent sources in a data-driven manner, and it is not necessary that linked sources are located at similar brain regions. Thus, a predefined ROI atlas might not be ideal as a proxy for ground-truth sources. Also, ROIs are not always the same for all types of MRI (for example, white matter tracts vs gray matter ROIs). While not seemingly a good depiction of the ground-truth, ROI-based data may be a good alternative to dimensionality reduction. We will consider that in future experiments.

3. Thank you for the thoughtful question. We train DeepIVA in a two-step procedure because of computational efficiency and practical consideration. If we train iVAE and then use MISA to align unimodal sources, it will require solving a combinatorial problem, which can be computationally expensive as the number of sources and datasets scale up. Additionally, if we first train iVAE independently, it is possible that parameters of one iVAE model might be stuck in local minima and the learned sources might not be linked. For example, in Figure 2 first row, we trained two iVAE models independently, and each model can identify sources fairly well, but cross-modal alignment is poor between two sets of independently identified sources. Iteratively training iVAE and MISA can guide the model to a solution not only maximizing for identifiability but also simultaneously maximizing for linkage.

We appreciate that the reviewer acknowledged the strengths of our work and include our point-to-point response as follows.

1. Identifiable representation. We have restated key assumptions and Theorems in the identifiability theory (Khemakhem et al. 2020), and provided a conceptual sketch of the proof showing that the generative model assumed in DeepIVA is identifiable up to a permutation and component-wise transformation in Appendix A. We have also updated Section 2.1 iVAE and DeepIVA parts accordingly in the revised manuscript.

2. Disentangled representation. We have updated “disentangled representations” to “latent sources”.

3. Scalability. Note that we utilize an approximate Jacobian by taking the average across all samples for subsequent computation (i.e. eigendecomposition), instead of using the Jacobian of each sample. Such an approximation leads to similar results while also providing computational efficiency gains. Moreover, we can compute the Jacobian for all modalities in parallel. Lastly, the identifiability theory of iVAE extends trivially to flow models (as discussed in Khemakhem et al. 2020), which have simple Jacobian form and could enable better scalability.

4. Generative performance. The iVAE loss from DeepIVA is typically not lower (better) than the vanilla iVAE loss after training for the same number of epochs, but this issue can be potentially mitigated by training the iVAEs alone following alignment at the last step of DeepIVA. Once aligned, it is likely that iVAE training would stay at the same local basin and keep the source permutation unchanged. Yet another solution is to update only the decoder weights for iVAE training while training the MISA loss on the encoder weights. This would help compensate for any negative effects of MISA on reconstruction.

5. Yes, the reviewer’s understanding is mostly correct, except that we did not apply a rotation matrix on the iVAE recovered sources (the synthetic dataset meets the conditions for P-identifiability within each modality) when we plotted the RDC matrices, such as Figure 2. Instead, we merely applied the same permutation matrix to all modalities such that high MCC values were along the main diagonal when we computed per-segment per-modality MCC.

6. Thank you for the thoughtful question. Let each dataset be a $N \times V$ matrix where $N$ is the number of samples and $V$ is the number of features. There are two ways to concatenate data: One is to concatenate data along the sample dimension (as in joint ICA, learning a shared transformation $f$ for both modalities and different latents for each modality) and the other is to concatenate data along the feature dimension (as in group ICA, learning shared latents and a single transformation $f$ that mixes all modalities indistinctly from one another). Either way, we learn a shared nonlinear transformation across datasets. If datasets were generated from a shared latent variable, it might be reasonable to use one of these two approaches. However, we are interested in studying different heterogeneous data modalities generated from modality-specific, non-shared conditionally independent latent variables and modality-specific nonlinear mixing processes. Thus, it makes more sense to estimate sources for each dataset separately, as in IVA, utilizing iVAE to obtain identifiability and MISA to capture linkage.

7. Sorry for the lack of clarification. Yes, our modification excludes the interaction between input features and auxiliary variables. We have updated Figure 1 and revised the related text in Section 2.1: “Additionally, since MISA is not designed to handle auxiliary information, we modify the original encoder architecture to distinguish between data features $\mathbf{x}^{m}$ and auxiliary variables $\mathbf{u}$ such that 1) the iVAE updates model parameters with respect to both $\mathbf{x}^{m}$ and $\mathbf{u}$ at the input layer, and 2) the MISA updates only those pertaining to $\mathbf{x}^{m}$ but not $\mathbf{u}$. The original iVAE model uses a single input layer taking the concatenated $\mathbf{x}^{m}$ and $\mathbf{u}$. In DeepIVA, we split this layer into two: one for data features $\mathbf{x}^{m}$ and another for auxiliary variables $\mathbf{u}$. The parameters with respect to $\mathbf{u}$ will only be updated at the iVAE training step but will remain frozen at the MISA training step. Also, the inputs for the auxiliary variables are set to $0$ during MISA training to ensure no influence from the frozen weights.”

We appreciate the reviewer’s insightful and constructive feedback. We include our point-to-point response as follows.

1. Thank you for the important suggestion. We have explicitly written the generative model assumed in DeepIVA, and provided a sketch of the conceptual proof showing that the generative model is identifiable up to a permutation and component-wise transformation in Appendix A. We have also updated Section 2.1 iVAE and DeepIVA parts accordingly in the revised manuscript.

2. Thank you for pointing out the multi-view ICA literature. Multi-view ICA assumes that the data can be modeled as a linear transformation of a common/shared source with an additive Gaussian noise. This assumption holds if all datasets are generated from a single shared latent variable, such as naturalistic datasets where subjects might share a similar temporal response. However, it doesn’t hold if each dataset is generated from a separate multidimensional latent variable, such as the expression level of structural and functional imaging features across subjects. The key assumption difference between multi-view ICA and DeepIVA is that sources are $**shared**$ across subjects in multi-view ICA while sources are $**linked**$ across data modalities in DeepIVA. In the context of multimodal fusion, it is more reasonable to assume that each data modality is generated by modality-specific latent variables, instead of a shared latent variable, especially for data modalities that are inherently heterogeneous (such as imaging and genomics). Thus, we believe that DeepIVA is more capable in multimodal fusion tasks, as it aims to identify a set of linked sources for each modality.

We have added a discussion to distinguish our approach and multi-view ICA in Section 1 Introduction: “Recent studies on multi-view BSS assume that observations from different views originate from a shared source variable and distinct additive noise variables (Richard et al., 2020; 2021; Pandeva & Forre, 2023; Gresele et al., 2020). However, in the context of multimodal fusion, it is more reasonable to assume that each modality is generated by modality-specific latent variables which, in turn, are linked across modalities, rather than a shared set, especially for data modalities that are inherently heterogeneous.”

3. Sorry for the lack of details about the neuroimaging dataset. We use structural MRI data (gray matter segmentation maps from T1-weighted images) and the voxel-wise amplitude of low frequency fluctuations (ALFF) from a resting-state fMRI dataset collected on the same subjects. The statistical dependence between modalities is modeled using subject expression levels. While we assume expression levels change (i.e., are nonstationary) according to age and sex groups, we assume the statistical dependence between data modalities remains the same and, thus, can be interpreted as structural and functional relationships associated with each latent source.

We have revised Section 2.3 Neuroimaging Data: “We utilize the UK Biobank dataset (Miller et al., 2016) $\mathbf{X}\in \mathbb{R}^{N \times V \times M}$ including two imaging modalities T1-weighted sMRI and resting-state fMRI ($M=2$) from $2907$ subjects ($N=2907$). We preprocess sMRI and fMRI to obtain the gray matter tissue probability segmentation (GM) and amplitude of low frequency fluctuations (ALFF) feature maps, respectively. Each GM or ALFF feature map includes $44318$ voxels ($V=44318$). Here, we use age and sex groups as auxiliary information, assuming that sources within each modality are conditionally independent given the age and sex group. This assumption is based on studies demonstrating the significant impact of age and sex on both brain structure and function (Raz et al., 2004; Good et al., 2001; Ruigrok et al., 2014).”

4. Regarding the MCC results, please first note that the per-modality per-segment MCC metric evaluates the MCC in the same way as in the nonlinear ICA literature, and these per-modality per-segment MCC values for 5 sources (>0.9) are very close to those reported in Khemakhem et al. 2020. Furthermore, in order to account for cross-modal linkage and cross-segment consistency, we propose a new approach to measure the MCC values (see Section 2.4 for details), and the proposed per-modality MCC, per-segment MCC and aggregated MCC metrics may be lower because the iVAE fails to align linked sources across modalities or consistent sources across segments.

MCC values are lower when we estimate more latent sources (15 sources) with the same sample size (2800 samples). Our hypothesis is that the sample size was not sufficient to accurately recover more latent sources. We repeated the experiments with a double sample size (5600 samples) and an intermediate problem scale (10 sources). According to the new result, we observe that 1) model performance decreases as the number of sources increases when the sample size stays the same; 2) DeepIVA performance improves as the sample size increases when the number of sources stays the same.

Please see Figure 4 and Section 3.1 in the revised manuscript: “We perform a systematic evaluation of model performance across different data-generating configurations by varying both the problem scale ($5$, $10$ and $15$ sources) and the sample size ($2800$ and $5600$ samples). The aggregated MD and MCC metrics are shown in Figure 4. Remarkably, DeepIVA outperforms iVAE and MISA in every configuration, showcasing its superior performance across all evaluated scenarios. Within each panel, we observe a consistent drop in model performance as the number of latent sources increases, suggesting that the optimization problem becomes more challenging as the latent dimension increases. Across horizontal panels, the DeepIVA performance improves for configurations with $10$ and $15$ sources when the sample size increases from $2800$ to $5600$, indicating that a larger sample size is necessary to better recover sources in a harder problem.”

I thank the authors for their reply, and for adding Appendix A.

While it helps clarify the method, the identifiability theory provided therein (mostly relying on results in (Khemakhem et al., 2020)) still leaves me with a few doubts. I will detail this below. Please let me know in case I am misunderstanding something.

Distinction between shared and linked. May I kindly ask you to elaborate on the distinction between shared and linked sources? Does "linked" mean that there is some kind of statistical dependence? In multi-view ICA, each view is generated by mixing a corrupted version of some shared sources ($\mathbf{s}$ are the shared sources, $\mathbf{s}+\mathbf{n}_i$ would be the corrupted version for the $i$-th view). So the $i$-th component of the corrupted sources is not independent across different views. Can I think of these corrupted components, or sources, as linked according to your terminology? My impression is that these could be considered linked, albeit possibly with a different kind of statistical dependence from the one in your equation (3).

MISA in the nonlinear case. Based on my understanding, the MISA procedure relies on "all-order statistics" to match linked sources. However, in nonlinear ICA (including the iVAE results), an identifiable model may separate the true sources, but element-wise non-linearly distort them w.r.t. the true ones. This element-wise distortion has, in principle, the capability to transfrom any univariate distribution into a completely different one, thereby potentially modifying statistics of all orders, and (in my understanding) destroy the signal MISA relies on to operate the matching. (Note that this problem does not arise for linear models, for which MISA was originally developed.) Separately training different models on different datasets may recover sources which are distorted up to distinct element-wise distortions. Based on this, I'm having difficulty understanding the rationale behind the possibility of matching linked sources.

fMRI. Thanks for the provided explanation. May I kindly ask you to further clarify what the linked sources are expected to capture, i.e., what is the meaning of linked sources across data modalities? Should I think of them as capturing some subject-specific characteristics influencing both functional activity in fMRI and structural properties captured by sMRI? Can I think of a generating process in which a first variable captures properties which depend on subject identity, $I_i$, and in turn this influences both structural properties $S_i$ and functional activity $F_i$, thus rendering them dependent? I would represent this as a DAG with arrows $S_i \leftarrow I_i \rightarrow F_i$. Is this a correct picture? If so, I believe that this would strengthen the connection to multi-view ICA (see e.g. Fig. 2 in (Gresele et al., 2019)), related to my concerns expressed above.

Thank you for your reply, which answered some of the points I raised.

Distinction between shared and linked; fMRI. My understanding of your reply is that corrupted sources with additive noise, $\mathbf{z}_1 := \mathbf{s}+\mathbf{n}_1$ and $\mathbf{z}_1 := \mathbf{s}_2+\mathbf{n}_2$, can only model simple (linear) dependence between $\mathbf{z}_1, \mathbf{z}_2$. More complicated statistical dependence cannot be modelled accurately by the additive model in (Richard et al., 2020), whereas the model you use may provide more expressivity. This sounds like a valid argument, where it appears that there is a strong connection between corrupted sources $\mathbf{z}_1, \mathbf{z}_2$, as defined above, and the linked sources you consider; your considered case may be more expressive and, potentially, more useful for neuroimaging data:

If it were the case that $p(I^S_i, I^F_i)$ is fully explained by linear dependence, then a model of shared sources might be useful, but that is currently unknown, and it is necessary to develop approaches that capture higher-order dependence.

I accept the explanation in principle, although it would be interesting to have some empirical check and comparison of the two methods, to verify whether approaches that capture higher-order dependence (and nonlinear mixing), as yours, allow for an improved performance or discovery of interesting biomarkers.

I would suggest discussing this in more detail in the paper, particularly because more complex (non-additive) corruptions $\mathbf{z}_i := \mathbf{g}(\mathbf{s},\mathbf{n}_i)$ have already been considered in the literature on the theory of nonlinear multi-view ICA (Gresele et al., 2019).

MISA in the nonlinear case. Thank you for the provided explanation. I agree that element-wise transformations cannot erase dependence among the linked sources. I would appreciate it if you could provide a self-contained proof along with a formal illustration demonstrating how this argument leads to identifiability for the method you proposed (the current presentation of the proof in Appendix A may be seen as not fully explicating all the formal passages, as also observed by other reviewers).

I appreciate the authors' efforts in replying to my questions. Since the answers clarified some of the points I raised, I will raise my evaluation.

$**Q1**:$ More detailed explanation regarding the motivation and unique contributions.

$**A1**:$ We thank the reviewer for requesting clarification on the motivation and contributions of our proposed method.

Regarding motivation, we have revised the manuscript to explain why we are interested in multimodal fusion in Section 1 Introduction: “Jointly analyzing two imaging modalities can uncover cross-modal relationships that cannot be detected by a single imaging modality, providing new insights into structural and functional interactions in the brain and its disorders (Calhoun & Sui, 2016).”

Also, another reason we choose MISA is that we want to utilize its flexible subspace modeling to further solve nonlinear independent subspace analysis (ISA) problems. We have added one more reason why we choose MISA to align latent sources in Section 4 Discussion: “We plan to extend our proposed method from nonlinear IVA problems to nonlinear ISA problems, aiming to capture source dependence by leveraging higher-dimensional subspaces.”

Regarding contributions, our study is an initial step towards learning linked and identifiable sources from multimodal data. We empirically demonstrate that DeepIVA can learn linked and identifiable sources by unifying iVAE and MISA, and “the idea of combining iVAE and MISA is new”, as acknowledged by reviewer queK. We would like to point out that our proposed method is not “simply a combination of existing methods”. We modified the iVAE encoder architecture to process data features and auxiliary information separately, as noticed by reviewer dLxh. Additionally, the linear demixing matrix in the original MISA framework was replaced by a nonlinear encoder, and we adjusted the loss term accordingly as described in Section 2.1 Deep Independent Vector Analysis.

We have updated Figure 1 to illustrate the extra details of our method, including the data input and the encoder layer. We have also updated Section 2.1 Deep Independent Vector Analysis to explain the architecture change: “Additionally, since MISA is not designed to handle auxiliary information, we modify the original encoder architecture to distinguish between data features $\mathbf{x}^{m}$ and auxiliary variables $\mathbf{u}$ such that 1) the iVAE updates model parameters with respect to both $\mathbf{x}^{m}$ and $\mathbf{u}$ at the input layer, and 2) the MISA updates only those pertaining to $\mathbf{x}^{m}$ but not $\mathbf{u}$. The original iVAE model uses a single input layer taking the concatenated $\mathbf{x}^{m}$ and $\mathbf{u}$. In DeepIVA, we split this layer into two: one for data features $\mathbf{x}^{m}$ and another for auxiliary variables $\mathbf{u}$. The parameters with respect to $\mathbf{u}$ will only be updated at the iVAE training step but will remain frozen at the MISA training step. Also, the inputs for the auxiliary variables are set to $0$ during MISA training to ensure no influence from the frozen weights.”

Furthermore, we have restated the identifiability theory in Khemakhem et al., 2020, and provided a conceptual sketch of proof showing that the generative model assumed in DeepIVA is identifiable up to a permutation and component-wise transformation in Appendix A. We have also updated Section 2.1 iVAE and DeepIVA parts accordingly in the revised manuscript.

$**Q2**:$ Related works on nonlinear ICA without auxiliary variables.

$**A2**:$ We introduce methods relevant to the current work in the introduction section and mention methods worth exploring in the future work section. We are interested in the application of neuroimaging analysis, and the iVAE has been shown to work on an fMRI dataset (Khemakhem et al. 2020) among the mentioned papers. Thus, we chose iVAE for this present work, and we will keep exploring other methods.