Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine

[Q3] On the motivation of using frequency-domain representations

We would like to further elaborate on the motivation to use frequency-domain representations which the paper touches upon in L203-221. We will add a more detailed discussion of the motivation and need for frequency-domain representations in the paper appendix.

• Signal preserving: The general motivation for signal-preserving aggregation for multimodal machine learning has been extensively studied in the context of fusion methods. Several related works in different domains and modalities [3, 4, 5] emphasise the importance of fusion or aggregation methods that avoid signal interference, where meaningful patterns or complementary information from individual modalities may be attenuated, distorted, or lost due to naive aggregation approaches such as averaging or summation. For example, bilinear pooling [3] of two latent vectors $h_1 \in \mathbb{R}^n$ , $h_2 \in \mathbb{R}^m$ and resulting tensor $h_{12} \in \mathbb{R}^d$ ($y=h_1^T A h_2 + b$) with a learnable parameter $A \in \mathbb{R}^{n \times m \times d}$ follows the same motivation. With the same motivation in mind, we take advantage of the Fourier transform’s orthogonal properties, as after the transform each frequency component (represented by sine and cosine waves) is orthogonal to the others, such that $\int_{-\infty}^{\infty} sin(\omega_1h_1) cos(\omega_2h_1) dh = 0$ for angular frequencies $\omega_1 \neq \omega_2$ [4]. This means that the contribution of each frequency is independent of others and there is no overlap between them. It follows that if we take the harmonic mean of two fourier-transformed latents $\mathcal{F}_1(\omega)$ and $\mathcal{F}_2(\omega)$ as $H(w) = \frac{2\mathcal{F}_1(\omega)\mathcal{F}_2(\omega)}{\mathcal{F}_1(\omega)+\mathcal{F}_2(\omega)}$, the harmonic mean aggregates each $\omega$ in a localised manner. This means that each frequency component in $\mathcal{F}_1(\omega)$ interacts only with the corresponding frequency component in $\mathcal{F}_2(\omega)$, i.e., without interference from other frequencies.
• Distance preserving: Parseval’s Theorem shows that the Fourier transform is a unitary operator, meaning that the sum or integral of the square of a function is equal to the sum or square of its transform [4,5]. As such, the distances between two signals are the same between the transformed and untransformed representations.
  • Concretely, the theorem states that the energy of a signal is preserved in both the time domain and frequency domain, where its energy is measured as the integral of the function. Formally $\int_{-\infty}^{\infty} |f(h)|^2dh = \int_{-\infty}^{\infty} |\mathcal{F}(f)(\omega)|^2 d \omega$ for latent signal $f(h)$ and its fourier transform $\mathcal{F}$.
  • The Euclidean distance between two signals $f_1(h)$ and $f_2(h)$ in the spatial domain is: $||f_1-f_2|| = \sqrt{\int_{-\infty}^{\infty}|f_1(h) - f_2(h)|^2 dh }$
  • The Euclidean distance between the Fourier transforms of two signals is:
  • $||\mathcal{F}(f_1) - \mathcal{F}(f_2)|| = \sqrt{\int_{-\infty}^{\infty} | \mathcal{F}(f_1)(\omega) - \mathcal{F}(f_2)(\omega)|^2 d\omega}$
  • From Parseval’s Theorem, it follows that $||f_1 - f_2 || = ||\mathcal{F}(f_1) - \mathcal{F}(f_2)||$.
  • This distance-preserving capability is beneficial in designing loss functions in a multimodal setting.
• Invertible: The Fourier transform is not an idempotent function but periodic with a period of 4, i.e., $\mathcal{F}^4(f) = f$. This would prevent the iterative architecture outlined in Section 3 from working since a repeat application of the transform would lead to different representations. Meanwhile, the Fourier Inversion Theorem [6] shows that we can invert the frequency-domain representation to its original function without the 4x repeat application, making it suitable for the chosen iterative architecture.
• Efficient: The Fast Fourier Transform (FFT) has a time complexity of $\mathcal{O}(n log(n))$ making it scalable to very large datasets.

Thank you for your thoughtful comments and suggestions for improvement. In the following, we provide detailed answers to your comments with additional explanations of the raised issues. We also include these clarifications and additions in the manuscript.

[Q1] Clarifications on "topological equivalence" and "topology agnostic."

In this paper, we focus on multimodal learning from multiple heterogeneous modalities. Most current state-of-the-art fusion methods, that require end-to-end training, can be very computationally demanding and often do not scale beyond two modalities. Moreover, in order to train end-to-end fusion methods, we usually require a large amount of paired data for training (e.g., a vision & language model, where each image needs to be paired with a text string to learn a meaningful representation). In many multimodal learning problems for biology and medicine, which this paper focuses on, large amounts of paired data are relatively scarce. At the same time, single modalities are often widely available (just not paired).

Model merging, on the other hand, allows for combining well-performing unimodal models trained in isolation, eliminating the need for paired data. Most work in this field focuses on model merging in multi-task settings, i.e., where you have two identical model architectures (topologies) trained on separate tasks and combine them to obtain a single model instance that performs well on both tasks [1, 2]. For equivalent topologies, this process is relatively straightforward as you have a 1:1 mapping of layers and their corresponding weights between the two task-specific models. However, in multimodal settings, we often cannot use equivalent architectures e.g., with imaging and tabular data as their architectures need to handle different heuristics and tensor shapes. Therefore, in multi-modal learning, having a topology agnostic model merging approach w.r.t the modality-specific encoders is critical to enable model merging.

This is the main contribution of our work: MM-Lego introduces modality-specific adapters that enable model merging even if the architecture is not equivalent — allowing users to combine any set of unimodal models (including current state-of-the-art foundation models) into a multimodal model with very minimal fine-tuning.

[Q2] Clarifications on the taxonomy of "model merging"

We propose two variants of MM-Lego: LegoFuse and LegoMerge. LegoFuse is indeed closer to a fusion method, where we aim at combining models before further fine-tuning them to learn cross-modal interactions on a limited amount of paired data. LegoMerge, on the other hand, is a model merging technique whose definition falls in line with other model merging methods, similar to that in multi-task literature — that is, the model weights are merged and inference is done without additional training. As mentioned in our response to Q1, what sets LegoMerge apart from other merging techniques is that it does not require equivalent architectures between the individual models being merged. As such, it can be readily applied in any scenario, be it merging multimodal from heterogeneous modalities (e.g., image + tabular) or merging multiple different models on a single dataset (e.g., combining multiple vision foundation models trained on separate distributions).

Thank you for the prompt response and clarification of your concerns. We further address the outlined concerns (C1-C4) below.

[C1] Clarification on paper contributions

We would like to further clarify several aspects of our work and our contributions. As we discuss in our paper (L67-90, 118-161) and outlined in our previous response (Q1), there is a difference between multimodal learning (and fusion strategies) and model merging. The former refers to a general task of learning from different data modalities that capture different information, often from different sources and formalised differently (eg. image, table, text etc.). Here, we can broadly differentiate between early, intermediate and late fusion, which determines how and where in model architecture the cross-modal signal is learnt and combined. Model merging, on the other hand, traditionally focussed on improving predictive performance by combining trained models trained on the same input data (often on different tasks [1,2]) without retraining or fine-tuning them. This is done by linear interpolation or combination of the model’s weights (not just the outputs of the model) and is feasible in a multi-task setting, as the input data is the same across tasks, meaning that the same architecture with a different task-specific head can be used. Meanwhile, in multimodal settings, we are dealing with different input distributions and shapes for each modality, meaning that the same architecture can often not be used across modalities.

Your observation "However, if we adopt the late-fusion strategy (feature fusion), there is no need to require the network architecture to be the same. Moreover, if we directly extract the feature for each modality and then train a classifier to combine these features, there is also no need to train the framework end-to-end." is partially true. Indeed, you can train a task-specific head on the combined feature representation of unimodal encoders where the original encoder weights can be both trainable or frozen. This is reflected in our experiments (see SNN+ABMIL in Table 1) and the discussion (Table 2). However, there are several reasons why the motivations of the paper are still justified:

• Strictly speaking, late fusion is not a model merging method: late fusion operates on the output space (i.e., combining the outputs of the model) while model merging operates on the parameter space across the models.
• Cross-modal interactions: a common problem with late fusion methods is that they struggle to pick up cross-modal interactions [3, 4, 5]. In some cases, this goes against the motivation of integrating multiple modalities in the first place, as we try to gain additional predictive performance from these interactions. Many intermediate fusion methods address this problem and, in fact, LegoFuse is an intermediate fusion approach.
• Performance: Our comparison to an ensemble (similar to what the reviewer is proposing) shows that MM-Lego is more performant (see Figure 4) and robust (see Appendix F). We believe this approach to be novel and very practically relevant.

[C2] Motivations of our methods

All of the aspects highlighted in L201-222 in the paper and our earlier response (Q3) are properties of the Fourier Transform which motivate and justify our decision to work with frequency-domain representations. Specifically, besides signal preservation, the following properties are desirable for model merging:

• Efficiency: It has a time complexity of $\mathcal{O}(n log(n))$ making it scalable to very large datasets, as it’s common in multimodal scenarios.
• Invertibility: This allows us to base the design of MM-Lego on an iterative architecture, which offers additional benefits (see C4, Q2 and Section 6). It follows from the Fourier Inversion Theorem [6] that we can efficiently invert frequency-domain representation to its original function.

Thank you for your thoughtful review and encouraging outlook on our work.

[Q1] On the necessity of frequency-domain representations

To emphasise the signal interference in other fusion methods in the spatial domain (compared to the MM-Lego frequency domain), we would like to further elaborate on some of the results in Table 1 (filtered below) which show the impact of the frequency-domain merge. For the four TCGA datasets, we used the SNN and ABMIL encoder wrapped in the LegoBlock and compared it to bilinear fusion (a commonly used end-to-end trained fusion method in the spatial domain). For the BLCA, BRCA, and KIRP datasets, LegoMerge (where we do not use any end-to-end training) outperforms the fusion method that requires paired training data. Meanwhile, the bilinear fusion performs worse than the best unimodal model on (BLCA, BRCA, and UCEC). These results improve further when using LegoFuse (see Table 1 of the paper).

We would like to offer additional theoretical intuition on why the frequency domain representations do not suffer signal interference. Several related works in different domains and modalities [1, 2, 3] emphasise the importance of fusion or aggregation methods that avoid signal interference, where meaningful patterns or complementary information from individual modalities may be attenuated, distorted, or lost due to naive aggregation approaches such as averaging or summation. For example, bilinear pooling of two latent vectors $h_1 \in \mathbb{R}^n$ , $h_2 \in \mathbb{R}^m$ and resulting tensor $h_{12} \in \mathbb{R}^d$ ($y=h_1^T A h_2 + b$) with a learnable parameter $A \in \mathbb{R}^{n \times m \times d}$ follows the same motivation. With the same motivation in mind, we take advantage of the Fourier transform’s orthogonal properties, as after the transform each frequency component (represented by sine and cosine waves) is orthogonal to the others, such that $\int_{-\infty}^{\infty} sin(\omega_1h_1) cos(\omega_2h_1) dh = 0$ for angular frequencies $\omega_1 \neq \omega_2$ [4]. This means that the contribution of each frequency is independent of others and there is no overlap between them. It follows that if we take the harmonic mean of two fourier-transformed latents $\mathcal{F}_1(\omega)$ and $\mathcal{F}_2(\omega)$ as $H(w) = \frac{2\mathcal{F}_1(\omega)\mathcal{F}_2(\omega)}{\mathcal{F}_1(\omega)+\mathcal{F}_2(\omega)}$, the harmonic mean aggregates each $\omega$ in a localised manner. This means that each frequency component in $\mathcal{F}_1(\omega)$ interacts only with the corresponding frequency component in $\mathcal{F}_2(\omega)$, i.e., without interference from other frequencies.

Beyond these results and the theoretical grounding, which we have emphasised in the paper revision, we provide two empirical examples of signal interference in latent variables in Appendix E of the manuscript.

[Q2] Extension of Table1

Thanks for the suggestion. We updated Table1.

[Q3] Merging more than two modalities

Thanks for pointing this out. All experiments performed on TCGA (i.e., BLCA, BRCA, KIRP, UCEC) already contain four different modalities. While this only contains two different heterogeneous data structures (tabular and imaging), the three tabular data sources are treated as separate modalities in the model. We briefly mention in L824 that all TCGA experiments use mutations (binary variables), copy number variations (categorical variables), and gene expression (continuous variables). Nevertheless, we acknowledge that the use of Modalities={tab, img} in Table 1 is ambiguous. We updated the manuscript accordingly to reflect that this is running on four modality-specific encoders.

References

[1] Yu, Z., et al., 2017. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In Proceedings of the IEEE international conference on computer vision (pp. 1821-1830).

[2] Zadeh, A., et al., 2017. Tensor Fusion Network for Multimodal Sentiment Analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (pp. 1103-1114).

[3] Chen, R.J., et al., 2020. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging, 41(4), pp.757-770.

Thank you for your thoughtful comments and suggestions for improvement. In the following, we respond to your comments with additional explanations of the raised issues. We also include these clarifications and additions in the manuscript.

[W1] No experiments with vision & language models

The clear focus of MM-Lego is to look at biomedical data domains where 1) modalities do not share semantics (i.e., an image and its description being a projection of the same concept), 2) we do not have a lot of paired data, and 3) we may have one-to-many cardinalities between the samples (e.g., one tissue sample with multiple DNA/RNA sequencing reads). These datasets are very common in biological and medical research domains and require fundamentally other learning paradigms to vision & text benchmarks, where most of the learning is happening on paired data using contrastive learning. We outline this distinction in L59-66, L166, and the discussion in L411-414, but made this aspect more prominent in the revised version of the manuscript.

[Q1] Experiments on CheXpert

Thank you for the suggestion to run experiments on CheXpert - we would like to point out that the dataset paper you mentioned is a multi-task vision-only benchmark containing “224,316 chest radiographs of 65,240 patients labelled for the presence of 14 common chest radiographic observations” [1]. While we are happy to run additional benchmarks, it is not obvious to us how this allows for further insight into the multimodal focus of the paper. This will also be less applicable to the multimodal fusion benchmarks, which expect multiple modalities (otherwise they are just a unimodal encoder).

[Q2/W2] Merging more than two modalities

All experiments run on TCGA (i.e., BLCA, BRCA, KIRP, UCEC) already contain four different modalities. While this only includes two specific data structures (tabular and imaging), the three tabular data sources are treated as separate modalities in the model. We briefly mention in L824 that all TCGA experiments use mutations (binary variables), copy number variations (categorical variables), and gene expression (continuous variables), but acknowledge that the use of Modalities={tab, img} in Table 1 is therefore ambiguous. We updated the manuscript accordingly to reflect that this is running on four modality-specific encoders. Additionally, we would like to point to the discussion in L465-478 which discusses the time complexity of scaling to a large number of modalities.

References:

[1] Irvin, J., et al., 2019, July. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 590-597).

[Q9] Implementation Details

The paper is accompanied by a code base (see zipped supplementary materials) which will make it easier for readers to reproduce the experiments. The current supplementary materials contain the main code module of the codebase and the full repository will be made available upon publication, including all details on hyperparameters and run instructions. We will also make sure to add further information that can currently only be found in the repository to the final manuscript. Answers to the additional questions:

• Encoders for TCGA slides: ResNet50 fine-tuned on Kather100k [7], a large collection of histopathology data, mentioned in L838.
• Mortality: The MIMIC-III mortality label corresponds to in-hospital mortality, i.e., to whether a patient died during the hospital stay. More information about the task label can be found on the physionet website.
• Time series data: We use the same Perceiver as the one used in the Unimodal baseline. The exact Perceiver configurations can be found in the config files of the repository.

[Q10] Comparable effort with concatenated multimodal embedding

• “[…] missing a comparable effort to tune a neural network that processes the concatenated multimodal embeddings”: The SNN+ABMIL (CC, Late) baseline in Table 1 does exactly this, i.e., it is using an Attention-based Multiple Instance Learning model for the whole slide images and encodes all tabular modalities with a self-normalising network. The resulting embeddings are concatenated and passed into task-specific fully-connected layers. This baseline is tuned end-to-end (with the exception of the vision encoder).
• “[…] one could employ a multi-layer perceptron with S layers and an attention mechanism at each layer”: This suggestion closely corresponds to the MultiModN [8] and HEALNet [9] baselines. MultiModN uses multiple iterative MLPs as linear projections of modalities into a shared, fixed-size space. HEALNet uses a similar iterative setup coupled with an attention layer for each update. The baselines were specifically chosen since they are a relevant comparison of an end-to-end trained model to our setup.

[Q11] Experiments with >2 modalities

All experiments run on TCGA (i.e., BLCA, BRCA, KIRP, UCEC) already contain four different modalities, namely WSIs, mutations (binary), copy number variations (categorical), and gene expression (continuous) (see L824). While this only contains two specific data structures (tabular and imaging), the three tabular data sources are treated as separate modalities in the model. We acknowledge that the use of Modalities={tab, img} in Table 1 is therefore ambiguous. We will update the manuscript accordingly to reflect that this is running on four modality-specific encoders.

[Q12] Missing Data Experiments

The missing data experiments are currently only implemented on the MIMIC dataset since this was the only paired dataset from our experiments that provides sufficient samples for this ablation study. The pathology datasets from TCGA have relatively few samples with all four modalities present, meaning that the subsample required to create the symmetric difference of overlapping modalities would be too small to train any stable model. However, the authors commit to looking for additional datasets that would allow for additional missing modality ablation studies in the main body before the camera-ready deadline.

[Q13] Modality order in LegoFuse

During the development of LegoFuse, we performed preliminary analyses on the influence of the order of modalities on the downstream performance: We did not find any significant difference. Since all modalities are encoded in the latent array and contextualise one another as long as the number of steps is $s>1$, we have not found any evidence that the order of modalities matters.

[Q14] Combining Table 1 and Table 4

Table 1 and Table 4 present slightly different presentations on mostly identical data. The main difference is that the LegoMerge performance is presented differently in Table 4. In hindsight, the authors agree with the reviewer’s sentiment that they should be combined and will update the manuscript accordingly.

Thank you for your in-depth review and valuable feedback. We are very encouraged that you describe MM-Lego as timely, well-written, and acknowledge the core idea as interesting and novel. In the following, we provide detailed answers to your comments with additional explanations of the raised issues. We will also include these clarifications and additions in the manuscript.

[W1/Q1] Motivation for frequency-domain representations:

We would like to further elaborate on the motivation to use frequency-domain representations which the paper touches upon in L203-221. We will add a more detailed discussion in the paper appendix.

• Signal preserving: The general motivation for signal-preserving aggregation for multimodal machine learning has been extensively studied in the context of fusion methods. Several related works in different domains and modalities [1, 2, 3] emphasise the importance of fusion or aggregation methods that avoid signal interference, where meaningful patterns or complementary information from individual modalities may be attenuated, distorted, or lost due to naive aggregation approaches such as averaging or summation. For example, bilinear pooling of two latent vectors $h_1 \in \mathbb{R}^n$ , $h_2 \in \mathbb{R}^m$ and resulting tensor $h_{12} \in \mathbb{R}^d$ ($y=h_1^T A h_2 + b$) with a learnable parameter $A \in \mathbb{R}^{n \times m \times d}$ follows the same motivation. With this motivation in mind, we take advantage of the Fourier transform’s orthogonal properties, as after the transform each frequency component (represented by sine and cosine waves) is orthogonal to the others such that $\int_{-\infty}^{\infty} sin(\omega_1h_1) cos(\omega_2h_1) dh = 0$ for angular frequencies $\omega_1 \neq \omega_2$ [4]. This means that the contribution of each frequency is independent of others and there is no overlap between them. If we take the harmonic mean of two fourier-transformed latents $\mathcal{F}_1(\omega)$ and $\mathcal{F}_2(\omega)$ as $H(w) = \frac{2\mathcal{F}_1(\omega)\mathcal{F}_2(\omega)}{\mathcal{F}_1(\omega)+\mathcal{F}_2(\omega)}$, the harmonic mean aggregates each $\omega$ in a localised manner. This means that each frequency component in $\mathcal{F}_1(\omega)$ interacts only with the corresponding frequency component in $\mathcal{F}_2(\omega)$, i.e., without interference from other frequencies.
• Distance preserving: Parseval’s Theorem shows that the Fourier transform is a unitary operator, meaning that the sum or integral of the square of a function is equal to the sum or square of its transform [4,5]. As such, the distances between two signals are the same between the transformed and untransformed representations.
  • Concretely, the theorem states that the energy of a signal is preserved in both the time domain and frequency domain, where its energy is measured as the integral of the function. Formally $\int_{-\infty}^{\infty} |f(h)|^2dh = \int_{-\infty}^{\infty} |\mathcal{F}(f)(\omega)|^2 d \omega$ for latent signal $f(h)$ and its fourier transform $\mathcal{F}$.
  • The Euclidean distance between two signals $f_1(h)$ and $f_2(h)$ in the spatial domain is: $||f_1-f_2|| = \sqrt{\int_{-\infty}^{\infty}|f_1(h) - f_2(h)|^2 dh }$
  • The Euclidean distance between the Fourier transforms of two signals is:
  • $||\mathcal{F}(f_1) - \mathcal{F}(f_2)|| = \sqrt{\int_{-\infty}^{\infty} | \mathcal{F}(f_1)(\omega) - \mathcal{F}(f_2)(\omega)|^2 d\omega}$
  • From Parseval’s Theorem, it follows that $||f_1 - f_2 || = ||\mathcal{F}(f_1) - \mathcal{F}(f_2)||$.
  • This distance-preserving capability is beneficial in designing loss functions in a multimodal setting.
• Invertible: The Fourier transform is not an idempotent function but periodic with a period of 4, i.e., $\mathcal{F}^4(f) = f$. This would prevent the iterative architecture outlined in Section 3 from working since a repeat application of the transform would lead to different representations. Meanwhile, the Fourier Inversion Theorem [6] shows that we can invert the frequency-domain representation to its original function without the 4x repeat application, making it suitable for the chosen iterative architecture.
• Efficient: The Fast Fourier Transform (FFT) has a time complexity of $\mathcal{O}(n log(n))$ making it scalable to very large datasets.

[Q2] - “LegoMerge does not require any fine-tuning” is ambiguous

The authors would like to clarify the ambiguity that may arise from this statement. For MM-Lego’s architecture, we broadly have two sets of parameters:

• The original encoder weights: a higher number of parameters (often 100M+ depending on the foundation model used), if we unfreeze and retrain these, we refer to this as “fine-tuning”.
• The weights for the LegoBlock adapters: a lower number of parameters (typically <1-2M) which need to be fitted in a supervised way. This can be done in two ways: 1) retrospectively (for large foundation models) or 2) during training of the original model as a wrapper (which Table 3 refers to).

The statement you are referring to is correct in the latter case where the model was already wrapped with the LegoBlock during training. When fitted retrospectively, this still uses significantly fewer parameters than when using fine-tuning. As an illustrative example, fine-tuning just the final three layers of a ViT-large model would tune about 22M parameters, while fitting the LegoBlock adapter is more parameter efficient with approx. 2M parameter. As such, the statement and values reported in Table 3 are correct in the second case, while they are not in the first. We will update Table 3 for both training modes and emphasise this important distinction.

[Q3] - $h^{(A)}$ or $\mathcal{F}(h^{(A)})$ in attention mechanism

The attention mechanism is applied in the frequency domain, as indicated in Figures 1 and 2. We agree that Equation 3 is inconsistent with that representation, which is now corrected to fall in line with Figure 2 such that $(L_{t+1}^\mathcal{F})^r = \text{softmax}\left(\frac{(L_t^\mathcal{F})^r W_m^q \cdot (\mathcal{F}(h^{(A)})^r W_m^k)^\top}{\sqrt{d_k}}\right) \cdot (\mathcal{F}(h^{(A)})^r W_m^v)$ where $\mathcal{F}(\cdot)^r$ denotes the real component of the Fast Fourier Transform.

[Q4] - $L(A)$ as LegoBlock input

This is further illustrated through Figure 2 and the architecture section in L173-185. As shown in Equation 1 and 3, each cross-attention layer takes in the $L_s$ and outputs the update $L_{s+1}$. The same update is reflected in Figure 1.

[Q5] - Initialisation of $L_0$

$L_0$ is randomly initialised as a nn.Parameter() with the specified latent dimensions and is with the corresponding modality in each cross-attention pass.

[Q6] - Number of update steps

Across all experiments, we use a depth of S=4 which we found to be stable. We initially ran a grid search across crucial hyperparameters for the architecture such as the number of update steps $S$ and the dimensions of $L$. We will update the Appendix with a table of all hyperparameters used for the experiments, which can also be found in the supplementary materials and further implementation details can be found in the codebase that would be made public upon publication.

[Additional concern 2] Responses to Q7 and Q8 of the original questions

Apologies and thanks for pointing this out — it appears that these responses were not posted as part of our original response:

[Q7] - Different dimensions of embeddings

We would like to emphasise that the manuscript states a 1D encoding for tabular data in L302 and L840. We illustrate the Fourier transform in Equation 2 on 2D Tensors since both the image embedding of the whole slide images uses a bag of patches ($n_{patches} \times dim_{patch}$) and the latent bottleneck $L \in \mathbb{R}^{c \times d}$ is two-dimensional. For the one-dimensional tensors, the Fourier transform in Equation 2 is adjusted accordingly. Since each modality gets its own FusionBlock, the transform can be adjusted to the dimensions of each modality’s embedding, while the dimensions of the latent bottleneck remain consistent across modalities. This makes the approach flexible to work with different encoder topologies.

[Q8] Repeated Fourier and inverse transforms

No artefacts are occurring through the repeat Fourier conversion, as pointed out in the Fourier Inversion Theorem [6] mentioned in an earlier response. To the contrary, the repeat conversion is necessary to prevent the occurrence of artefacts during the training process:

• Tracking complex numbers: The Fourier transform $\mathcal{F}(h)$ decomposes the latent into its frequency components which are represented by complex numbers, each of which has a real $(h^{\mathcal{F}})^r$ and imaginary component $(h^{\mathcal{F}})^i$. Most machine learning frameworks only operate on real numbers, meaning we cannot pass the imaginary component into the cross-attention layer. However, dismissing the imaginary component entirely would discard all sine-wave contributions from the signal’s frequency domain which would lead to a drastically distorted signal after the inverse transform $\mathcal{F}^{-1}$. Therefore, we keep track of the imaginary component and use it in the inverse transform after each update. This process is illustrated in Figure 2.
• Artefacts through repeat transforms: As outlined in an earlier response, the FFT is a periodic transform and applying it repeatedly would lead to different representations for different layer passes of the model, making it practically impossible for the model to pick up any patterns.

Thank you very much for the prompt response to our comments and revising your original score. We would like to address the remaining concerns below:

[W1] Experiments to demonstrate the utility of the proposed method

Based on the comment, we would like to clarify a suspected misunderstanding: the current experimental setup does not use “tabular data […] to aggregate information from several domains”, i.e., the experiments do not take features from different sources and put them in a centralized feature table. For instance, in the case of TCGA, each tabular modality is a high-dimensional vector and uses a modality-specific encoder. Therefore, each modality in the TCGA experiments obtains its own encoder such that we have:

• WSI encoder
  • Sample Input: Raw image converted to a bag of patches, $d_{wsi} = n_{patches} \times 256 \times 256 \times 3$
  • Encoded input using ResNet50, pre-trained on Kather100k : $h_{wsi} = n_{patches} \times 2048$
• Mutation encoder
  • Non-variable mutation genes were filtered (i.e., every sample contains mutation or none contains mutation)
  • Input: Raw input mutation data vector for cross-attention
  • Encoder: none as the vector is already relatively small after ETL: $d_{mut} = h_{mut} = 21$
• Copy Number encoder:
  • Non-variable copy number genes were filtered
  • Input: Copy number variations for each gene (categorical variable ranging -2 (deep deletion) to +2 (strong oncogenic amplification): $d_{cnv} = 1333$
  • Encoder: SNN (1333 → 512): $h_{cnv} = 512$
• Gene expressions
  • Low variable genes were filtered
  • Input: log1p transformed bulk RNAseq gene expression: $d_{rna} = 1558$
  • Encoder: SNN(1558 → 512): $h_{rna} = 512$

Note that these relate to the Breast Invasive Carcinoma (BRCA) patients as an example. The implementation details for all TCGA and MIMIC encoders will be available in the appendix and code.

Therefore, MM-LEGO indeed allows for combining specialized embeddings from different encoders across the modalities, such as using (pre-trained) ResNets for image and SNN for tabular data, and using the raw input (no encoding).

We hope that this addresses your primary remaining concern.

[Additional concern 1] Starting latent initialization as $L^{(A)}$

Thank you for the clarification. We recognize the confusion with the notation on the RHS of Figure 1. Indeed, the starting initialization in both the “stack” and “weave” cases should be $L_0$ rather than a domain-specific latent. This latent will then be updated in the iterative pass through the LegoBlocks. Thank you again for pointing this out - we updated the figure accordingly.

For all reviewers,

We would like to thank all reviewers who have participated in the discussion so far and send a friendly reminder to those who have not. We uploaded a revised version of the manuscript that incorporates the changes requested by all reviewers, highlighted in blue.

We are looking forward to a productive discussion for the remaining period.