Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders

We appreciate your thoughtful and detailed comments. We will respond to your concerns and questions point by point.

Confusion about the concept of sub-sampling modalities

Thank you for pointing this out. In the paper, we use the concept of sub-sampling to refer to ELBO sub-sampling and to the use of mixture distributions to approximate consensus distributions. We will make this clear in the revised version.

Alternative Datasets

Based on your and Reviewer FV9Z comment, we provide results on the CELEB-A data. Due to the limited time, we only considered a latent space with 32D. For CoDE-VAE we assume $\rho=0.6$, as we observed to be a value that performs well in other experiments. We obtained the following results

which may improve further by cross-validating $\rho$ in CoDE-VAE.

Comparison to Deep Generative Clustering with Multimodal Diffusion Variational Autoencoders

Thank you for pointing out this paper, which we were not aware of. We acknowledge the importance of the paper in the field of multimodal VAEs and will discuss it in the related work section and will include the experimental comparison on PolyMNIST in the appendix of the revised version of the paper, as the Clustering Multimodal VAE (CMVAE) model introduced in [2] is not 100% comparable with our proposed CoDE-VAE model. The main focus of our research is to present a novel approach to estimate consensus distributions and to learn the contribution of each ELBO subset, while the goal in CMVAE is to couple multimodal VAEs with clustering tasks by leveraging clustering structures in the latent space and to introduce diffusion decoders, which is certainly a novel and relevant line of research. CMVAE captures clustering structures using a mixture model as a prior, and we hypothesized that this flexible prior in CMVAE plays an important role in the performance of unconditional generative tasks. We will include this discussion and the relation between the methods in the updated manuscript.

Notation and Clarity

Thank you for pointing this out. We will change the notation in the revised version to ensure consistency in the use of subscripts and superscripts, which will improve the clarity of Section 3.

Thorough comparison on the generative quality gap for PolyMNIST

We agree that an average generative quality gap would provide a robust comparison. However, the computational costs of such experiment is significant, as to assess the generative quality for each of the 5 modalities as a function of the number of input modalities requires to train at least 12*3=36 times each model (considering 3 different runs to report standard deviations). This will require 252 runs in total, as there are 7 different models in the evaluation of the quality gap (without considering the unimodal VAEs). Given that our research includes 3 data sets, 6 benchmark models, and several ablation experiments, we let such a robust comparison for future research.

Add grid-search hyperparameters

Thank you for pointing this out. We will add to the Appendix the hyperparameters for the grid search and their optimal values, including the ones for the new experiments on the CELEB-A data.

Is the dependency between expert errors assumed to be constant across the D latent dimensions?

Yes, for simplicity, we assume a common $\rho$ parameter for all dimensions. However, the CoDE is a flexible approach that does not impose any restriction on the way $\Sigma_d$ is specified. We let future research explore whether model performance can be improved by using different correlation values for different dimension.

Beta values and latent space for MMVAE+ on the MNIST-SVHN-Text data.

For all models in Section 4.1, we cross-validate $\beta$ using the grid $[0.1,1,5,10,15,20]$. All models, but MMVAE+, assume that the latent space has 20 dimensions as in previous research. To select the dimensionalty of common and modality specific (MS) variables in MMVAE+, we follow a similar approach as in the original paper, where the authors choose the dimensions of these two to be equal to the dimension of the latent space in MMVAE (a model without MS variables) divided by the number of modalities. Therefore, MMVAE+ assumes that both common and MS variables have 7 dimensions. These details are explained in Appendix D3. We also tested to use 10 dimensions in both common and MS variables, so the decoders in the MMVAE+ model would generate modalities based on 20 dimensions, just as the other models. We did not observe significant differences.

Thank you for your thoughtful and detailed comments. We will address your concerns and questions point by point.

Experiments and analysis on edge cases

We agree that analyzing CoDE-VAE on edge cases helps to understand its behavior, and makes our research more robust. We believe that the experiments in Section 4.2 Generative quality gap (Figure 3), in Appendix D.4. PolyMNIST (Figure 13), and the classification results in Figures 8 and 10 provide a good indication that the performance of CoDE-VAE improves with the number of modalities and with the cardinality of the subset on which consensus distributions are conditioned on.

To address the scenario where the assumption about dependent experts may not hold, we trained the CoDE-VAE model on PolyMNIST using the modality $m_1$ in the following way. We apply 3 different levels of noise to the modality $m_1$, 0 %, 25%, and 95%. Then, we paired the noisy version with the original modality to obtain a bi-modal data. For each of these data, we train CoDE-VAE assuming $\rho=0$ and $\rho=0.9$, and generate the non-noisy version of $m_1$. When CoDE-VAE is trained with the data with 0% noise, both modalities are the same and we expect that $\rho=0.9$ will have relatively high generative quality. On the other hand, when CoDE-VAE is trained with the data with 95% noise, the modalities are uncorrelated and $\rho=0$ is expected to have relatively high generative quality. We obtain the following average FID scores (link1, backup_link) :

showing that CoDE-VAE correctly captures the dependency between experts distributions through the $\rho$ parameter.

CoDE-VAE in cases with missing data or when modalities are not available.

The evaluation setup in our research follows the standard method in multimodal VAEs where all possible combinations of missing modalities are evaluated at test time (Appendix B). To handle missing data is not trivial, as we need to estimate consensus distributions. To estimate $q(z|x_1,x_2)$, any aggregation method would require the same number of observations $x_1$ and $x_2$. This problem could be overcome by using only complete pairs $(x_1,x_2)$, and re-weighting the ELBO terms with fewer samples and could be an interesting direction to pursue in future work.

CoDE-VAE when one modality is much more informative?

CoDE-VAE learns the contribution of each k-th ELBO term to the optimization, balancing the importance of relatively more informative modalities. The empirical results of Section 4.4 show that the text modality in the MNIST-SVHN-Text is relatively more important to the optimization of the ELBO as shown by the weight learned for the subset containing that modality. This result seems reasonable, as there is more noise in the MNIST and SVHN modalities. The ablation experiments of Section 4.5 show that CoDE-VAE achieves higher performance (coherence and FID), when the contribution of each subsets to the optimization, and so each modality, is learned.

CoDE where the assumption of expert dependence does not hold?

CoDE is a flexible approach that does not impose any restriction on the way $\Sigma_d$ is specified as long as it is invertible. For independent modalities, it should be enough to use $\rho=0$ (see answer on edge cases).

CoDE-VAE generative quality. Could you provide more concrete examples?

We recognize that the wording of this claim could be improved and made it more concrete. We have replaced the original sentence with "CoDE-VAE minimizes the generative quality gap as the number of modalities increases, achieving quality similar to unimodal VAEs measured by unconditional FID scores.", which corresponds to the experiments in Section 4.2 Generative quality gap. Furthermore, Figure 3 shows that CoDE-VAE achieves higher generative performance as the number of modalities used for model training increases, something most of the benchmark models are not able to achieve. We have added figures (backup) that show the generated modality $m_0$ as a function of input modalities, as well as generated samples by the unimodal VAE for qualitative comparisons.

Computational complexities of CoDE-VAE

We agree that this is an important aspect to be considered. Therefore, in Appendix C we mention that CoDE-VAE has a relatively high computational cost $\mathcal{O}(2^M-1)$, where $M$ is the number of modalities. However, given that the size of the matrix $\Sigma_d$ depends only on $M$ (see the question from reviewer FV9z), model training is feasible on a single GPU even for 5-modality datasets.

Thank you for your careful and comprehensive comments. We will address your concerns and questions, point by point:

How accurately is ELBO minimized:

We are not completely sure if we follow your concern. If the comment refers to how the ELBO is maximized (the loss), we train the CoDE-VAE model until the ELBO converges, confirmed by visual inspection. These plots (backup backup backup) show the convergence of the ELBO. If your concern is about how close the ELBO is to the intractable marginal log-likelihood, we calculate log-likelihoods on the test sets using importance-sampling (shown in Figures 2 and 4 for all models). Please let us know If we are misunderstanding your concern.

$\Sigma_d^{-1}$ costly in high-dimensional data.

$\Sigma_d$ is not a sample covariance matrix and its size depends on the number of expert distributions assessing consensus distributions (CD). So, it is limited by the number of modalities M, which typically is small and only one CD is conditioned on all modalities. In our research the largest M is 5 (PolyMNIST). Therefore, the computational costs to find the inverse of $\Sigma_d$ are affordable.

$\Sigma_d$ if it is not full rank.

$\Sigma_d$ is guaranteed to be full rank by construction, as $\sigma_{i,j}>0$ for all $i,j$, where $\sigma_{i,i}=\sigma^2_i$. To see this, we need to show that the quadratic form $\beta^T\Sigma_d\beta = 0$, is only satisfied for a zero-vector $\beta$. Let $\kappa$ be the smallest $\sigma_{i,j}$ value, which is positive by construction. Therefore, $\sum_i \sum_j \beta_i \sigma_{i,j} \beta_j > \kappa \sum_i \sum_j \beta_i \beta_j$. Since $\kappa>0$, the only solution that satisfies $\kappa \sum_i \sum_j \beta_i \beta_j=0$ is the zero-vector $\beta$. We will add this discussion in the revised version.

More complex datasets

We added a new section in the appendix with experiments on CELEB-A for CoDE-VAE and MMVAE+, which is the model that stands out in the other experiments. Due to the limited time, we evaluate both models only for $\beta=1$, assuming a latent space with 32D. For CoDE-VAE we assume $\rho=0.6$, as we observed in other experiments to be a value that performs consistently well. We obtained the following results:

which may improve further by cross-validating $\rho$ in CoDE-VAE.

Generation Quality and Classification accuracy

Multimodal VAEs trade high generative quality with reduced generative coherence [1]. The experimental setup in our research shows that CoDE-VAE performs as well as or better than SOTA multimodal VAEs in terms of balancing the trade off between generative coherence and generative quality. When we assess in isolation whether multimodal VAEs are able to improve generative quality as the number of modalities increases, CoDE-VAE clearly shows a higher performance (Figure 3). It is possible to add modality-specific (MS) latent variables to CoDE-VAE to further improve the generative quality, which requires a careful design of the number of dimensions in the common and MS variables to avoid the short-cut problem [1]. When it comes to the classification results, Figures 8 and 10 in the appendix show that models using the mixture-of-experts are not able to achieve higher classification accuracy as latent representations are learned from subsets with more modalities. On the other hand, CoDE-VAE ranks 1st and 3rd, while balancing the trade-off between generative quality and generative coherence at the same time.

[1] Palumbo et al. Enhancing the Generative Quality of Multimodal VAEs Without Compromises, ICLR, 2023.

Typos and L89, Col 1

You are correct that there is a typo in L89, Col 1. The sentence should read "Furthermore, CoDE-VAE minimizes the generative quality gap as the number of modalities increases...", referring to the results in Figure 3. We will fix this in our revised version of the paper.

Softmax instead of categorical

We assume that you are referring to the Gumbel-Softmax distribution, which is useful when we need to sample and backpropagate at the same time. As our main interest is learning the $\pi$ parameters, we do not expect a significant different behavior by using the Gumbel-Softmax distribution. The main concern when using a different distribution, is whether the entropy term in the ELBO of the CoDE-VAE model can be evaluated, ideally in closed form.