Aligning Multimodal Representations through an Information Bottleneck

We would like to begin by thanking you for the time dedicated to give us feedback to improve our work. We address your main concerns next:

these datasets seem to be rather old (from 2015 and 2002).

We believe that you may be refering to the metrics instead of to the datasets. These are still widely used despite of the fact that they are old. See for example [1] and [2], two recent papers with a great impact in the field.

[1] Li, J., Li, D., Savarese, S., & Hoi, S. (2023, July). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.

[2] Team, G., L. (2023). Gemini: a family of highly capable multimodal models.

the role of Theorem 1 is rather unclear.

Theorem 1 states that sufficiency is a necessary condition to solve all the downstream tasks (that can be derived from the essence). In other words, if a representation is not sufficient, then it cannot solve all the downstream tasks. The opposite is also true, but we do not find it so valuable. Having a representation that can potentially solve all the downstream tasks is not valuable because it does give no information on how “easy-to-use” the information to solve this task is. In the extreme case, the input can be seen as a representation of itself (since it satisfies Definition 3) and it can potentially (through an oracle model) solve any downstream task. Thus, obtaining a representation that can potentially solve all the downstream tasks is not meritorious nor theoretically valuable. However, the importance of this theorem lies in the fact that any representation that is not sufficient cannot be used to perfectly solve any downstream task, no matter how “easy-to-use” the information it contains is, which enhances the value of sufficient representations. This idea connects to that of usable information [3], which allows to formulate this differentiation between having information present in a representation and how “easy-to-use” this information is. This clarification can be made in the paper.

[3] Xu, Y., Zhao, S., Song, J., Stewart, R., & Ermon, S. (2020). A theory of usable information under computational constraints. arXiv preprint arXiv:2002.10689.

Consider using the mathrm or texttt...

mathrm will be used.

If we normalize the output representation $\mu$ ...

In the case in which the embeddings are unit-norm, our loss becomes $\mathcal{L}_i = \log\frac{\exp(s\_{ii}/\tau)}{\sum_k \exp(s\_{ik}/\tau)} + 2\beta(1-s\_{ii})$, where $s\_{ik}$ is the cosine similarity between $z^{(i)}$ and $z^{(k)}$. We have the following:

• $\frac{\partial{\mathcal{L}_i}}{\partial s\_{ii}} = -\frac{1}{\tau} \left(1 - \frac{\exp(s\_{ii}/\tau)}{\sum_k \exp(s\_{ik}/\tau)} \right) - 2\beta$
• $\frac{\partial{\mathcal{L}_i}}{\partial s\_{ij}} = \frac{1}{\tau} \frac{\exp(s\_{ij}/\tau)}{\sum_k \exp(s\_{ik}/\tau)}$

We also analyze the gradients of a modification of the $\mathrm{InfoNCE}$ in which a different temperature is used for the numerator and the denominator, i.e., $\mathcal{L}'_i=\log\frac{\exp(s\_{ii}/\tau')}{\sum_k \exp(s\_{ik}/\tau)}$. Then, it can be easily checked that:

• $\frac{\partial{\mathcal{L}'_i}}{\partial s\_{ii}} = -\frac{1}{\tau'} \left(1 - \frac{\exp(s\_{ii}/\tau')}{\sum_k \exp(s\_{ik}/\tau)} \right)$
• $\frac{\partial{\mathcal{L}'_i}}{\partial s\_{ij}} = \frac{\partial{\mathcal{L}_i}}{\partial s\_{ij}}$

Thus, we have that optimizing our loss is equivalent to optimizing a modification of $\mathrm{InfoNCE}$ with different temperature in numerator and denominator. Rearranging, $\beta=\frac{1}{2} \left[ \frac{\tau-\tau'}{\tau\tau'} + \frac{\exp(s\_{ii}/\tau) - \exp(s\_{ii}/\tau')}{\sum_k \exp(s\_{ik}/\tau)} \right]$. Thus:

• The difference in the temperature between numerator and denominator depends on the similarity with respect to all the elements in the batch.
• If $s\_{ii} \ll \sum_k \exp(s\_{ik}/\tau)$, i.e., the predictions are far from the target distribution, then $\beta \approx \frac{1}{2} \Delta\tau$, where $\Delta\tau = \frac{\tau-\tau'}{\tau\tau'}$. Thus, the larger the $\beta$, the larger is the difference between the numerator and denominator temperatures.
• If $s\_{ii} \gg \sum_k \exp(s\_{ik}/\tau)$, i.e., the predictions are close to the target distribution, then $\beta \approx \frac{1}{2} \left[ \Delta\tau + 1 - \exp\Delta\tau \right]$. Thus, the temperature difference between numerator and denominator for a given value of $\beta$ is lower than in the previous case.

Thus, our term can be seen as a regularizer the adapts the value of the temperature in the denominator based on the how close to the true distribution the prediction is.

These comments will be added in an appendix. Thank you for your interest. Please let us know if you have any other insight with respect to this last analysis.

We hope that your questions have been addressed. If this is the case and you consider that our paper deserves an increase in the score, we would thank you if you made it effective.

We would like to begin by thanking you for the time dedicated to give us feedback to improve our work. We address your main concerns next:

could the choice of image encoder introduce bias in the URR measurements?

To analyze this point, we have performed experiments that are identical to those in Table 1, but using a small ViT as image encoder. We show the results next:

By comparing the latter and Table 1 we can observe that:

1. Local attributes, such as Location and Size, are better preserved in convolutional encoders, which is consistent with the inductive biases towards local structures of convolutional layers.
2. Global attributes, such as Shape and Objects Color are almost equally conserved in both image encoders.

Subsection 5.1 would be extended with these two experiments and conclusions. Thanks for noticing.

how are invariance and equivariance with respect to different factors accounted for in this analysis?

Sorry, we do not understand this question, could you please further develop it?

I recall an ICLR 2021 paper...

We think that the paper that you could be referring to is [1]. If this is the case, your question is very interesting and points of [1] and ours perfectly compatible.

On the one hand, what the mentioned paper states is that, if the task to be solved is correlated with other simpler features, then a model trained with SGD will tend to learn these simpler features rather than the task. If we relate this idea with the concepts used in our work, the point of [1] would be something like: “given a task Y, models trained with SGD tend to learn minimal sufficient statistics of Y instead of Y”. For example, if our task were to classify between images of bananas and strawberries, our model would simply learn to differentiate between yellow and red.

On the other hand, our paper argues that, even given the previous statement, models tend not to remove all the information that is unnecessary to solve the task (i.e., nuisances). In the previous example, a model could be learning, for instance, the background color of the images. Thus, our model could learn the fruit color and the background color, which makes the point of [1] and ours perfectly compatible.

If this is the paper you refered to, thanks for bringing it up, its results are very interesting. If this is not the case, we are open to discuss other works.

[1] Ahmed, F., Bengio, Y., Van Seijen, H., & Courville, A.. Systematic generalisation with group invariant predictions.

For the Information Homeostasis..., could this phenomenon be linked to issues inherent to softmax formulations?

First, for higher values of $\beta$, representations tend not to retain the information that is unique to their input. Then, all the representations are more similar to each other and closer in the space. Thus, maybe predictions are less sharp, so the temperature is decreased when $\beta$ increases in order to maintain the same level of sharpness in the predictions no matter the value of $\beta$. Thus, an hypothesis that could connect to Veličković et al. (2024) is the fact optimization processes "prefers" a given level of sharpness. Also, since the level of sharpness changes with the batch size, using Sigmoids instead, as Zhai et al. (2023) suggests, would make the temperature to be invariant to changes in $\beta$. Intuitively, since the rest of the similarities are ignored for the Sigmoid, the fact that representations of negative pairs are closer in the space, should not affect the predictions. This point is pure speculation and excessive importance should not be attributed to it.

I think Equation 18 should be derived in the appendix.

The derivation of this equation will be included in an appendix.

Table 2 needs to show std by running repeated experiments with different seeds.

Please see the second answer to Reviewer DSZo.

I think preliminaries can be cut down to related work without the equations.

We agree that the expressions of HSIC and CKA are not necessary for the understanding of the paper. They will be removed from the paper. Thanks for noticing.

Also, it would be great if you annotated the essence and nuisances in the examples.

Essence and nuisances will be annotated in the examples. We suppose that you refer to the examples in section 3. Please let us know if you refer to other examples.

We hope that your questions have been addressed. If this is the case and you consider that our paper deserves an increase in the score, we would thank you if you made it effective. If this is not the case, we are open to continue with the discussion in the next stage of the rebuttal process. Similarly, if you could elaborate on the question that we were unable to answer, we will try to address it.

We would like to begin by thanking you for the time dedicated to give us feedback to improve our work. We address your main concerns next:

1. Quantitative results of image retrievals are missing.

What will the model perform when having a stronger $L_M$ constraint?

Please see second answer to Reviewer DSZo.

2. providing the quantification of essence and nuisances and real-world dataset will strengthen the soundness of the paper.

Quantifying the essence and nuisances is impossible in general because they are abstract variables that we simply define for our formulation. For this reason (and also because computing mutual information is expensive), it is in general impossible to quantify the essence and nuisances in the representation. In fact, this is the reason why we use variational approximations for the derivation of our loss function. If it were possible to exactly calculate the amount of the essence or nuisances in the representations, then we could simply maximize and minimize these quantities, respectively. The toy datasets are precisely used to set by ourselves the essence and nuisances and having a scenario in which it is straightforward to calculate a reliable estimation of the mutual information between the essence (or the nuisances) and the representations.

The ideas of essence and nuisances are similar to the unique and shared information concepts in [1].

Thank you for your recommendation. This paper will be included in the related work section.

Is the spherical Gaussian assumption reasonable for general multimodal representation learning?

On the one hand, the use of Gaussian distributions in the representation space is not an assumption but a design choice that we make mainly for tractability reasons of the KL divergence in equation 17. We argue its reasonability next.

If the data are non-Gaussian, this assumption might be questionable.

On the other hand, the fact that the data distribution $p(x)$ is not Gaussian should not be problematic. For example, most of the VAEs (or other SoTA generative models, such as Diffusion Models and Flow Matching) use a encoder $p_\theta(z|x)$ that is Gaussian (also mainly for tractability reasons of the KL divergence) and they provide impressive performance in real-world applications, in which the data $p(x)$ is far from being Gaussian.

...it is difficult to justify that the learned representations strictly follow a simple Gaussian distribution.

Finally and most importantly, we must note that choosing the encoder $p_\theta(z|x)$ as Gaussian, does not imply at all that the representation space is Gaussian. What is Gaussian is the distribution of an embedding given a single input $p_\theta(z|x)$, but not the whole representation distribution $p_\theta(z)$.

More specifically, we have that $p_\theta(z)=\int p_\theta(z|x)p(x)dx$. If we approximate the data distribution $p(x)$ as its empirical distribution given by $N$ datapoints, i.e., $p(x) \approx \sum_{i=1}^N \delta\left(x-x^{(i)}\right)$, then the distributions of the representation space $p_\theta(z)$ becomes a Gaussian mixture, i.e., $p_\theta(z) \approx \frac{1}{N} \sum_i \mathcal{N} \left(z; \mu_\theta\left(x^{(i)}\right), \sigma^2 I\right)$, which "is a universal approximator of densities, in the sense that any smooth density can be approximated with any specific nonzero amount of error by a Gaussian mixture model with enough components" [1].

Furthermore, we note that using a stochastic encoder (even if it is Gaussian) results in a distribution that is richer than in the case of deterministic encoders (i.e., vanilla encoders). In the latter, $p_\theta(z|x)$ is a delta distribution, i.e., simpler (one parameter) than a gaussian (two parameters) and using deterministic encoders is rarely seen as problematic.

[1] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning., p. 65

Eq. (20) assumes identical covariances for the two modalities, which is even less realistic given typical differences in modality-specific representation distributions.

Following with the previous answer, this should not be problematic. In fact, having the same covariance matrices in the representation spaces of both modalities could be considered as desirable since we would like these spaces to be as similar (or aligned) as possible.

The notation $t$ is not explained.

$t$ means a realization of the task $T$. In probability theory, random variables are usually denoted by uppercase letters while their realizations are denoted by their corresponding lowercase letters (see https://en.wikipedia.org/wiki/Notation_in_probability_and_statistics). We will add a paragraph in section 2 to clarify this.

We hope that your questions have been addressed. If this is the case and you consider that our paper deserves an increase in the score, we would thank you if you made it effective. If this is not the case, we are open to continue with the discussion in the next stage of the rebuttal process.

We would like to begin by thanking you for the time dedicated to give us feedback to improve our work. We address your main concerns next:

(i) additional multimodal benchmarks would enhance generalizability.

More experiments were not included due to space limitations.

(ii) task-specific metrics beyond alignment would strengthen the assessment.

In section 5, we have Fig.6 for this purpose. For section 6, we have calculated retrievals, shown next.

We have that:

1. Text generation (TG) and retrieval performances are inversely correlated. This makes sense, since TG gets benefited from minimal representations and retrieval from sufficient representations (retrieval is a specific type of downstream task). This inverse correlation can be observed also in Fig. 6.
2. Our loss increases TG performance for low or medium values of $\beta$, since its goal is to increase the minimality of representations.
3. For $\beta=0.3$, the performance starts to excessively decrease the part of the essence that representations retain, similarly to in Fig. 6.

This would be included and better explained in the final version.

additional experiments to isolate confounding factors would strengthen this argument.

We agree that the experimental setup that analyzes the Information Homeostasis phenomenon does not serve to demonstrate any causal relationship. However, this analysis would require more space and, additionally, it could confuse the main line of the paper.

the paper does not compare its method against several state-of-the-art techniques designed for alignment...

We compare in section 6 our loss function with the ITM, which is used in most of the state-of-the-art methods to obtain alignment between text and image modalities. As explained in lines 379-382, ITM is defined for a very specific architectural choice so it cannot be used in experiments in section 5.

a formal guarantee on how different encoders may lead to equivalent partitions in practical settings could be added in Lemma 1.

We are not sure to understand this point very clearly. Lemma 1 is stated in line 134 and representations are defined in line 161, so there is no notion of encoder or representation at the point in which Lemma 1 is stated. Thus, this lemma is completely independent of the encoders.

Theorem 1 assumes that tasks derived from the essence fully capture all relevant downstream tasks, which may not always hold universally across applications.

This is never assumed. Theorem 1 simply states that sufficient representations are a necessary condition to solve all the tasks that can be derived from the essence. Sometimes, contrastively trained models are used to solve downstream tasks that are not in the essence. However, there are no theoretical guarantees that this should work no matter if the representation is sufficient or not.

The paper assumes that modality-specific information always contributes to misalignmentt, but this may depend on the dataset and task.

Theorem 2 relies on the assumption that perfect alignment can only be achieved if modality-specific information is entirely removed.

This is not an assumption but a theorem that is demonstrated in Appendix A.3. If the demonstration is correct, then it is universally true. Apart from that, alignment is totally independent of the task, since it is an intrinsic property of the representations. We are open to elaborate on this, but we are not sure to understand your point.

Essential References Not Discussed

Thank you for the given references. Some of them were already mentioned in the paper and the rest of them will be discussed in the related work.

how this approach is related to PID.

We believe that the connection is very weak. PID analyzes how two source inputs contribute to the information in a target variable. This is relevant in the cases in which we are using two inputs at the same time, but this is rarely the case in multimodal learning. The main connection that we could make between our work and PID is that, in case that we wanted to use both representations to solve a downstream task, then the redundant information would be equal to zero for minimal representations.

We hope that your questions have been addressed. If this is the case and you consider that our paper deserves an increase in the score, we would thank you if you made it effective. If this is not the case, we are open to continue with the discussion in the next stage of the rebuttal.