Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach

A1. Legend issue in Fig. 1. and non-monotonic performance trend in text shifts.

Thank you for pointing out the visualization issue—we will revise the legend in Fig. 1 to improve clarity and avoid confusion!

• Regarding the non-monotonic trend in win rate under text shifts, this behavior arises in part from the inherent stochasticity of win rate computation based on GPT-4 API evaluations, making it fundamentally difficult to observe strictly monotonic trends in practice [1]. Additionally, the x-axis in Fig. 1 is sorted by embedding space distances—computed using CLIP ViT for visual shifts and XLM-RoBERTa for text shifts—which may not always reflect the true degree of distributional shift.
• That said, we still observe a meaningful overall relationship between embedding distance and performance degradation, both in the 27 natural shifts presented in Fig. 1 and in the 34 synthetic shifts shown in Fig. 6 of the Appendix. By taking inspiration from this empirical analysis, we derive a much more rigorous framework, i.e., EMID upper bound (Theorem 4.5), to quantify the performance gap that consistently shows statistical significance across diverse settings.

A2. Clarification for the meaning of Spearman correlation and Kendall’s tau.

• Spearman's $\rho$ and Kendall’s $\tau$ are both representative measures for monotonic relationships between two variables, where the former is preferred for detecting weak correlation and the latter is preferred to capture strong correlation in small sample sizes and is more robust to outliers with large sample sizes. Both of them are standard approaches to measure the correlation between LLM Judge score and other metrics [2], and are good for measuring the relation of two variables, even if they have different data types, e.g., discrete (win rate) versus continuous (EMI).
• Both correlation coefficients range from -1.0 (negative correlation) to 1.0 (positive correlation), where 0.0 indicates there is no monotonic relationship between two variables. For Spearman's $\rho$, a 0.2-0.4 range of values denotes weak correlation, 0.4-0.6 denotes moderate correlation, and 0.6-0.8 and 0.8-1.0 denote strong and very strong correlations, respectively, and Kendall's $\tau$ can be similarly interpreted by multiplying 1.5, i.e., $\rho=1.5 \tau$, to compensate its relatively smaller scale in practice.
• Our analysis in the paper (Table 2) indicates that the EMI consistently shows moderate or strong correlation with the LLM-judge evaluation metric, win rate, across different types of shifts and model architectures.

We will add this description in our future version of the manuscript. Thank you for the suggestion.

A3. Intricate visualization of Figure 3 and the tightness of EMID upper bound.

• On the left two panels in Figure 3, we presented the overall relationship between EMID and its upper bound, whereas we distinguished four different models on the right two panels to show the model-dependent difference over the relationship. We did so to show that Theorem 4.5 actually differentiates the model itself through the output entropy of each model $H(P_{\theta}(\cdot|x))$, i.e., LLaVA NeXT shows higher sensitivity to the shifts compared with LLaVA v1.5 implied by the larger slope.
• In this work, we do not claim the tightness of the derived upper bound. Moreover, verifying the tightness of the proposed bound can be affected by the choice of estimators for the MI and JSD terms during empirical validation. However, we would like to emphasize that the bound shows consistent correlation with statistical significance across 61 cases of distribution shift over four different models, which means that our analytic bound of EMID effectively stands for performance degradation of MLLM. We appreciate your valuable concern, and we will explore devising a much tighter bound in future work.

Reference

1. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al. 2023
2. PROMETHEUS: INDUCING FINE-GRAINED EVALUATION CAPABILITY IN LANGUAGE MODELS, Kim et al. 2024

A1. Effect of modality fusion/interaction on generalization.

• MLLMs commonly undergo a modality alignment phase during training, which may affect generalization, and it is known that modality fusion can reduce the sample complexity to improve generalization [1]!
• As noted in line 250-254 in our paper, $I(P_{\mathbf{X}Y})$ can be factorized into $0.5 I(P_{X_{v}Y}) + 0.5 I(P_{X_{t}Y}) + 0.5 I(P_{X_{t}Y|X_v}) + 0.5 I(P_{X_{v}Y|X_t})$ where the conditional MI terms $I(P_{X_{t}Y|X_v})$ and $I(P_{X_{v}Y|X_t})$ encapsulate modality interaction.
• Based on this factorization, we can define per-modality EMI based on $I(P_{X_{v}Y})$ and $I(P_{X_{t}Y})$, and then derive a new upper bound that is constructed with $I(P_{X_{t}Y|X_v})$ and $I(P_{X_{v}Y|X_t})$ terms to capture the effect of modality interaction. We leave the explicit derivation for future work.

A2. Interdependencies between $\epsilon$-representation capacity and the model size, number of training samples, and input dimension.

The $\epsilon$-representation capacity assumption captures the minimum achievable discrepancy between truth distribution $P_{Y|X}$ and the model's distribution $P_{\theta}(\cdot|X)$. Due to the expectation and $\min$ operator, it does not depend on the training sample size but is mainly influenced by model capability.

Specifically, as the models become more expressive—e.g., by increasing model size [5] and leveraging advanced positional encoding [6]—MLLM approaches the universal approximator of sequence-to-sequence mapping [6,7], as a result, the minimum expected discrepancy tends to decrease, leading to a smaller $\epsilon$.

We will elucidate this in the next version, thanks!

A3. How to leverage EMID upper bound to guide model optimization?

While our primary focus is on presenting the theoretical framework to quantify the performance gap of MLLMs, we also showcase an application of the EMID upper bound in this rebuttal, a regularization term for visual instruction tuning.

Due to the space limit, we have included the setups and results in the rebuttal to reviewer o1Fb, response A3. Please refer to that thread! As shown in the tables, our instantiation of Theorem 4.5 can indeed be used to optimize model to improve robustness under shifts.

A4. Eq. (4) can be an inaccurate estimate of MI lower bound due to potentially large $H(P_Y)$.

• In the Eq. (4): $I(P_{XY})\geq\mathbb{E}[\log P_{\theta}(y|x)]+H(P_Y)$, maximizing $\mathbb{E}[\log P_{\theta}(y|x)]$ through instruction tuning can be interpreted to learn a parameter $\theta$ that maximizes $I(P_{XY})-H(P_Y)$ rather than solely $I(P_{XY})$.
• We do not claim that the log-likelihood term is a tight lower bound of MI but rather suggest that maximizing $\mathbb{E}[\log P_{\theta}(y|x)]$ can implicitly maximize the MI between input and model's response. We will revise the paper to make this clear.
• To validate this, we reproduce the visual instruction tuning of LLaVA-v1.5-7B on a 10% subset of data, and show how the empirical estimate [2] of MI evolves during training.

  Step	$\hat{I}$
  1	0.166
  5	0.172
  20	0.182
  100	0.187
  200	0.194
  500	0.197

• As shown above, visual instruction tuning can effectively maximize MI between input and model response.

A5. Encoder sensitivity analysis to domain shifts for EMI estimation.

• In Table 4 and 5 of Appendix, we already discussed two alternative choices of the encoders, e.g., [3], and showed that our theoretical claims hold in practice consistently across varying encoders with statistical significance.
• We further conduct encoder sensitivity analysis under domain shifts by replicating experiments on medical domains with a CLIP-ViT-B32 and XLM-RoBERTa encoders.
• Specifically, we use 200 samples of LLaVA-Med [4], get three splits of them based on embedding distance with COCO images, and translate English queries into six different languages used in the paper by using GPT-4o to induce 28 subsets of shifts to conduct correlation analysis for EMID and its upper bound.

  Model	Pearson $r$	$p$-val
  LLaVA-v1.5.7B	0.93	0.00

• We see the correlation between EMID and its upper bound estimates is very strong, even though the medical image and text are relatively minority instances compared with general object and text, implying that our theorem robustly holds even on the special domains that encoders may not excel at.

Reference

1. A Theory of Multimodal Learning, Lu 2023
2. A Contrastive Log-ratio Upper Bound of Mutual Information, Cheng et al. 2020
3. Universal Embeddings with Multimodal Large Language Models, Jiang et al. 2024
4. LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day, Li et al. 2023
5. Scaling Laws for Neural Language Models, Kaplan et al. 2020
6. Your Transformer May Not be as Powerful as You Expect, Luo et al. 2022
7. Transformers are Universal In-context Learners, Furuya et al. 2024

We appreciate sTSP's effort to read our paper and provide comments. Here is a notation table and our responses.

What problem is being solved? What is the risk?

• As we noted in the introduction, MLLMs suffer from performance degradation when they encounter distribution shifts. We denote risk as this performance degradation.
• Problem focus: As noted in the abstract, introduction, and motivation sections, our goal is to quantify the performance degradation of MLLMs under distribution shifts by presenting an information-theoretic framework.

It seems this is about (bi-modal) vision and language models with vision only in the input.

• As we noted in L107-109, MLLMs take multimodal input (both visual and text) to produce text output, not "vision only in the input".
• The term MLLM is commonly used in the literature to denote LLMs that receive a visual input as well as text [1,2], so we took this term by following the convention.

Definition of $P_X,P_Y,P_{XY},P_{\theta}$.

• In L92-103, we put the definition of random variables and distributions. $P_X,P_Y,P_{XY}$ denote the probability distributions of the input $\mathbf{X}$, target response $Y$, and their joint $\mathbf{X}Y$, respectively.
• The $P_\theta$ is a model being trained.

Definitions of $D(P_{X_v}||Q_{X_v})$ and $D(P_{X_t}||Q_{X_t})$. How are these defined in terms of next token probabilities? What is $P_{X_v}$ and $P_{X_t}$?

As noted in L96-97, $X_v$ and $X_t$ are the sequences of visual and text input tokens, so the $P_{X_v}$ and $P_{X_t}$ are the corresponding distributions. $D(P_{X_v}||Q_{X_v})$ and $D(P_{X_t}||Q_{X_t})$ are defined at the input, which should not be confused with the next token probabilities (at the output level).

Put the definitions of variables before shift examples. Domains of the variables in L93? How is (X_v,X_t) combined into a sequence?

• We put the definition of all variables in L92-103 before the motivation section.
• We concatenate the visual input tokens $X_v$ and textual input tokens $X_t$ into a single sequence, where the visual tokens are obtained by encoding an image using a vision encoder (CLIP-ViT), and then projecting them into the language embedding space. This follows the standard practice in MLLMs, where visual tokens are prepended to the text tokens to form a unified input sequence.

Definition of MI and EMI

• Our main interest is to express the model performance gap across different distributions. Thus, instead of defining MI with individual variables, we define MI as their joint distribution, which is mathematically equivalent to random variable-based MI, as can be seen in Eq 3.
• The definition of EMI was explicitly introduced in Eq 6. Please refer to L168-169.

L101: instruction tuning has not been introduced. L99: joint probability? Eq 1, should this not be argmin? Eq 2. Should this be P_{\theta} instead of \theta

• Instruction tuning was introduced from L104.
• In statistics, the population (distribution) [3] is used to denote a distribution of the entire collection of objects in contrast to a sampled distribution.
• In Eq 1, both min and argmin can be valid, where the former aims at the objective-centric whereas the latter stands for the parameter-centric perspective.
• In Eq 2, we use the first argument to denote the data distribution that the metric is computed on, and use the second argument to denote the model parameter to be evaluated. We will use $P_{\theta}$ in the revised version.

Reference

1. A Fully Open, Vision-Centric Exploration of Multimodal LLMs, Tong et al. 2024
2. Exploring The Design Space for Multimodal LLMs with Mixture of Encoders, Shi et al. 2025
3. Sampling of Populations: Methods and Applications, Levy and Lemeshow 2013

A1. Applicability on SOTA MLLMs.

Thanks for the suggestion! Following your comment, we additionally conduct the full evaluation withQwen2.5-VL-7B-Instruct and InternVL2.5-7B.

Specifically, we first evaluate the official release of Qwen2.5-VL-7B-Instruct and InternVL2.5-7B models on 35 synthetic shifts, then compute EMI estimates over the model responses. We perform a correlation analysis between estimates of EMI difference (EMID) and its upper bound. Consistent with our existing finding, we observe a strong correlation between EMID and its theoretical upper bound in both models.

A2. How do assumptions impact the applicability of proposals to more complex cases? This is an insightful question. First, to claim the closeness between EMI and win rate (Thm 4.4), we assumed $\epsilon$-representation capacity of the MLLM.

• $\epsilon$-representation capacity essentially reflects the model’s ability to approximate the target task’s conditional distribution, meaning that the model can approximate this distribution with a KLD no greater than $\epsilon$.
• The assumption mainly argues the strong expressive and approximation capabilities of MLLMs. Given the strong expressive and approximation capabilities of recent large-scale MLLMs, this assumption is generally reasonable in practice.
• Moreover, numerous efforts improve the diversity of an instruction tuning dataset and robustness of the visual encoder of MLLM [1,2], which makes the learned distribution robustly approximate conditional distributions encountered during evaluation.
• As we continually pursue enriching dataset construction and enhancing visual recognition of the encoder, $\epsilon$-representation capacity assumption becomes reasonable in more complex cases.

Second, to claim the relationship between EMID and its upper bound in a simple case (Thm 4.5), we assumed consistent conditional distributions over $X_v|X_t$, $X_t|X_v$, and $Y|X$.

• This assumption zeros out the discrepancy between conditional distributions. If the conditional distributions are quite different between ID and OOD in some real-world scenarios, this makes our upper bound underestimate the performance gap, e.g., EMID.
• However, we highlight that the strong correlations between EMID and this upper bound have been observed through 61 distribution shifts, implying the validity of our upper bound to quantify EMID.
• Meanwhile, we also provide a bound for general cases to address non-consistent conditional distributions in Thm 4.6. This general-case bound can also be empirically estimated using a procedure similar to that of Thm 4.5.
• Therefore, as mentioned in our manuscript (L302-303), we recommend choosing a proper bound based on the knowledge of the data-generating process for datasets.

A3. Details for practical implications of EMID upper bound.

While we confined the scope of this project to presenting the first theoretical framework to quantify MLLM's performance gap, we further provide a potential application of EMID upper bound, instruction tuning with regularization, for this rebuttal.

Without loss of generality, we can assume the input sequence $X=(X_v,X_t)$ as a sequence of intermediate representation vectors of MLLM, i.e., $Z=(Z_v,Z_t)$, and can also assume that $P_{\theta}(.|.)$ maps this representation to responses, i.e., $P_{\theta}:Z \rightarrow Y$. This induces a modified bound with representation variable $Z$ rather than raw data input $X$.

We instantiate this modified EMID bound in two distinct setups below, where we set the 24th layer's hidden states as $Z$, and adopt RJSD [3] as a differentiable estimator for JSD and an average of empirical model output entropy. We provide evaluation results with LLaVA-v1.5-7B on in-distribution (ID) and visual (V), text (T), and joint (J) synthetic shifts.

Regularization term for instruction tuning: $\mathbb{E}[H(P_{\theta}(\cdot|z))] \cdot ((D_{JS}^{0.5}(P_{Z_v}||N(0,I))+(D_{JS}^{0.5}(P_{Z_t}||N(0,I)))$

• One can not access $Q_X$ during the training phase, so we alternatively enforce the distribution of the intermediate representation to be close to the standard Gaussian.
• We sampled 10% of the instruction dataset from LLaVA-v1.5, and trained the entire LLM and modality connector parameters of LLaVA-v1.5 with and without the regularization.
• As shown in the table, we confirm that the EMID can be leveraged as a regularizer during instruction tuning to pursue better robustness to distribution shifts. |Method|ID|V Shift|T Shift|J Shift| |-|-|-|-|-| |Baseline|72.7|65.8|68.0|59.6| |Baseline + Ours|72.7|66.3|68.3|60.8|

Reference

1. Qwen2.5-VL Technical Report, Alibaba Group 2025
2. Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders, Shi et al. 2025
3. The Representation Jensen-Shannon Divergence, Hoyos-Osorio et al. 2024