Efficient Quantification of Multimodal Interaction at Sample Level

We sincerely thank the reviewer for the positive feedback and constructive comments on our work. We have carefully addressed the citation issue mentioned in the review, and examine other reference and formualtions carefully. Thank you again for your time and effort.

Thank you for their valuable suggestions. Here are our responses to the questions.

Q1: Visualization of $r^+$ and $r^-$

A1: Thank you for this valuable suggestion. To demonstrate the function of the $r^+$ and $r^-$ components derived from the total redundancy $r$, we conducted experiments on a Gaussian Mixture Model (GMM) dataset. We visualize the distributions of $r$, $r^+$, and $r^-$ in the provided figure a.

As indicated by the distributions, $r^+$ appears to capture the core information consistently present across modalities within a sample, reflecting the minimum entropy along the unimodal marginal distributions (i.e., the most reliably available information from at least one modality). It represents the inherent information content shared by the modalities. In contrast, $r^-$ quantifies the portion of redundancy where individual modalities might struggle or provide conflicting cues regarding the task. It reflects the ambiguity or potential for error introduced if relying on the worse modality.

Q2: Scalability to three modality

A2: Thank you for this thoughtful suggestion. Extending interaction analysis on more than two modalities inherently involves quantifying complex interactions, such as the mutual information shared among three modalities and the target task. However, a key challenge arises because the clear theoretical definitions used to decompose two-way interactions into distinct synergistic, redundant, and unique contributions do not readily or uniquely translate to these higher-order cases [1,2].

Given this theoretical limitation, we adopt the widely-used pairwise analysis approach (Appendix C.4 of [3]). This method systematically examines interactions between each pair of modalities. We implemented this approach on the UCF-101 dataset, analyzing the interactions among its three modalities: Vision, frame difference (Diff), and optical flow (OF). The quantitative results of these pairwise interactions are presented in the following Table.

The pairwise analysis revealed that vision is the strongest modality, as its unique interactions consistently surpass those of other modalities. Furthermore, we observed a primarily redundant relationship between vision and frame difference, while frame difference and optical flow demonstrated notable synergy. This observed synergy likely arises because frame difference and optical flow, when considered individually, carry less task-relevant information and thus benefit significantly from combining their complementary information to achieve better task performance.

Q3. physical meanings of $i(x_1;x_2; y)<0$.

A3: Thank you for raising this important question. To build intuition, consider the simpler case of negative unimodal information: $i(x_1;y) = \log\frac{p(y|x_1)}{p(y)}$. Negativity implies $p(y|x_1) < p(y)$, meaning that observing modality $x_1$ decreases the likelihood of the target $y$ compared to its prior probability. This suggests $x_1$ provides information that contradicts $y$, a situation potentially arising from mislabeled samples.

For the interaction term, defined as: $i(x_1;x_2;y) = i(x_1;y) + i(x_2;y) - i(x_1,x_2;y)$ a negative value ($i(x_1;x_2;y) < 0$) signifies that the joint information $i(x_1,x_2;y)$ is greater than the sum of the individual information components ($i(x_1;y) + i(x_2;y)$). This phenomenon represents synergy: the modalities $x_1$ and $x_2$, when combined, provide more information about $y$ (or potentially, more "mis-information" if the individual terms are strongly negative) than would be expected from merely summing their individual contributions.

Regarding the redundancy term $r(x_1;x_2;y)$ discussed in our paper (which forms part of the unimodal information, e.g., $i(x_1;y) = r + u_1$), this can also be negative. This is particularly likely if both individual modalities carry contradictory information about the label (i.e., $i(x_1;y) < 0$ and $i(x_2;y) < 0$). In such cases, the redundant information component may also reflect this contradictory concerning the target $y$.

[1] Tobias Mages et al. "Decomposing and Tracing Mutual Information by Quantifying Reachable Decision Regions." Entropy, 2024.

[2] Paul Williams et al. "Nonnegative Decomposition of Multivariate Information." arXiv, 2010.

[3] Paul Pu Liang et al. "Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework." NeurIPS, 2023.

Thank you for your valuable comments and suggestions. Here are our responses:

Q1: Model biases in dataset interactions

A1: We appreciate the reviewer's valuable point. Indeed, quantifying multimodal interactions requires models, making the estimates inherently model-dependent. Consistent with prior work yielding reliable outcomes [1], our method employs fully trained multimodal models. We focused on models demonstrating strong generalization, as we posit these better reflect the underlying data distribution and better modelling the interactions within it.

Following the reviewer's suggestion, we constructed an ensemble integrating several of best-generalizing models to measure interactions. As detailed in Table a, the interaction estimates derived from this ensemble were highly consistent with those from individual top-performing models, exhibiting negligible differences.

Q2: Comparison against attention-based methods

A2: Thank you for this valuable suggestion. We validated LMSI on a Multimodal Transformer with hierarchical attention (from multi-stream to single-stream)[2], where we varied the layer $l$ at which cross-modal fusion was first introduced. A smaller $l$ indicates earlier fusion architecture. The experiment results are shown in Table b. Our analysis revealed that interactions learned from later fusion stages were predominantly characterized by redundancy, whereas interactions learned from earlier fusion stages demonstrated a stronger tendency to capture synergistic relationships between modalities.

Q3: Understand the absolute scores of LMSI

A3: Thank you for this insightful question. LMSI components ($r$, $s$, $u_1$, $u_2$) measure information quantity (in nats/bits) from different multimodal interactions. We can utilize relative proportions (e.g., $\hat{r} = \frac{r}{i(x_1,x_2;y)}$) as a bounded, standardized metric. These relative scores ($\hat{r}$, $\hat{s}$, $\hat{u}_1$, $\hat{u}_2$) sum to 1 and provide a normalized interaction view. The optimal values of interaction varies by task - video tasks show high $\hat{r}$, while sentiment analysis favors $\hat{u}$, as shown in Table 2 in the main paper.

Q4: Comparison with human judgment

A4: Thank you for this point. Direct numerical comparison is challenging because human evaluations are typically sample-specific and inherently subjective, making it difficult to accurately represent the information quantity across the entire data distribution. Despite these differences, human judgments can provide valuable insights into the perceived relative importance or ranking of interactions within specific instances. Following precedents like [1], we investigated the alignment between LMSI and human perception by focusing on the dominant interaction types identified by both. As shown in Table 2, LMSI demonstrates reasonable consistency with human assessments in identifying top interactions across datasets.

Q5: LSMI Behavior with Noise

A5: Thank you for your question. LSMI quantifies task information $Y$ across modalities $(x_1, x_2)$, making it sensitive to information quality. Label noise reduces the multimodal mutual information $i(x_1, x_2; y)$, affecting LSMI's interaction measures. We added noise to image-text pairs Table c, observing decreased interactions, especially for information-rich samples. This confirms LSMI's sensitivity to noise.

Q6: Request for concrete tasks

A6: Thank you for this suggestion. We wish to clarify that our method, LMSI, has already been validated on several concrete tasks: sentiment recognition (MOSEI), action recognition (KS), and sarcasm detection (URfunny).

To further demonstrate LSMI's practical utility, we applied it to guide ImageBind fine-tuning on the KS dataset (Audio+Visual modality). Using LSMI, we quantified sample-wise audio-visual redundancy and partitioned the KS dataset into two subsets: high-redundancy ($R$) and low-redundancy ($U+S$). We then fine-tuned ImageBind separately on these subsets as well as the full dataset ('alldata'), comparing their classification performance. The results, presented in Table d, demonstrate that our data partitioning strategy based on LSMI analysis can enhance fine-tuning outcomes. While our VQA task experiments are still in progress due to time limitations, these results highlight the potential of LSMI for improving model fine-tuning.

[1] Paul Pu Liang et al. "Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework." NeurIPS, 2023.

[2] Peng Xu et al. "Multimodal Learning with Transformers: A Survey." TPAMI, 2023.

We thank the reviewers for their valuable suggestions. Here are our responses to the questions raised about our paper.

Q1: The contribution of this work compared to the work from Morency's Lab.

A1: Thank you for this question. The core contribution of our work, compared to foundational studies from Morency's Lab [1], is the development of efficient sample-wise quantification of multimodal interactions.

We build upon the same established interaction concepts (redundancy, uniqueness, synergy) common to both [1] and other literature [2,3]. However, the key distinction lies in mathematical modeling of interactions: Morency's work (PID) [1] typically provides dataset-level aggregate measures of interactions. Our method enables quantifying these interactions for each individual sample.

This efficient sample-level resolution is our main technical contribution and is vital for applications requiring instance-specific insights, such as targeted model integration or knowledge distillation strategies explored in our experiments (Section 4.3). Aggregate measures (PID) cannot readily support these tasks.

Notably, the value of such pointwise measures was recognized as an important next step by Paul and Morency 2023: "A natural extension is to design pointwise measures... which could help for more fine-grained dataset and model quantification..." Our work provides a concrete method to realize this, demonstrating its contribution and relevance.

Q2: Application of LSMI to Downstream Tasks

Thank you for the suggestion. Our LSMI method enables the estimation of sample-level interactions, which can indeed benefit downstream tasks beyond the model ensemble and distillation examples discussed in the main paper.

To demonstrate this, we applied LSMI to guide the fine-tuning process for a specific task. We used the ImageBind model, which focuses on modality alignment, and fine-tuned it on the Kinetics-Sounds (KS) dataset (Audio+Visual). Specifically, we employed LSMI to quantify the degree of redundancy between the audio and visual modalities for each sample. Based on this metric, we partitioned the KS dataset into two subsets: one containing samples with relatively high redundancy ($R$) and another with relatively low redundancy ($U+S$, comprising samples dominated by unique or synergistic information).

We then fine-tuned the ImageBind model separately on these two subsets ($R$ and $U+S$) and compared the results against fine-tuning on the entire dataset ('alldata'). The performance, evaluated by classification accuracy on KS dataset, is shown below:

As observed, fine-tuning specifically on the high-redundancy subset ($R$) identified by LSMI yielded the best performance for the combined modalities (V+A) and the visual modality (V). This suggests that leveraging LSMI to identify and focus on highly redundant samples can be an effective strategy for enhancing model fine-tuning, potentially by reinforcing learning when sufficient information is present in individual modalities.

Q3: Result state in Table 1

A3: We appreciate you pointing it out. Table 1 focuses on evaluating the precision of different interaction estimation methods. We utilize synthetic datasets based on logic relations with additive noise, which allows for comparison against known ground truth (GT) interaction values. This discrete logic carries explicit interaction information and has been widely used in previous studies [1,4]. The objective is to assess how closely each estimator's output matches the GT. As the results indicate, our method achieves interaction estimates that align more closely with the GT compared to the baseline estimators evaluated. This finding underscores the robustness of our approach, demonstrating its effectiveness in capturing underlying data interactions despite the presence of noise. We have enhanced the description surrounding Table 1 in the revised manuscript to better emphasize this point.

[1] Paul Pu Liang et al. "Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework." NeurIPS, 2023.

[2] Benoit Dufumier et al. "What to align in multimodal contrastive learning?" ICLR, 2025.

[3] Paul Pu Liang et al. "Multimodal learning without labeled multimodal data: Guarantees and applications." ICLR, 2024.

[4] Nils Bertschinger et al. "Quantifying Unique Information." Entropy, 2014.