RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer

Dear reviewer gn7r,

Thanks a lot for your valuable review, suggestions, and questions.

Q1: Extension to more fusion paradigms

Q1.a: Analysis of cooperation dynamics.

To analyze cooperation dynamics across fusion methods, we monitor the gradient of unimodal encoders and the attention which are the two driving factors of the self-reinforcing cycle. As shown in Figure 2, the gradient difference declines and the attention becomes balanced and more unstable as the fusion becomes earlier.

Q1.b: Systematic comparison and QRR behaviors.

Following descriptions in Appendix B, we train the model progressively and apply QRR. The comparison results are shown in Table 1, where Vanilla MT denotes the vanilla multimodal transformer. The results reveals that our method can be applied to earlier fusion paradigms achieving better performance around (2.3% / 1.1% / 0.2%), but it's more suitable for late fusion paradigm.

Q2: Verification and validation.

Q2.a: Results on OOD benchmarks.

Thanks for providing a novel aspect for testing QRR. We adopt MultiOOD [1]. As shown in Table 2, our QRR algorithm outperforms the baseline on all metrics, showing that our QRR algorithm can ease the modality bias during training dynamics.

[1] Hao, D., et al. "MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities", NeurIPS 2024.

Q2.b: Comparison with QUAG[2]

QUAG is designed based on VideoQA tasks using frozen unimodal encoders, which sets it apart from our work. Besides, QUAG performs analysis on averaging attention score but our analysis focus on feature space of key, query in attention.

[2] Rawal, Ishaan Singh, et al., “Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion”. ICML 2024

Q2.b: QRR might not leverage both modalities due to modality bias.

In this work, we focus on modality bias that one modality dominates the training process[3], leading to inequality modality feature, unreasonable attention and sub-optimal performance. Hence, we propose QRR to balance the learning of unimodal encoders, revive the cooperation dynamics to leverage both modality.

We ablate QRR through mask or average attention score inspired by QUAG[2]. As shown in Table 3,4, when masking one modality, QRR outperforms the baseline from 0.2 ~ 3%. This confirms QRR's ability to enhance unimodal feature quality. When using averaged attention scores, in CREMA-D (audio-dominant), vanilla MT exhibits a performance degradation of 2.3% due to over-reliance on audio features. For Kinetic-Sound (relative balance), vanilla MT's performance increases by 1.2% since the model learns an unreasonable attention score due to the self-reinforcing cycle. Conversely, QRR maintains stable performance (with <1% drop), demonstrating it's leveraging complementary modality information and robustness.

[3] Peng, X., et al. "Balanced multimodal learning via on-the-fly gradient modulation." CVPR 2022.

Q3: Supporting details on the stability, efficiency, and performance

Q3.a: Batch size ablation.

Through batch size ablation on CREMA-D in Table 5, QRR maintains stable performance improvements (0.7%-3.1%) compared to baseline from 16 to 256 batch size, demonstrating remarkable training stability.

Q3.b: Limited performance improvements.

Compared to methods requiring additional parameters and specialized modules, QRR only requires：

• Parameter increase: 1.0%
• FLOPs increase: 0.1%

as shown in Table 7, while accuracy achieve 3.1% improvements on CREMA-D and 2.3% improvements on Kinetic-Sound compared to vanilla multimodal transformer. , QRR is a simple yet effective method that achieves comparable and even better results with less computation cost.

Q4: Details on paper writing and figures

Q4.a: Notation in figure 2.

In a single fusion layer setting, $q_b$ and $q_r$ are the same. As the number of fusion layers increases, the query $q$ becomes influenced by the input and is no longer a static value. Hence, for each input $q$, the rotation matrix $q_b$ can not ensure to rotate the $q$ to $q_b$, but should be located near this region.

Q4.b: Detailed captions, missing of related works[5,6] and header of the paper.

Thanks for pointing this out, we'll add relative discussion in our revised manuscript.

[5] Buch, Shyamal, et al. "Revisiting the "video" in video-language understanding." CVPR 2022.

[6] Pătrăucean, Viorica et al., “Perception Test: A Diagnostic Benchmark for Multimodal Video Models”. NeurIPS 2023

Dear reviewer FKSM,

We appreciate your time and great efforts in reviewing.

We carefully considered your comments on the validation of assumptions, lack of in-depth analysis and extension to benchmarks and methods and conducted corresponding experiments and theoretic analysis.

Q1: The validation of the deactivation of the cooperation dynamics assumption is not adequate since the datasets and settings are limited.

Thanks for pointing out. In the previous version, we provide visualization and analysis of Kinetic-Sound for this assumption. To holistically and systematically validate our assumption, which mainly states that the modality bias in the multimodal training process triggers a self-reinforcing cycle that leads to inequality in feature quality and unreasonable attention score, we conduct experiments on more datasets including CREMA-D, Kinetic-Sound, CMU-MOSEI(A+T), CMU-MOSEI(V+T), UCF-101, HMDB51, whose modality ranging from audio, RGB, text and optical flows and exhibiting variation in modality sequence length. As shown in Figure 1, with different modalities combinations and dataset scales, the unreasonable attention score attribution agree with our previous study. Besides, the visualization of average key distribution, where the noise input consistently has higher cosine similarity, further validates our assumption and theoretic analysis towards cooperation dynamics under imbalance multimodal learning.

Q2: Lacks in-depth analysis

Q2.a: in-depth analysis of self-reinforcing cycle.

In addition to the observation of attention score during training dynamics, we monitor the gradient of unimodal encoders to verify our proposition of a self-reinforcing cycle. As shown in Figure 3, the gradient of both modalities's encoder are similar during the start time. However, the biased modality (audio) encoder increases significantly during the mid-stage, when the attention score begins to accumulate in the biased modality. Later, both of the gradients drop due to the minimization of total loss. This observation further verifies our theoretical analysis and ensures the existence of the self-reinforcing cycle.

Moreover, from the perspective of fusion settings, we visualize the gradient and distribution of attention scores on different modalities under comprehensive fusion paradigms. As shown in Figure 2, even as the starts of the fusion layers varied, the gradients of the biased unimodal encoder and attention score towards biased modality consistently exhibits considerable greater than the unbiased one, verifying our assumption and analysis of the self-reinforcing cycle.

Q2.b: in-depth analysis of significance and stability.

To evaluate the stability and result significance of our method. We conduct repetitive experiments over CREMA-D and Kinetic-Sound with only differences in random seeds. By applying Pearson correlation analysis as shown in Table 6, with 0 denotes vanilla multimodal transformer while 1 denotes applying QRR algorithm, the coefficient is 0.765 for CREMA-D and 0.698 for Kinetic-Sound indicates its statistically significant on performance improvements with p-value < 0.01 ensures the confidence of the analysis.

Besides, ablation on batch size ensures the stability of our method. As results shown in Table 5, with batch size varying from 16 to 256 on CREMAD, the QRR consistently outperforms baseline methods by 0.7% ~ 3%, revealing the stability of our method.

Q3: Extension

Q3.a: Extension to more benchmarks

In addition to existing benchmarks and testing, We adopt MultiOOD [1] for the Out-of-Distribution benchmark and further validate the performance of our QRR method. Here, we consider both near-OOD and far-OOD scenarios discussed in MultiOOD. As results shown in Table 2, our QRR algorithm consistently outperforms the baseline on all metrics and fully tests the effectiveness of our method from another perspective.

[1] Hao, D., et al. "MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities", NeurIPS 2024.

Q3.b: Combine QRR with existing methods

Due to the time constraints, we haven't done enough attempts to combine our method with existing multimodal methods, which would be conducted during discussion period.

If you have any valuable questions or constructive suggestions that could help you understand our work, please tell us and help to produce a higher quality paper.

Dear reviewer 61wP,

Thank you very much for your affirmation and constructive comments.

We carefully considered your comments and conducted corresponding experiments.

Q1: The change of gradients during training.

Thank you for your question. From the optimization perspective, previous works on imbalanced multimodal learning have extensively discussed the evolution of gradients under a late fusion structure with concatenation or summation fusion, which shows similarities to our setup[1, 2]. A well-established observation is that the superior modality always gains more optimization momentum, as its gradient is higher than that of the weaker modality.

However, our setup differs from previous simpler structures where we adopts more complex transformer blocks for multimodal fusion, and your question prompted us to create a clearer visualization to provide more convincing evidence for our theoretical analysis.

As shown in Figure 3, the gradient of the audio encoder increases significantly during the mid-stage, when the attention score begins to accumulate in the biased modality. This results in a noticeable gap between the modalities. As the total loss decreases over time, the gradients of both modalities drop, but their relative relationship remains. This further validates our theoretical analysis, showing that the biased modality consistently receives more optimization momentum. The gradient of the weaker modality never exceeds that of the superior modality, which can be explained by Equations 7 and 8 in our paper. Since the only difference in gradients for each modality is $\frac{\partial h_i}{\partial z_i^m}$ and the loss is the multimodal loss, where "one modality becoming converged" equals "the multimodal model becoming converged," the total loss becomes very small. This small loss cannot provide enough momentum for the superior modality to optimize effectively. Hence, the gradient of the superior modality will be consistently greater than the weak one.

[1] Peng, X., et al. "Balanced multimodal learning via on-the-fly gradient modulation." CVPR 2022.

[2] Fan, Y., et al. "Pmr: Prototypical modal rebalance for multimodal learning." CVPR 2023

Q2: Cold-Start and pretraining's influence on self-reinforcing cycle.

Thank you for your question. As discussed in the Experiments section, we are not using cold-start but using a 4-layer ViT-B/16 as the backbone, initialized with pre-trained weights from ImageNet-21k. For the MOSEI dataset, we use the Vanilla Transformer without pretraining. The results shown in Table 1 and the visualizations in Figure 4 demonstrate that even under pretraining conditions, the self-reinforcement cycle still exists and harms the overall performance of multimodal transformers.

Q3: Limited Improvements

Q3.a: Significance tests to prove the gain is statistically meaningful.

Thank you for your suggestion. To prove the significance and stability of QRR, we selected 10 random seeds and conducted repeated experiments with the same settings on the CREMA-D and Kinetic-Sound datasets. By applying Pearson correlation analysis as shown in Table 6, with 0 denotes vanilla multimodal transformer while 1 denotes applying QRR algorithm, the coefficient is 0.765 for CREMA-D and 0.698 for Kinetic-Sound indicates its statistically significant on performance improvements with p-value < 0.01 ensures the confidence of the analysis. We'll further accomplish a more systematic analysis of pretraining's influence during the discussion period.

Q3.b: Limited improvements.

Thanks for mentioning this. However, the QRR, which requires only:

• 1% increase on parameters
• 0.1% on GFLOPs

As shown in Table 7. It achieves considerable improvements on its baseline vanilla multimodal transformer (accuracy increase by 3.1% on CREMA-D and by 2.3% on Kinetic-Sound), while gaining comparable and even better performance over other specially designed transformer architecture with much more requirements on computation resources and increased complexity. Hence, the improvements are significant especially taking the computation and time cost into consideration.

As for CMU-MOSEI, we acknowledge this is an difficult dataset for multimodal sentiment analysis. With all of our comparison methods achieving limited improvements and previous research [3] with three modalities input could only achieve around 1.5% improvements, our QRR algorithm performs relatively significant increment.

[3] Paul Pu Liang et al. "Multibench: Multiscale Benchmarks for Multimodal Representation Learning." NeurIPS, 2021

If you have any valuable questions or constructive suggestions that could help you understand our work, please tell us and help to produce a higher quality paper.