Deep Equilibrium Multimodal Fusion

Thanks for your time reviewing our paper. Please find our responses below.

Computation cost.

We have tracked the runtime of DEQ fusion on VQA-v2, the results are 0.338 and 0.405 seconds per batch without and with DEQ fusion respectively. Since the majority of the runtime is due to the unimodal feature extractors, we believe that we can tolerate this computational overhead to achieve higher performance

Scalability.

Thanks for your advice. Please find the pseudo codes below written in PyTorch. Inserting our DEQ fusion requires only modifying three line of codes within the model. Also note that we will release our source codes.

    class Model(nn.Module):

        def __init__(self, ...):
            ...
+           self.fusion = DEQFusion(...)

        def forward(self, x):
            ...
-           fused_feature = torch.cat([feat for feat in unimodal_feats]) # unimodal_feats contains all features extracted from unimodal encoders
+           fused_feature = self.fusion(unimdaol_feats)
            ...

Motivation.

Thanks for pointing us to MMTM. MMTM leverages the squeeze and excitation (SE) technique to recalibrate the channel-wise features from multiple CNN streams for multimodal feature alignment. This work provides additional evidence for one of the mentioned key elements: stabilizing and aligning multimodal features. We have included MMTM in our revised paper to provide further evidence to strengthen this element.

We are also aware that MMTM is also capable of capturing high-level information. Similar to other works that perform higher-level information integration, this high-level information is captured by stacking multiple fusion units. As a result, MMTM requires manual design of different architectures for different applications. Quoting from MMTM:

"Although the module design is generic and could potentially be added at any level in the network hierarchy, the optimal locations and number of modules are different for each application".

As such, how many fusion units are needed to obtain the optimal fused feature still remains a question for these types of works, whereas our approach is fully self-adaptive and requires minimal manual design.

Thanks for your valuable feedback. Please find our responses below.

Confusion with the phrase "every level".

As mentioned in Section 3.3, our DEQ fusion is indeed a type of late fusion. The term "every level" is thus within the late fusion process itself, where the level indicates the level of information integrated into the fused feature. When we consider the high-level information, it is mainly the product of deeper networks.

Concerns with ablation studies.

Our focus is to develop a robust and generalizable multimodal fusion framework. As shown in Tables 5 and 7, naively applying DEQ to unimodal feature extractors or the fused feature leads to suboptimal results, whereas considering intra- and inter-modality features jointly (our approach) gives the best results. We would like to emphasize that our designs of $f_\theta$ and $f_{fuse}$ are not unique. One may consider employing modality-specific alternatives to further boost performance, however, this is not the focus of our work.

Further explanation on the three keys of successful multimodal fusion.

We have included a further explanation in our revised paper, please find our revision for the related work section (Appendix B). Please note that these findings are suggested and backed up by several previous works.

Explanation in stabilizing representations.

The principle of DEQ fusion is that the equilibrium state of the fused feature satisfies $z_{fuse}^{(i+1)}=f(z_{fuse}^{(i)})$ (slightly simplifies the notation here), which achieves stable multimodal representation while considering multi-level intra- and inter-modality feature fusion. Because it is important to ensure the dynamic fusion process, i.e., any modality needs to account for information from all other modalities (as suggested by [1,2,3]), we thus do not particularly emphasize unimodal feature stability.

DEQ for unimodal learning.

The results for this are in Tables 5 and 7, where we evaluate DEQ applied to the unimodal encoders ($f_\theta$ with DEQ). On all three benchmarks, applying DEQ to unimodal encoders contributes to a 1-2% gain over most evaluation metrics. By jointly considering unimodal and multi-modal features, we are able to achieve optimal results.

Dimensions of the vector.

The features $z$ are of the same dimension as the unimodal features $x$, such that we can compute the summation in equation 8. The dimensions of the features can be freely altered depending on the model output dimensions and different fusion module designs.

Confusion about Multi-bench.

Multi-bench integrates many multimodal fusion methods and benchmarks, which is readily deployable for experiments. Our experiments including compared methods were reproduced with Multi-bench on MM-IMDB.

Vectorized $\alpha$.

This is updated in the revised paper.

[1] Han, Zongbo, et al. "Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification." CVPR 2022.

[2] Wang, Yikai, et al. "Multimodal token fusion for vision transformers." CVPR 2022.

[3] Xue, Zihui, and Radu Marculescu. "Dynamic multimodal fusion." CVPR 2023.

Thanks for following up on the discussion. We have revised our paper accordingly. Please also see our point-to-point responses below.

1. This is updated in the revised paper.

2. The relevant results are provided in the Tables 5 and 7. We will also highlight our results here and provide further explanations for your reference. On BRCA, we found that directly applying DEQ on unimodal encoders and then performing fusion with simple concatenation or summation improves the baseline accuracy to 88.8%. This result is shown in the third row of Table 5 with $f_\theta$ and DEQ. Similarly, we also experimented with applying DEQ on the fused feature (which resulted from concatenating or adding all unimodal features), which also improved the baseline accuracy to 88.3%. This result is shown in the fourth row of Table 5 with $f_{fuse}$ and DEQ. Our DEQ fusion jointly considers uni-modal and cross-modal features, leading to the most superior result of 89.1% accuracy. We conducted such ablation experiments on CMU-MOSI and MM-IMDB in Table 7 and found similar results. These experiments hopefully could demonstrate the superiority of our DEQ fusion compared with naively applying DEQ to multimodal fusion.

3. We have revised the introduction according to your advice. Please see the updated paper.

4. Please see 2 for the comparison.

Thanks again for your effort and please let us know if you have further concerns.

Thanks for your efforts spent on reviewing and improving our work. Please find our responses below.

Comparison against weight-tied approach.

There may be some confusion here. As mentioned in Section 2.1 of our paper, the idea of finding the equilibrium is motivated by two interesting observations, which were both claimed in [1]. They found that the hidden layers of some very deep networks tend to converge to some fixed points, and employing the same weight in each layer (so-called weight-tied) also achieves similar results. From this perspective, by applying any weight a large number of (or infinitely many) times, the output would eventually satisfy $z_{i+1}=f_{\theta}(z_i)$. The advantages of applying DEQ are to accelerate the "infinite" forward process and memory-efficient analytical one-step backward pass.

With this in mind, please note that the memory efficiency and the acceleration of the forward process come from the fixed-point solvers and analytical backpropagation, hence are not our contributions. Instead, we focus on performing robust and generalizable multimodal fusion based on DEQ. As we show in Tables 5 and 7, naively applying DEQ to unimodal feature extractors or the fused feature leads to suboptimal results, whereas considering intra- and inter-modality features jointly (our approach) gives the best results.

'Our fusion design is flexible from the standpoint that $f_{\theta_i}$ can be altered arbitrarily to fit multiple modalities' -> 'Our fusion design is flexible from the standpoint that $f_{\theta_i}$ can be altered arbitrarily to fit multiple level features'.

Our original sentence is trying to express that the design of $f_{\theta_i}$ is not unique and our design might not be optimal. One may consider altering the architecture of this module to be modality-specific to possibly further boost the performance, however, this is not our focus. Our goal instead is to propose a general framework that is applicable to most modalities and multimodal tasks. We will double check our writing to avoid such ambiguities.

When the equilibrium state is reached, why an informative unified representation in a stable feature space for multimodal learning be obtained? What is the relationship between these two?

As mentioned both in Section 1 and Appendix B in our paper, various works have suggested the importance of capturing stable and higher-level representations to enforce better cross-modality for better multimodal fusion [2,3]. The principle of DEQ fusion is that the equilibrium state of the fused feature satisfies $z_{fuse}^{(i+1)}=f(z_{fuse}^{(i)})$ (slightly simplifies the notation here). By recursively applying the fusion layer, we gradually obtain higher and higher levels of feature information. The process stabilizes at the equilibrium state, as such, the unified representation at this stage captures stable and multi-level representations.

The advantages of finding the equilibrium state to extract stable and multi-level unified representation are mainly evaluated empirically in our paper, rather than theoretically. Previous methods mostly extract finite levels of features, where the multi-level feature extractions are generally achieved by stacking multiple fusion blocks. Our experiments show that DEQ fusion outperforms these finite-level methods by a considerable margin, thus demonstrating the informativeness of the extracted feature from our DEQ fusion.

Confusion with drawings.

We would reconsider our figures along with our captions to improve the clarity.

[1] Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. "Deep equilibrium models." NeurIPS 2019.

[2] Pan, Yingwei, et al. "X-linear attention networks for image captioning." CVPR 2020.

[3] Duan, Jiali, et al. "Multi-modal alignment using representation codebook." CVPR 2022.

Thanks for your feedback and encouraging comments, please find our responses below.

Convergence and computational resources.

About convergence, we have examined the convergence of DEQ fusion on three benchmarks, namely BRCA, MM-IMDB, and CMU-MOSI, involving various modalities in Table 8 of our paper. We found that our approach converges to a relatively small value on all three benchmarks, especially for common modalities like image, text, and audio, whereas the convergence on rarer medical-omics modalities is somewhat slower.

Regarding the computational resources, we have tracked the runtime of DEQ fusion on VQA-v2, the results are 0.338 and 0.405 seconds per batch without and with DEQ fusion respectively. Since the majority of runtime comes from unimodal feature extractors, we believe that we can tolerate such inference speed to obtain higher performance.

More evaluations & related works.

Thanks for your suggestions. We have referred to the mentioned works in our revised paper, please find our revision in the related work (Appendix B).

We would again like to highlight that our DEQ fusion was evaluated on four widely used multimodal benchmarks, involving several different modalities, thus demonstrating strong generalizability. We will consider adding more benchmarks to further strengthen this point.