HyperRep: Hypergraph-Based Self-Supervised Multimodal Representation Learning

[Concern 4] On high-order correlation issue.

Thank you for your insightful question regarding the role of high-order correlations in multimodal self-supervised learning. As indicated in our manuscript, high-order correlations are essential for comprehensively understanding multimodal data. They encapsulate intricate relationships within and across modalities that simple pairwise correlations might not fully capture. These complex relationships are crucial for thoroughly understanding the data and are often overlooked by existing methods that predominantly focus on pairwise interactions.

While AGC, a graph clustering method, achieves commendable performance without explicitly using high-order correlations, it's important to note that the graph structure it employs can inherently contain high-order information. In our experiments, we construct a graph adjacency matrix $\boldsymbol{A}=\boldsymbol{HH^\top}$ using the hypergraph incidence matrix $\boldsymbol{H}$ for a fair comparison with AGC. This clique expansion approach effectively retains the high-order information originally present in the hypergraph structure.

Our ablation study results reinforce this point. By setting $k=2$ for hypergraph construction, thereby reducing them to graphs, we noticed a performance decline compared to including high-order correlations ($k=7$). Specifically, HyperRep ($k=2$) underperformed compared to HyperRep ($k=7$), highlighting the importance of high-order correlations in capturing multimodal interactions' complexity. Moreover, HyperRep ($k=2$) still consistently and significantly outperforms AGC across three datasets, demonstrating our method's effectiveness.

Thus, while AGC and other methods might achieve good performance without explicitly modeling high-order correlations, our findings suggest that these correlations are crucial in advancing multimodal representation learning, especially for tasks involving complex data relationships. Your question has inspired us to refine our ablation studies further, and we deeply appreciate it.

AVE dataset:

MSR-VTT dataset:

YouCook2 dataset:

[Concern 5] On the performance of MIL-NCE.

The performance of MIL-NCE on the text-to-video retrieval task on the MSR-VTT dataset is the same as the original paper, as showcased in Fig.5 (b), page 8 in the original paper. However, we did make a typographical error in reporting the R@5 for I3D-based MIL-NCE as 22.0; it should indeed be 22.2 as stated in the original paper. We appreciate your attention to detail and thank you for bringing this to our notice.

[Concern 3] On comparison with recent and popular methods.

We are grateful for your suggestion to broaden our comparison with more recent methods. While OmniVL [2] does not have available implementation code for direct comparison within our current time constraints, we have included additional comparisons with popular and recent methods [3,4,5] from 2022 and 2023 in following charts. The best and second-best performances are highlighted in bold and italic respectively.

From the following results, it is evident that HyperRep excels in the text-to-video retrieval task on the MSR-VTT dataset, achieving the best performance across all metrics. On the action localization task on the CrossTask dataset, HyperRep attains the best performance in Recall. This demonstrates the model's effectiveness against recent and popular methods.

These results affirm the state-of-the-art performance of HyperRep and its significant contributions to the field of multimodal learning, illustrating the model's robustness and the impact of our novel approach even when compared with the latest methodologies.

Text-to-video retrieval task on MSR-VTT dataset:

Action localization task on CrossTask dataset:

[1] VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding. EMNLP 2021.
[2] OmniVL: One Foundation Model for Image-Language and Video-Language Tasks. NeurIPS 2022.
[3] Video-Text Representation Learning via Differentiable Weak Temporal Alignment. CVPR 2022.
[4] Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval. CVPR 2022.
[5] Learning actionness from action/background discrimination. Signal Image Video Process 2023.

We thank the reviewer for the careful reading and valuable feedback on our submission. We are deeply encouraged by the reviewer's appreciation of our work's interesting ideas and the detailed, extensive nature of our experiments. In response, we provide more detailed explanations and descriptions to further clarify the motivation behind our method and to present additional experimental results. We hope that our response comprehensively addresses your concerns. We look forward to any further feedback or suggestions you might have.

[Concern 1] On motivation of high-order correlations.

Thank you for your insightful query regarding the motivation for integrating high-order correlations in multimodal self-supervised learning. As elucidated in our manuscript, high-order correlations are a natural extension and generalization of pairwise correlations, effectively encompassing and expanding upon them.

In the realm of hypergraphs, which we utilize in HyperRep, a conventional graph (encompassing pairwise correlations) is essentially a 2-uniform hypergraph where each edge connects exactly two vertices. By extending this concept to high-order correlations, we are not only capturing pairwise relationships but also encompassing more complex interactions that involve multiple modalities simultaneously. This is crucial for a thorough and nuanced understanding of multimodal data, which often involves intricate relationships that go beyond simple pairwise interactions.

While conventional methods focusing on pairwise supervision have demonstrated effectiveness, they are often insufficient for modeling the complexity inherent in multimodal datasets. These complexities are often inherent in multimodal scenarios but are underrepresented in pairwise models. For example, as we illustrate in Fig.1(a) of our paper, a multifaceted scenario like a video of a car drifting involves complex correlations with related images, engine sounds, and textual descriptions that go beyond simple pairwise associations.

High-order correlations naturally encapsulate the pairwise interactions while providing a broader and more comprehensive framework for understanding the complex dynamics of multimodal interactions. This is why our HyperRep approach leverages hypergraphs to model these correlations, thus capturing a richer and more intricate representation of multimodal data than would be possible with a focus solely on pairwise relationships. It sets HyperRep apart from existing methods and underscores the unique value it brings to the field.

Therefore, our focus on high-order correlations is rooted in their ability to encompass and go beyond pairwise relationships, offering a more complete and effective approach to multimodal representation learning.

[Concern 2] On clearer presentation.

Thank you for your valuable feedback on improving the presentation and clarity of the Multimodal Fusion Bottleneck (MFB) in our work. In Figure 2, the MFB module is depicted as encompassing the principles represented in Equations 12 and 13. The MFB principle is fundamentally about balancing the mutual information between shared and instance features, as elaborated in the introduction section in the manuscript. Specifically, it aims to maximize the mutual information between each modality and the instance, while simultaneously minimizing the mutual information between the instance and the aggregated modality information. This principle is visually represented in Fig. 1(c), where the white and shaded circles denote modality and instance information, respectively, and the slashed area symbolizes the cross-modality shared information. The goal of MFB is to ensure that the instance information (represented by the shaded circle) covers as much of the shared multimodal information (denoted by the slashed area) as possible. This alignment effectively constrains the solution space, directing the model’s learning process to focus primarily on the shared information across modalities.

Dear Reviewer scVn,

We express our gratitude for your valuable time in reviewing our submission. We welcome any further questions you may have regarding our work or rebuttal. Please do not hesitate to share any additional concerns. We are always open to further discussion.

Sincerely,
Authors.

We thank the reviewer for the careful reading and valuable feedback on our submission. We are deeply encouraged that the reviewer appreciated our idea and the presentation is clear and we apologize for the typos and have corrected them in the revised manuscript. The acceptance of our work is particularly heartening, and we extend our sincere thanks for the insightful review. Here we provide more explaination towards detailed descriptions of our method. We hope that our response addresses your concerns.

[Concern 1] On the introduction on hypergraph propagation module.

Thank you for your feedback regarding the clarity of Figure 3, which illustrates our Hypergraph propagation module.

In essence, the module applies an attention mechanism to the vertices, which represent features from each modality, and the hyperedges, which aggregate these features to capture instance-level semantics. Attention weights are determined via a scaled dot-product attention function, allowing the model to dynamically emphasize the most relevant modal features during aggregation. The process involves iteratively updating hyperedge features by aggregating information from the vertices, followed by updating the vertices with the new hyperedge information. This bi-directional flow ensures that both modality-specific and cross-modal information is captured and refined over multiple iterations. The final output after $L$ layers of propagation is a set of hyperedge features that embody the learned instance representation, ready for downstream tasks. The whole process can be succinctly described by Eq.8-11 in our manuscript.

[Concern 2] On the pre-trained encoder issue.

Thank you for highlighting the encoder aspect of our method. As detailed in Appendix F of our manuscript, HyperRep utilizes pre-trained models for feature extraction from different modalities: ResNet for video, DAVEnet for audio, and Word2vec for textual modality features​​. The choice of these pre-trained models is motivated by their proven effectiveness in capturing rich, modality-specific features, which are essential for multimodal tasks.

Regarding the concern about training from scratch, it's widely acknowledged in the multimodal learning community that most methods benefit from pre-extracted features or a fixed backbone due to challenges in aligning different modalities and the prohibitive computational costs associated with training complex models from scratch. HyperRep focuses on leveraging initial multimodal features to learn instance representations, rather than retraining these backbones. Therefore, ablating the pre-trained models is not feasible within the scope of our approach.

In terms of the experiments, we have ensured a fair comparison by using identical input features across methodologies wherever possible. For instance, in the clustering task, except for SeLaVi, which operates directly on raw videos, all other compared methods use the same input features as HyperRep. This consistent approach extends to our text-to-video retrieval and action localization tasks, where several methods employ the same encoders as ours. We also include methods using different backbones for a comprehensive comparison, as shown in Table 2 and Table 4.

[Concern 3] On the instance feature utilzation.

Thank you for the opportunity to clarify the terminology used in our manuscript. We would like to emphasize that in the context of our HyperRep model, the terms 'instance feature' and 'instance hyperedge feature' are used interchangeably and denote the same concept. The apparent distinction is a matter of terminology, for which we apologize for any confusion.

To elucidate further, our model does not utilize raw instance features directly from the input data. Instead, it learns a composite representation that encapsulates the combined information of multiple modalities, which we refer to as the 'instance hyperedge feature'. This learning is achieved through a hypergraph propagation process where individual vertex features (modality features) are aggregated to form hyperedges, representing the instances. Thus, the 'instance hyperedge feature' effectively serves as the learned 'instance feature'.

[Concern 3] About leaverage the lower bound and upper bound of the MFB for optimization.

We appreciate the opportunity to clarify the optimization process concerning the bounds of the Multimodal Fusion Information Bottleneck (MFB). As delineated in our manuscript, the MFB principle introduced in Eq.12 is designed to balance maximizing mutual information between the instance and each modality while minimizing mutual information between the instance and the collective modalities. To facilitate the optimization of this principle, especially when the probability distribution is unknown, we derived tractable estimations for both upper and lower bounds of mutual information as outlined in Proposition 1.

Concretely, we sought the lower bound for the first term in Eq.12, as it is negated, and the upper bound for the second term. These bounds are crucial for transforming the intractable mutual information components into a computable loss function suitable for gradient-based optimization methods. The upper bound of Eq.12 is thus formulated in Eq.13, which we refer to as the MFB loss. This loss function encapsulates the InfoNCE loss as the lower bound for the mutual information, effectively allowing us to optimize the MFB principle using backpropagation, as the InfoNCE loss provides a gradient-friendly surrogate objective.

The detailed derivation of the MFB loss, including the application of Proposition 1, which presents the mutual information estimations, is meticulously documented in Appendix C of the manuscript. This derivation ensures that the optimization process aligns with the theoretical underpinnings of MFB, guiding the learning process to focus on shared multimodal information, thereby effectively constraining the solution space and directing the model’s attention towards inter-modality shared information, as intended by the MFB principle.

We trust this explanation elucidates the optimization strategy for the MFB principle and its implementation within our model. We are grateful for your feedback, which has allowed us to better communicate the theoretical and practical aspects of our approach.

[Concern 4] On the contribution issue.

We appreciate your feedback and the opportunity to more explicitly clarify the contributions of our paper, HyperRep.

HyperRep introduces a novel hypergraph-based framework for capturing high-order correlations across and within multiple modalities, which significantly diverges from existing methods in the literature. This framework offers a comprehensive approach to model the inter- and intra-modality high-order correlations in multimodal data. A key element of our approach is the proposed Multimodal Fusion Bottleneck (MFB) principle, a unique mechanism designed to optimize information fusion by balancing mutual information within and across modalities. This principle is rigorously defined, with detailed proofs, derivations, and calculations.

The effectiveness of HyperRep is further evidenced through extensive experiments. Contrary to incremental advancements, our model demonstrates significant improvements across several benchmarks. In clustering tasks, for example, HyperRep outperforms second-best methods by substantial margins, achieving an average improvement of 6.45% across three datasets. Our rigorous ablation studies also highlight the significant impact of each component, including the high-order correlations and the MFB loss function.

Additionally, the adaptability and scalability of HyperRep have been proven across various downstream tasks, such as text-to-video retrieval and temporal action localization. Complementing these, we conducted comprehensive analyses, including sensitivity analysis, computational complexity analysis, convergence analysis, and quantitative analysis, all of which underscore the robustness and versatility of our method.

We trust that these points adequately demonstrate the significant contributions of HyperRep to the advancement of multimodal representation learning.

We thank the reviewer for the careful reading and valuable feedback on our submission. We are deeply encouraged by the reviewer's appreciation of our idea's originality and the detailed, extensive nature of our experiments. The acceptance of our work is particularly heartening, and we extend our sincere thanks for the insightful review. In the following sections, we provide more detailed descriptions and illustrations to further elucidate the novelty, contribution, and effectiveness of our method. We hope that our response comprehensively addresses your concerns.

[Concern 1] On the novelty issue against HAN[1].

Hypergraph Attention Networks (HAN) [1] are specialized for the Visual Question Answering (VQA) task, employing hypergraphs to create co-attention maps between multimodal inputs and considering structural similarity to embody high-level semantic relationships within symbolic information. In our scenario, where the extraction of multi-level high-order correlations and information propagation from vertices to hyperedges for downstream tasks is essential, the methods employed by HAN prove to be inadequate. The random walks sampling method and graph-matching techniques utilized by HAN cannot achieve these objectives, as HAN only regards hyperedges as alternatives to subgraphs without specific semantic meaning.

HyperRep, our proposed method, addresses these limitations by constructing modality hyperedges using the K-NN algorithm, thereby capturing high-order semantic correlations within modalities. Instance hyperedges are naturally constructed to extract cross-modal correlations within instances. Additionally, the attention mechanism in HyperRep aims for accurate information propagation instead of just extracting a joint representation across modalities, further differentiating it from HAN.

Despite the self-supervised learning part, the discrepancies between HAN and HyperRep in terms of hyperedges' nature, attention mechanisms, construction methods, and types of hyperedges (with HyperRep considering cross-modality hyperedges while HAN neglects) underline the significant differences between the two. Though both approaches utilize hypergraphs and attention mechanisms, their underlying philosophies and methodologies are task-specific and markedly distinct. HAN's symbolic graph construction and graph-matching method are not directly applicable to our scenario, thereby emphasizing the novelty of HyperRep.

[1] Hypergraph Attention Networks for Multimodal Learning, CVPR 2020.

[Concern 2] On the performance gain.

We appreciate your observation and would like to address the concern regarding the perceived incremental nature of our work. HyperRep represents a substantial leap forward rather than a mere extension of existing methodologies. Our approach introduces a novel self-supervised learning framework within the realm of hypergraphs, which distinctly captures high-order correlations across multimodal data—a significant departure from prior models.

The performance gains of HyperRep are far from incremental. Our model demonstrates a marked improvement over existing state-of-the-art methods. For instance, in clustering tasks, HyperRep leads by significant margins such as 8.2%, 6.9%, and 16.7% in Acc, NMI, and ARI metrics respectively on the AVE dataset​​. This trend of outperformance is consistent across various datasets and tasks, as shown in our extensive experimental results. In text-to-video retrieval tasks, our method observes improvements of 10.5%, 4.4%, and 10.4% in Recall@1, Recall@5, and Recall@10 respectively, compared to the next best method​​. Such substantial gains underscore the effectiveness of our model's ability to learn nuanced, cross-modal representations that are deeply grounded in the data's semantic structure.

Furthermore, our approach to constructing hyperedges that capture both inter- and intra-modality semantic correlations, coupled with the introduction of the MFB principle to constrain the solution space, is unique and instrumental to our model's performance. The observed improvements are indicative of HyperRep’s enhanced capacity to understand the underlying clustering structure and the genuine similarities among instances, as evidenced by the strong results.

In essence, the development and application of the MFB loss function, which optimizes for shared information across modalities, represent a key innovation that substantially enhances the model's performance. These are not minor incremental updates; they are fundamental contributions that redefine the approach to multimodal representation learning.

We believe these points collectively illustrate that HyperRep is a significant and distinct contribution to the field, offering far-reaching implications for multimodal learning research.

We thank the reviewer for the careful reading and valuable feedback on our submission. We are deeply encouraged that the reviewer appreciated our idea and the presentation is clear. The acceptance of our work is particularly heartening, and we extend our sincere thanks for the insightful review. Here we provide more interpretable analysis, quantitative results and detailed descriptions of our method. We hope that our response addresses your concerns.

[Concern 1] On the computational complexity issue of hypergraph construction.

Thank you for bringing up this point. We acknowledge that the conventional k-NN algorithm used in hypergraph construction can be computationally intensive. However, it is crucial to note that this process, with a complexity of $O(n^2d)$, is a preprocessing step rather than an inherent component of our model, and it does not impact the model's efficiency during training and testing​​.

In practice, our hypergraph construction has shown acceptable preprocessing times, and we mitigate scalability concerns in two ways:

1. Offline Construction: The hypergraph is constructed offline and saved, allowing for a one-time computational cost that does not recur during model training or inference.
2. Alternative Methods: While we currently employ the k-NN algorithm, alternative methods that can group semantically similar vertices with lower complexity can be explored in future work.

Regarding the handling of complex data such as high-resolution images, the computational burden does not increase with data complexity since the feature extraction is handled by pre-trained models.

[Concern 2] On the interpretability of the learned representations.

We thank you for your constructive feedback regarding the interpretability of our learned representations. To address this, we have conducted a t-SNE analysis to provide a visual exploration of how our HyperRep framework captures high-order correlations, which is integral to its performance.

As shown in the newly included t-SNE visualization in the revised manuscript (Fig.10, Appendix L), the distinct clusters formed by the data points illustrate the efficacy of our model in learning separable and interpretable representations. Each color in the visualization represents a different category, revealing how well the HyperRep framework groups instances with high semantic similarity.

The visualization highlights that representations learned with MFB loss (Fig.10 (d)) result in more distinct and cohesive clusters, compared to those relying solely on InfoNCE (Fig.10 (c)). This supports the quantitative findings that high-order correlations play a significant role in the final performance scores, demonstrating their importance in capturing complex multimodal interactions that InfoNCE alone might not fully encapsulate.

Moreover, the ablation study without high-order correlation (Fig.10 (b)) falls short in terms of clustering quality, as evidenced by the more dispersed clusters and less distinct group boundaries. This visually corroborates the quantitative analysis which shows a notable drop in performance metrics when high-order correlations are omitted, underscoring their role in enhancing the discriminative power of the learned representations.

Through this qualitative analysis using t-SNE and the accompanying quantitative results, we hope to have adequately addressed your concerns regarding the interpretability of the learned representations.

[Concern 3] On quantitative results of text-to-video retrieval task.

Thank you for your suggestion. In response, we have incorporated the results of two baseline methods, MCN and MIL-NCE, into Figure 9 in our revised manuscript.

[Concern 4] About the selection of k-NN for training and validation.

Thank you for giving us the opportunity to clarify the selection of the k-NN parameter for both the training and validation phases in our experiments. We have set the k value for the k-NN method within the hypergraph construction to 7. As detailed in Appendix J, our sensitivity analysis indicates that this parameter is not highly sensitive, allowing for a range of values to be effective. This setting is consistently applied across all datasets to ensure methodological consistency, and it is used to construct the modality hypergraphs for the entire dataset, encompassing both training and testing instances.

For clustering tasks, the entire dataset is employed for both training and validation, hence the hypergraphs are constructed on the full dataset at once. For tasks such as text-to-video retrieval and action localization, which are performed in a semi-supervised manner as described in Appendix G, the hypergraph structure is also constructed using the full dataset. The only exception is that the instance hypergraph in the test set is excluded from this process. However, this does not impact the construction of the modality hypergraph, which still utilizes k-NN across the entire dataset.