EquiAV: Single-modal Equivariance Promotes Audio-Visual Contrastive Learning

Dear Reviewer 4TFL,

Thank you for your constructive and insightful comments. We respond to your questions point-by-point in the following paragraphs.

 

---

it is solely applying the methodology of [1] to the modalities of audio and video, and then fine-tuning the hyperparameters as would be done for any uni-modal framework. It is therefore unsurprising, and a direct conclusion of [1], that this would work in some way.

As self-supervised learning through invariance has been extended, some studies have shown that applying the concept of equivariance leads to an additional performance boost in various domains such as images and text. Specifically, in the image domain, the concept of equivariance has been applied to SimCLR through developments like E-SSL [2] and EquiMod [1]. Similarly, in the text domain, SimCSE [3] has been advanced with the application of equivariance as seen in DiffCSE [4]. These papers show the process of adopting the idea of equivariance for representation learning in each domain. On the other hand, to our knowledge, there is no prior work that incorporates equivariance for audio modality and audio-visual representation learning. Along these lines, we aim to design a framework for employing equivariant representation learning in audio-visual representation learning and demonstrate its benefits. However, extending the single-modal equivariant representation learning to multi-modal representation learning is not trivial. Our additional ablation studies (Table 3 in the revised manuscript) reveal that applying equivariance to inter-modal latent spaces is not beneficial and may even hinder the learning of audio-visual correspondences and joint representations. We found that augmentation-related information and audio-visual correspondence need to be learned in separate latent spaces to overcome this challenge. In addition, simply replacing the single-modal invariant representation learning with the equivariant one doesn’t lead to performance improvement. Consequently, we have meticulously designed a framework that maximizes the benefits of equivariance, while avoiding negative impacts on learning. Each component of our framework has been verified to be effective through ablation studies, including:

1. Loss function (Table 5),

2. Data augmentations for effective equivariant representation learning (Table 4), and

3. Projection head and augmentation encoder (Table C in Appendix).

Taken together, all of these factors indicate that our contribution goes beyond simply applying existing techniques to create a new and effective framework for audio-visual representation learning.

 

---

The lack of novelty would be acceptable if the results were state-of-the-art by far, but in Table 1 they are outperformed by MAViL in some cases, and in Table 2 they are only compared with a single previous work.

There are a couple of key factors to consider in the performance of MAViL:

• Firstly, MAViL utilizes 8 frames for training, in contrast to the single frame used in CAV-MAE and our work. Although using more frames improves performance, it significantly increases computational costs (MAViL uses 64 gpus vs. EquiAV uses 8 gpus).
• Secondly, MAViL adopts a two-stage training approach. The first stage involves training the model with masked modeling and contrastive learning techniques, followed by a second stage of retraining using knowledge distillation (KD). It's worth noting that KD could potentially boost performance in other studies too, such as CAV-MAE and Audiovisual MAE, including our own. In addition, we have compared our work with various other methods by reporting zero-shot retrieval results on the MSR-VTT dataset, as shown in Table A. The revised manuscript also includes further fine-tuning experiments on UCF101, HMDB51, and ESC50, which can be found in Table B.

 

---

As a side note, I don't think it's reasonable to grey out MAViL as concurrent work in Table 1 - the paper was released on arXiv in Dec. 2022.

As you correctly noted, MAViL was initially uploaded in December 2022. Our decision to classify it as concurrent work stems from the fact that at the time of our submission in September 2023, we were referencing its revised version from July 2023. Given that MAViL demonstrates performance comparable to our study on the same benchmarks, we included it in our main results following the guideline of covering all relevant papers. If there has been any oversight on our part in this regard, we are open to making any necessary revisions to how MAVil is mentioned in our paper.

I really appreciate the detailed response.

Regarding novelty, I understand your argument and am not undermining the effort it took to make this configuration work, but still firmly believe that the contribution is not sufficient for a full ICLR paper.

Regarding MAViL, I understand your argument since the results have been updated in the July version but even then I think it's unreasonable to grey it out as concurrent work. I think it's a reasonable comparison.

Thank you for adding RAVEn as a reference, correcting the typo, and agreeing to release the code.

Overall, while I appreciate the authors' detailed responses and new experiments, I don't think it's appropriate to raise the score since I still think the lack of novelty is a major issue.

Dear Reviewer A7KE,

Thank you for your valuable suggestions to improve our paper. Please see the point-by-point responses below.

 

If the loss is formulation (1) is contrastive in nature then the $\mathcal{L}$ should not just output the dissimilarity as the optimization minimizes $\mathcal{L}$ and contrastive loss is about maximizing similarity of aligned data and minimizing similarity of non-aligned data. I assume that the author's meant a formulation of contrastive loss and not just a dissimilarity measuring mechanism. Please clarify.

First, it's correct to consider $\mathcal{L}$ as a contrastive loss, as you've mentioned. In our work, we aimed to establish a more general formulation for representation learning. Specifically, we formulate it as maximizing the similarity between pairs of inputs, which is equivalent to minimizing their dissimilarity. The four equations described in Section 2 of our manuscript are designed to be applicable not only to contrastive learning but also to other forms of representation learning. Furthermore, these formulations are adaptable to a variety of techniques used to prevent collapse in Siamese networks, such as the use of 'stop gradient', illustrating their broad applicability in recent advancements in the field.

 

---

Conceptually, the augmented and non-augmented variants can be considered as two different modalities. Therefore, standard contrastive loss setup (as used in Eq 11) can be used. Is there a specific reason for using the Eq 9 formulation.

As you pointed out, they can be treated as two different modalities, which would allow the use of Eq 11. However, this method halves the number of negative pairs (from 2N-2 to N-1), which is similar to decreasing the batch size. Such a reduction in negative pairs can have a negative impact on performance, particularly in the context of contrastive learning. While Eq 11 may have its advantages when using different encoders for significantly different spaces, like in video-audio or image-text pair, we can conclude that Eq 9 is more suitable for intra-modal training.

 

---

Furthermore, the reasoning behind inclusion of 'positive pair ...in the denominator ' is not clear. The appendix does seem to allude to the different weights during gradient calculations but does not elaborate on why those specific weights should necessarily be a factor. Table 5 does give results that demonstrate that it can be important but there is not an explanation as to why it is so.

As explained in Appendix A of the manuscript, including the positive pair in the denominator results in a relatively larger update for hard positives. This becomes particularly advantageous when stronger augmentations lead to an increased frequency of hard positives. Training with more hard positives, alongside the application of equivariance, substantially enhances the model's capability to comprehend detailed features in single-modal contexts. The intra-modal representation quality plays a pivotal role in understanding and integrating different modalities. Consequently, the semantically rich intra-modal representation promotes effective learning of audio-visual correspondence and audio-visual joint representations. We will add a detailed analysis in Appendix A.

 

---

There are two types of features for each modality, one from the head corresponding to the inter-modal space and the other from the intra-modal space. Perhaps I missed it, but it is not apparent which features were used in the experiments.

We employed features projected into the inter-modal space for all downstream tasks, and Table C in Appendix C.2 demonstrates which features yield the best performance. As you mentioned, we conducted experiments using features projected into the intra-modal space, as well as features that were processed solely through the encoder without any heads. The results indicate that features projected into the inter-modal space exhibit the best performance.

 

---

It would be good to dig deeper into this facet perhaps through heatmaps to show stronger evidence as Table 4 results show that the proposed setup is good for the given downstream tasks which is perhaps not enough evidence to strongly claim the models ability to capture augmentation related information.

In response to your suggestion, we have further demonstrated the capability of EquiAV to capture augmentation-related information by conducting a visual sound source localization task. The qualitative results are detailed in Figure B in Appendix D of the revised manuscript. The results demonstrate that EquiAV consistently maintains audio-visual correspondence and accurately identifies the sound source, even in scenarios involving augmented audio-visual inputs. This highlights the strength of our method in not only capturing augmentation-related information within each modality but also in preserving audio-visual correspondence.

Dear Reviewer eQDc,

We appreciate your valuable comments. We list the responses to the concerns below.

 

---

Applying intra-modal representations equivariant to the augmentations has been proven effective in previous research. What is the novelty of this article?

As self-supervised learning through invariance has been extended, some studies have shown that applying the concept of equivariance leads to an additional performance boost in various domains such as images and text. Specifically, in the image domain, the concept of equivariance has been applied to SimCLR through developments like E-SSL[2] and EquiMod[1]. Similarly, in the text domain, SimCSE[3] has been advanced with the application of equivariance as seen in DiffCSE[4]. These papers show the process of adopting the idea of equivariance for representation learning in each domain. On the other hand, to our knowledge, there is no prior work that incorporates equivariance for audio modality and audio-visual representation learning. Along these lines, we aim to design a framework for employing equivariant representation learning in audio-visual representation learning and demonstrate its benefits. However, extending the single-modal equivariant representation learning to multi-modal representation learning is not trivial. Our additional ablation studies (Table 3 in the revised manuscript) reveal that applying equivariance to inter-modal latent spaces is not beneficial and may even hinder the learning of audio-visual correspondences and joint representations. We found that augmentation-related information and audio-visual correspondence need to be learned in separate latent spaces to overcome this challenge. In addition, simply replacing the single-modal invariant representation learning with the equivariant one doesn’t lead to performance improvement. Consequently, we have meticulously designed a framework that maximizes the benefits of equivariance, while avoiding negative impacts on learning. Each component of our framework has been verified to be effective through ablation studies, including:

1. Loss function (Table 5),

2. Data augmentations for effective equivariant representation learning (Table 4), and

3. Projection head and augmentation encoder (Table C in Appendix).

Taken together, all of these factors indicate that our contribution goes beyond simply applying existing techniques to create a new and effective framework for audio-visual representation learning.

 

---

The specific details of Figure 1 are missing. What is the specific meaning of augmentation level? What specific settings were used to draw this figure?

We drew Figure 1 based on the ablation results in Table 4. The term "augmentation level" refers to the variety and strength of applied augmentations, which correspond to the augmentation settings described in each row of Table 4. For example, augmentation level 0 corresponds to the first row, 1 corresponds to the second row, and so on. We added the notion “For detailed augmentation settings, refer to Table 4" to the caption of Figure 1. If there is a better way to describe Figure 1, we would appreciate it.

 

---

In Table 1, the performance of MAViL is better than EquiAV. What is the specific reason?

There are a couple of key factors to consider in the performance of MAViL:

• Firstly, MAViL utilizes 8 frames for training, in contrast to the single frame used in CAV-MAE and our work. Although using more frames improves performance, it significantly increases computational costs (MAViL uses 64 gpus vs. EquiAV uses 8 gpus).
• Secondly, MAViL adopts a two-stage training approach. The first stage involves training the model with masked modeling and contrastive learning techniques, followed by a second stage of retraining using knowledge distillation (KD). It's worth noting that KD could potentially boost performance in other studies too, such as CAV-MAE and Audiovisual MAE, including our own. Please note that MAViL is our concurrent work and we included it as grayed out in our main results following the guideline of covering all relevant papers.

 

---

Although the author tried to compare different pre-training methods, he did not clearly explain the advantages of equivariant representation learning.

As demonstrated in Table 4, learning representations equivariant rather than invariant to augmentations allows for more robust learning of the semantics of inputs without compromising them. Consequently, this leads to better learning of audio-visual correspondence and joint representation. We also provide additional supporting evidence through an ablation study (Table 3 in the revised manuscript) for a more comprehensive understanding of our findings.

Dear Reviewer 3ZPL,

Thank you for giving us valuable comments. We present responses to the raised concerns and questions below.

 

It extends the existing EQUIMOD [1] method to audio and visual modalities, applying it to audio-visual self-supervised learning. I do not see any specific contributions of this work to the audio-visual learning field.

As self-supervised learning through invariance has been extended, some studies have shown that applying the concept of equivariance leads to an additional performance boost in various domains such as images and text. Specifically, in the image domain, the concept of equivariance has been applied to SimCLR through developments like E-SSL [2] and EquiMod [1]. Similarly, in the text domain, SimCSE [3] has been advanced with the application of equivariance as seen in DiffCSE [4]. These papers show the process of adopting the idea of equivariance for representation learning in each domain. On the other hand, to our knowledge, there is no prior work that incorporates equivariance for audio modality and audio-visual representation learning. Along these lines, we aim to design a framework for employing equivariant representation learning in audio-visual representation learning and demonstrate its benefits. However, extending the single-modal equivariant representation learning to multi-modal representation learning is not trivial. Our additional ablation studies (Table 3 in the revised manuscript) reveal that applying equivariance to inter-modal latent spaces is not beneficial and may even hinder the learning of audio-visual correspondences and joint representations. We found that augmentation-related information and audio-visual correspondence need to be learned in separate latent spaces to overcome this challenge. In addition, simply replacing the single-modal invariant representation learning with the equivariant one doesn’t lead to performance improvement. Consequently, we have meticulously designed a framework that maximizes the benefits of equivariance, while avoiding negative impacts on learning. Each component of our framework has been verified to be effective through ablation studies, including:

1. Loss function (Table 5),

2. Data augmentations for effective equivariant representation learning (Table 4), and

3. Projection head and augmentation encoder (Table C in Appendix).

Taken together, all of these factors indicate that our contribution goes beyond simply applying existing techniques to create a new and effective framework for audio-visual representation learning.

 

---

It would be helpful to provide some specific examples to illustrate how augmentations can distort the inter-modal correspondence.

For a more straightforward understanding, we came up with a case where data augmentation causes distortion of the inter-modal correspondence. Let’s suppose that given an audio-visual pair of a barking dog, the dog disappears due to augmentation (e.g. random cropping). When using intra-modal invariant learning, the features of the original image (i.e. the dog is barking) and the augmented one (i.e. the dog disappears) are forced to become closer in the latent space. This will give the wrong signal to the model, adversely affecting the learning of correct inter-modal correspondence (i.e. the dog and the barking sound). On the other hand, in the context of equivariant learning, the embedding of the original image is transformed by the augmentation predictor and made closer to the embeddings of the augmented image in the intra-modal latent space. Then, we utilize original inputs (i.e. dog and the barking sound) that retain the semantics of the two modalities in the inter-modal latent space while ensuring that the embeddings of the augmented image and the original audio will not become directly close. This insight allows equivariance to take advantage of the effect of strong data augmentation in audio-visual representation learning.

 

---

it is unclear why the positive pair is included in the denominator.

As explained in Appendix A of the manuscript, including the positive pair in the denominator results in a relatively larger update for hard positives. This becomes particularly advantageous when stronger augmentations lead to an increased frequency of hard positives. Training with more hard positives, alongside the application of equivariance, substantially enhances the model's capability to comprehend detailed features in single-modal contexts. The intra-modal representation quality plays a pivotal role in understanding and integrating different modalities. Consequently, the semantically rich intra-modal representation promotes effective learning of audio-visual correspondence and audio-visual joint representations. We will add a detailed analysis in Appendix A.