Bootstrapping Audio-Visual Segmentation by Strengthening Audio Cues

Thank you for your valuable comments. Below are our responses.

Q1: The proposed bidirectional framework is not entirely novel and has been previously explored in [a,b,c]，and the use of reconstruction to preserve the semantic meaning of the audio can also be viewed as a preliminary version of [d].

A1: Our bidirectional framework is quite different from [a,b,c]. The novelty of our bidirectional framework (BAVD) lies in 3 aspects: bi-direction, continuity and motivation emphasis. The modality interaction is bidirectional and continuous, and the emphasis of our motivation is on enhancing audio cues to evade modality imbalance. [a,b,c] all focus on building better correlation between audio-visual features instead of on audio cues. Moreover, [a] interacts modalities only once so it lacks continuity; [b] continuously interacts modalities but it is unidirectional, which is not as sufficient as ours; [c] use different cross attention mechanism as ours and also its interaction is only once and unidirectional.

Also, our AFR module is not a preliminary version of [d], since the two differs much in motivation, structure and task. The motivation of [d] is to use audiovisual information already present in video to improve self-supervised representation learning. To achieve this, [d] design an audiovisual masked autoencoder that can use audio and mask video tokens to reconstruct video tokens. However, the motivation of our AFR is to further curtail audio information loss. To achieve this, we design a linear module to use visual feature to reconstruct lost audio feature and propose a loss function to constrain this process. Also, our AFR’s task is audio-visual segmentation while the task of [d] is to improve self-supervised representation learning.

Q2: The dataset bias in this paper aligns with the finding in [e]. Properly cite this prior work and discuss it if necessary.

A2: We have cited this work and highlighted the place in yellow in our updated version. Notably, [e] and us adopt different ways to mitigate data bias. [e] discovered that AVS datasets are biased and proposed a relatively unbiased AVS benchmark to mitigate this issue from dataset level. While we think the bias may be caused by lack of enough audio guidance and seek to reconstruct lost audio cues with AFR module to mitigate the issue. Both of our solutions work well.

Q3: Include the SOTA method AQFormer in the experiment section and discuss the discrepancy. Also, list the AVSegformer's performance listed in table 6 when comparing on AVSS.

A3: Thank you for your question. We have added it in our updated pdf file and highlighted it in green color in Table 1. The reason why we did not compare with AQFormer is that this paper belongs to concurrent work, since it was newly released on arxiv on Sep.18.2023 and has not been officially published before. Here we show our comparison with AQFormer in Table. i., where our AVSAC generally surpasses AQFormer more significantly than AQFormer surpasses ours, despite the inferiority on MS3 with ResNet-50 as visual backbone.

We provide AVSegformer's performance on AVSS on Table.ii where our AVSAC is superior. We have updated our pdf file and added this to Table. 6 in the appendix.

Table. i: Comparison between AVSAC and AQFormer on two S4 and MS3.

Table. ii: Comparison between AVSegFormer and our AVSAC on AVSS.

Dear Authors,

Thank you so much for the detailed explanation of my comments. The author has effectively addressed my concerns regarding Q2 and Q3. I agree with the author's decision for Q3, considering that it is not necessary to include AQFormer in the current version, given that it was a newly released paper.

I appreciate the author highlighting the differences between the proposed work and the listed literature against Q1. However, I still believe that the proposed bidirectional architecture, linking between two modalities, offers sufficient novelty despite the high similarity mentioned previously. In addition to the literature mentioned earlier, there is also an audio-visual paper [g,h] that adopted the same bidirectional modulation, for example: \hat{f}{t}^{a} = f^{a} + self-attention(f^{a},f^{a},f^{a}) + cross-attention(f^{a}, f^{v}, f^{v}), \hat{f}{t}^{v} = f^{v} + self-attention(f^{v},f^{v},f^{v}) + cross-attention(f^{v}, f^{a}, f^{a}), and the claim about the cross-modality continuous interaction is the same as [I, h]. In the referential expression segmentation paper [I], they proposed Language Feature Reconstruction (LFR), which shares a similar concept with the AFR module. The LFR module aims to prevent the loss or distortion of language information in the extracted features which is very similar to AFR. Additionally, the issue of modal imbalance has also been addressed in [h].

In short, the proposed method and its motivation have significant overlap with the previous method [h] in terms of:

1. bidirectional/mutual attention and continuous, in-depth interactions
2. reconstruction of weak modality features to prevent information loss.
3. modal imbalance issue.

Thus my original concern about novelty and originality of the proposed method remains unaddressed.

[g] Lin, Y.B., Tseng, H.Y., Lee, H.Y., Lin, Y.Y. and Yang, M.H., 2021. Exploring cross-video and cross-modality signals for weakly-supervised audio-visual video parsing. Advances in Neural Information Processing Systems, 34, pp.11449-11461.

[h] Liu, C., Ding, H., Zhang, Y. and Jiang, X., 2023. Multi-modal mutual attention and iterative interaction for referring image segmentation. IEEE Transactions on Image Processing.

[I] Zhou, J., Wang, J., Zhang, J., Sun, W., Zhang, J., Birchfield, S., Guo, D., Kong, L., Wang, M. and Zhong, Y., 2022, October. Audio–visual segmentation. In European Conference on Computer Vision (pp. 386-403). Cham: Springer Nature Switzerland.

Dear reviewer:

Thank you for your reply. We clarify our differences with your mentioned works in task, motivation, structure etc. below. Hope our responses may address your concerns.

Q1: Audio-visual paper [g] adopted the same bidirectional modulation, and the claim about the cross-modality continuous interaction is the same as [I, h].

A1: Our bidirectional module is very different from [g] since the modules are designed for different tasks, which also lead to different interaction modes.

Task: The task of [g] is audio-visual video parsing, which is a categorization task that aims to temporally parse a video into audio or visual event categories. While our task is AVS, which needs audio guidance to help segmenting sounding targets in a video.

Interaction Mode: The interaction of [g] is at a inter-video level, not continuous, and treats audio and visual signals equally. [g] first aligns audio and video features and then does cross attention for interaction only once for each video. [g] also equally treats audio and video signals since its aim is modality interaction. However, the interaction mode of our bidirectional module is at intra-video level, continuous and lays emphasis on audio cues. We also do not adopt self-attention in our audio branch as [g] does. Our interactions are done in each video frames. We think audio cues should be emphasized and enhanced because it easily gets lost during training, otherwise other silent objects may be falsely segmented. We use enhanced visual features to extract audio signals useful to AVS from audio cues and then continuously feed these useful audio signals back to visual decoder, so we do not enhance audio cues again with the self-attention. Our experiment also proves that adding self-attention to our audio branch is harmful (Figure 7 (b) in our updated appendix).

Our cross-modality continuous interaction is also different from [I, h]. The interaction of [I] is unidirectional and cannot prevent audio cue loss, but ours is bidirectional and can prevent this. The cross-modality continuous interaction part of [h] is its multi-modal mutual decoder, but its structure is very different from our decoder BAVD since (1) [h]’s interaction is based on one branch while ours has two branches (2) [h]’s interaction needs to intake multi-stage vision features from encoder, while ours just needs to input the single visual feature output from visual encoder only once. The reason is that our parallel design is more efficient and makes multiple inputs not necessary.

Q2: The proposed method and its motivation have significant overlap with [h] in terms of: 1.bidirectional/mutual attention and continuous, in-depth interactions 2.reconstruction of weak modality features to prevent information loss. 3.modal imbalance issue.

A2: [h] solves the referring image segmentation (text-vision task), while we solve the AVS task. Text signals contain highly condensed semantic information and have no noise, while audio signals are not as information-rich as texts and may have noise. Since the modality differs, our designs also differs in motivation and structure:

1.The motivation of our bidirectional interaction design is to curtail audio-vision modality imbalance, while for [h] is to better fuse text-visual information. We achieve this by continuously feeding the useful audio cues to AVS extracted by visual features back to visual branch, while [h] turns to better interaction designs. For structure, [h] and our method adopt different ways to conduct attentions and the attentions are conducted in different decoder structures. Our attentions are conducted on two parallel decoders, while [h] implements cross attentions in one decoder. Also, the decoder of [h] takes multi-stage vision features as inputs to serve as k, v in cross attentions, while ours only intakes one visual input for self-attention and serve as q in our audio-guided cross attention. The reason is that our parallel design is more efficient and makes multiple inputs not necessary.

2.Our AFR module also differs from the reconstruction module of [h] in motivation, structure and location. The motivation of [h]'s module is to reconstruct text feature from visual feature, while our AFR is to reconstruct audio cues from visual features. For structure, our AFR module is simpler in structure without the position embedding and average pooling layers that [h]'s module has. For location, our AFR is located at the end of decoder to further guarantee our final visual output won’t lose audio cues, while [h]’s is placed between decoders.

3.Modality imbalance generally exists in many multi-modality tasks, and [h] notices the same issue as us despite modality difference. The differences of our solutions have been explained in the two points above.

If you have any other concern, please reply at any time. Wish you all the best!

Unfortunately, I remained unconvinced after the rebuttal. The M^3Dec [h] states clearly that they have noticed the modal imbalance issue in the attention-based network and suspect it is due to the characteristics of the Transformer’s decoder architecture. They further mention, "This implies a modal imbalance issue: the network will tend to focus more on the vision information, and the language information may be faded away during the information propagation along the network". This differs from the author’s response in Q2(1), where the design is simply described as better fusing text-visual information. Notably, the sentence in [h] exhibits significant similarity with the author’s claim on page 2 that "This implies a modal imbalance issue: the network will focus more on the visual information while neglecting the audio ones because the audio feature is not as information-rich as the visual feature, so unidirectional and deficient fusion of audio features may easily cause audio cues to fade away". However, the reference was not provided in the paper. For the feature reconstruction module, [h] tries to reconstruct the language feature from the last decoder layer’s output (multi-modal feature), which is similar to the AFR. In summary, the notable overlap between the proposed method and [h] raises doubts about the originality and distinctive contribution of the proposed model.

Dear Reviewer xbK3,

Sorry for bothering you again. Since there is less than a day left until the end of the discussion, we would like to further discuss with you to address your concerns about our overlaps with [h] in solving the similar modality imbalance issue, sentence similarity with [h] without reference and AFR’s similarity with [h]’s language reconstruction module in reconstruction from the last decoder layer’s output.

As for the modal imbalance issue, we have already explained in our previous response modality imbalance issue generally exists in multi-modality fields and has been studied by many other works including [h]. Considering the modalities of our task are different from [h]’s, when the modalities change, the solutions also differ.

Regarding the issue you raised about sentence similarity, we have updated our describtion in the paper. Furthermore, we have cited [h] in the related work section and added more discussions on our differences with [h] in our updated paper. Specifically, the audio and visual feature extraction process of [h] only involves single modality because the multi-stage vision features are extracted from a ViT encoder without any audio cue participation. While we propose to embed visual and audio signals to feature extraction process, so that the interaction can be more efficient, and the decoder features can focus more on our interested sounding object areas.

For the AFR’s similarity with [h]’s language reconstruction module, we repeat and reshape audio cues to align with visual features (AGV) of higher resolution than [h]’s in every spatial location.

In summary, we believe that our method differs significantly from h in both design and implementation aspects.

Sincerely, 
Authors

Thank you for your valuable comments. Below are our responses.

Q1: I do not understand the aftermath of such imbalance - why the model's being focused more on visual information is a bad thing? This needs to be motivated a bit more.

A1: Focusing more on visual information is ok but should be in moderation. Present AVS methods focus so much on visual information that neglects that some effective audio features will get lost during training and therefore cannot provide enough audio guidance for correct audio-visual segmentation of sounding objects. This can be visually observed from the right part example of Figure 4 in our paper, where irrelevant objects get mistaken as sounding objects because audio cues do not receive sufficient focus. After all, the AVS task is acoustic in nature as reviewer 7Z8k agrees.

Q2: Do you see any possibility that the architecture can be optimized to improve the efficiency?

A2: Yes, thank you for your valuable suggestion. More efficient and lightweight attention module, for example the squeeze-enhanced axial attention of SeaFormer [1], can be adopted to replace our used deformable attention, self-attention and cross attention modules to increase efficiency. Increasing model efficiency will be our future direction.

Q3：At the beginning of Section 3.2 - what is the dot product in MLP() means in Equation 1?

A3: It is the inner product of two metrics, multiplying $F_{VGA}$ of size 5 $\times$ 300 $\times$ 256 with $F_{AGV}$ of size 5 $\times$ 256 $\times$ 128 $\times$ 128 to get a new matrix of size 5 $\times$ 300 $\times$ 128 $\times$ 128.

[1] Wan, Qiang, et al. "SeaFormer: Squeeze-enhanced Axial Transformer for Mobile Semantic Segmentation." The Eleventh International Conference on Learning Representations. 2022.

Dear Reviewer MwLf:

Thank you for your valuable comments on our paper. We have explained the aftermath of modality imbalance, way to improve efficiency and what does the dot product in MLP () means in Equation 1. It would be nice if you could let us know your opinions on our responses. Hopefully, our responses can solve your concerns. If you have additional questions, we would be happy to answer.

Sincerely, Authors

Thank you for your valuable comments. Below are our responses.

Q1: Audio Feature Reconstruction seems to be very similar to the bidirectional generation module in AVSBG [1].

A1: Our AFR module is quite different from the bidirectional generation module in AVSBG in 3 aspects: motivation, structure and constraint loss.

Motivation: Our AFR aims to evade harmful data bias and also curtail audio information loss by reconstructing lost audio feature from AGV, while the bidirectional generation module in AVSBG just aims to build a strong correlation between visual and audio signals.

Structure: The bidirectional generation module in AVSBG adopts a mask encoder-decoder structure to turn masked visual feature to reconstructed audio feature, while our AFR is simpler and more efficient in structure by using only two linear layers for projection.

Constraint loss: AVSBG designs a consistency loss to minimize the distance between the norms of the original and reconstructed audio feature along the last dimension, while we adopt MSE loss to minimize the distance between the projected reconstructed audio feature and the projected multi-model feature.

Q2: I can't see from Figure 5 which areas the attention mechanism obviously pays attention to.

A2: The darker areas are visually very different from other lighter areas and highlight the attention regions of the network. You may regard the dark areas as where the attention pays attention to.

Dear Reviewer qkmM:

We thank so much for your valuable comments on our paper. We have clarified the differences between our AFR module and the bidirectional generation module in AVSBG and the areas in Figure 5 that the attention mechanism pays attention to. Could you let us know your thoughts on our responses? We hope our responses can address your concerns. We would be delighted to answer additional questions if any.

Sincerely, Authors

Thank you for your valuable comments. Below are our responses.

Q1: Not sure however if cross model attention has been used for AV segmentation task.

A1: As far as we know, we are the first to use the cross model attention mechanism in AVS task, since most other AVS works only adopt either Attn(Q_a, K_v, V_v) or Attn(Q_v, K_a, V_a) instead of combining both. Though cross model attention has been used in multi-modal related tasks, the characteristics and motivation are always different from ours. Other works generally use this mechanism to build better correlation between two modalities and is often used only once, which makes the interaction insufficient. Prior works also did not consider from the modality imbalance aspect. While our design features both bidirectional and continuous modality interaction and our motivation is to enhance audio cues to mitigate modality imbalance issue.

Q2: Not clear about the claims of richness of information in audio. Removing self-attention leads to better results, discuss this in a bit more detail and clarify how this conclusion is reached.

A2: We claim in paper that the audio feature is not as information-rich as visual feature, so it may not need too much encoding operation, which means merely using cross attention is enough. The reason for the performance dropping when adding self-attention may be that adding self-attentions leads to overfitting. We demonstrate this by comparing the training and eval mask loss functions w/ and w/o self-attention and update the line chart in Figure 7 (b) in the appendix of our pdf file. We find that adding self-attention makes the mask loss drops lower while the test result is inferior to no self-attention case, which means overfitting.

Q3: Define L_dice, L_mask and L_afr more clearly.

A3: Thank you for your advice. We have clarified the definitions and highlighted them in orange color in the updated version. $L_{mask}$ = $L_{focal}$ + $L_{dice}$, $L_{afr}$ = MSE ($F_{afr}$, $F_{proj}$).

Q4: How does the overall method behave when the sounding object is outside of the field of view or the sounding objects move in and out of field of view.

A4: We visualize this case by comparing with AVSegFormer and AVSBench and the visual result is updated in Figure 7 (a) in our appendix. We find that present methods including ours still cannot well tackle this issue since in the last column the sounding ambulance car moves out of the view but all three methods falsely segment the motorcycle as the sounding object. This will be our future research direction. However, our method is still better than the other two since only our method correctly segments the ambulance car in view in the fourth column.

Q5: The AFR loss is essentially forcing feature similarity between F_AGV and F_audio (through the feature learning). Why would this reduce the type of bias mentioned in section 3.3 (single and multi-source etc.)

A5: We mitigate data bias from audio feature preservation aspect since we try to evade the harmful case where the network happens to “guess” the right target region even if the audio information has been lost or distorted. To achieve this, we design AFR module to reconstruct lost audio feature from AGV so that the "guess" case is curtailed. We have updated our pdf file and added visual comparison results with and without AFR to visualize its effect in Figure 8 in the appendix.

Dear Reviewer 7Z8k:

We deeply appreciate your valuable comments on our paper. We have provided corresponding responses to your questions. It would be great if you could let us know your thoughts on our responses. We hope our responses can address your concerns. We would be happy to answer additional questions if any.

Sincerely, Authors