AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation

Dear Reviewer jhmt:

We greatly appreciate your insightful and constructive suggestions, which have significantly improved our paper. Let us address your concerns point by point:

1. For W.1, we apologize for the grammatical errors in our paper and any discomfort they may have caused while reading. We have uploaded a revised version that has been carefully proofread to address these minor issues. We want to assure you that these errors are limited to the writing and do not affect the accuracy of our experiments or conclusions.

2. For W.2, we have conducted additional ablation experiments on the number of audio queries as per your request. The AVSBench dataset's AVSS sub-task is a multi-class semantic segmentation task. This dataset includes 71 different categories (including the background). We set the number of audio queries to 71 and conducted the experiments as follows:

   Number of queries	mIoU (S4)	F score (S4)	mIoU (MS3)	F score (MS3)
   8	79.3	88.9	55.8	66.0
   16	79.9	89.1	57.9	68.1
   32	79.4	88.9	56.2	66.6
   64	79.1	88.9	55.8	67.0
   71	79.2	88.9	56.0	67.2
   128	79.0	88.8	56.0	67.4
   256	79.3	89.0	57.3	67.8

   We observe that when the number of audio queries is equal to the number of categories, there was no significant improvement in the model's performance. This may be attributed to the similarities and overlaps in audio features across different categories (like laughter, singing & speech), suggesting that a smaller number of queries can effectively represent these audio characteristics.

3. For W.3, 2e present the corresponding flops and memory usage of AVESFormer with different configurations of the ELF decoder in the table below. Although there is a minor difference in flops and memory between the two, our decoder can significantly improve latency.

   decoder	#Params	fps	latency (ms)	GFLOPs	GPU memory (MB)
   ELF	127	83.5	12.0	40.3	1178
   pure transformer	128	67.1	14.9	43.9	1210

4. For W.4, we have conducted additional ablation experiments on the number of decoder layers as per your request. For easier observation, we conducted the following experiments with models having different numbers of decoder layers under the pure Transformer decoder structure as follows：

   Number of stages	mIoU (S4)	F score (S4)	mIoU (MS3)	F score (MS3)
   6	77.3	87.6	56.2	66.6
   5	75.7	87.1	52.6	65.7
   4	74.4	86.9	51.4	65.9
   3	73.0	86.7	50.5	66.2
   2	71.0	86.5	47.0	65.9
   1	66.7	86.2	33.1	64.9

   The table demonstrates the model's performance at different numbers of decoder layers. As the number of decoder layers decreases, there is a certain degree of performance degradation.

5. For W5, we realize we have missed these important works that have been really influential in the development of audio-visual segmentation. In our updated version, we've made sure to include their findings. We notice that reference [3] uses a more advanced Transformer backbone (Swin-B), which might make the comparison of results a bit uneven. To keep things clear and fair, we've shown the relevant performance in the appendix of our AVESFormer with PVT-v2 for further comparison.

Dear Reviewer g4aY:

We appreciate your effort in reviewing our article and your valuable feedback. We would like to address your concerns as follows.

Your concerns and questions are very reasonable, and they highlight what we also recognize as limitations and pain points.

• For the huge Vggish, given that most existing methods use this backbone, in order to ensure a fair comparison, we have to use the same network as our audio backbone as well. We believe that in future work, exploring how to compress this network to maintain performance while reducing the parameter overhead to meet the demands of mobile devices will be a worthwhile direction to pursue.

• For the temporal information, our network, in its current design, indeed does not utilize temporal information; each frame of data is processed independently without any inter-frame association. We also believe that incorporating temporal information could enhance the network's performance. We are confident that in our future work, we will conduct a more in-depth investigation into this aspect.

• For more datasets, our experiments are also conducted on a new AVS dataset VPO by Ref1. We report the results compared with available methods under ResNet50 backbone:

• For quantitative analysis of attention maps, we measure the attention entropy of each decoder stage by:

  $$H(A)=-\sum_i p_i\log(p_i)$$

  A higher entropy indicates a boarder attention heatmap, while a lower one means a more narrow heatmap focusing on local regions. The results is shown in the table below:

  Num of decoder stage	attention entropy
  1	1.67
  2	1.05
  3	0.76
  4	0.54
  5	0.51
  6	0.44

• For the comparison of non-transformer methods, we have demonstrated the comparison with various popular AVS models from non-transformer and transformer architectures in Table 1, providing a comprehensive analysis on its performance.

• For ethical consideration, it's essential to address a variety of ethical considerations to ensure that the technology is developed and used in ways that are respectful, fair, and accountable. First, for privacy and data consent, Audio-visual data often contain sensitive personal information, such as people’s voices, faces, or behaviors. It is essential to obtain informed consent from individuals whose data is being used for training or testing purposes. Second, for bias and fairness, Audio-visual segmentation systems may exhibit biases depending on the data used for training. Third, for impact on vulnerable populations, Audio-visual segmentation technologies can be misused to track or exploit vulnerable populations (e.g., children, marginalized groups).

Ref1: Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation. CVPR 2024

Dear Reviewer m7g3:

We appreciate your effort in reviewing our article and your valuable feedback. We would like to address your concerns as follows.

We understand that, to some extent, our work might appear simpler and thus be seen as incremental modifications. However, we must emphasize that compared to designing complex, intricate, and redundant modules and mechanisms, trying to figure out how and why audio-visual segmentation model works or fails holds greater and more valuable significance.

Our work provides a deep analysis of the long-standing failure of cross-attention in the previous AVSegFormer (CRA) and the appearance of plain attention heatmaps in the CAVP paper (Figure 11 (b) in their appendix). We reveal the attention dissipation issue present in the cross-attention fusion stage, which holds significant guiding implications for future work on audio-visual fusion. Similarly, it is precisely because we delve into the essence of the critical fusion problem that our proposed method can achieve sufficient performance improvements in a simple form. Also, it is through our essential analysis that adopting a simple approach can significantly boost the performance of audio-visual segmentation while saving substantial additional computational costs.

Dear Reviewer 6kjk:

We appreciate your effort in reviewing our article and your valuable feedback. We would like to address your concerns as follows.

1. For Q.1, this is the analysis of the research we've cited in that paragraph. Specifically, softmax attention possesses superior properties for measuring the probabilistic relationships between tokens involved in the attention mechanism. Other attention variants, while having lower computational costs, do not enjoy the same probabilistic superiority. For detailed conclusions, please refer to reference 1.

2. For Q.2, Q.4, Q.6, audio token is provided by audio backbone. It's a case where the k and v are 1-d vectors and can be found in:

   • CRA in AVSegFormer,
   • Query generator in AVSegFormer,
   • Figure 11(b) in the Ref1(CAVP),
   • The Temporal A-to-V Fusion of CATR,

   and so on. As proved by our preliminary, attention map under 1-d k and v leads to an all-1 matrix. Therefore the model could not figure out where to attend.

   For why QKV is defined in current way, to determine which sound-emitting object a visual patch represents, we need to use it as the query, with the audio serving as the key and value. By querying the keys with this query, we can identify the corresponding values, thus determining the sound-emitting object at the current position. Therefore, in the QKV setup, the visual patches should serve as the query (Q), while the audio information should serve as the key (K) and value (V).

   For the misleading alphabet, we apologize for the small errors in our paper and any discomfort they may have caused while reading. We have uploaded a revised version that has been carefully proofread to address these minor issues. We want to assure you that these errors are limited to the writing and do not affect the accuracy of our experiments or conclusions.

3. For Q.3 and Q.8, "narrow" means for a certain audio token, after undergoing the attention mechanism, it exhibits higher response and attention scores in a very small and localized region, while the rest of the area shows much lower scores. And ELF decoder is not used to solve it. This is a strategy proposed to save computational resources based on observations of current cross-attention behavior, where the original attention map has not been specifically altered.

4. For Q.5, the audio backbone is used to extract audio features from mel-spectrum of a piece of audio. Vggish is commonly used by almost all AVS methods.

5. For Q.7, as shown in Figure 9, for a given frame, certain audio queries attend to the corresponding sounding object, while others may focus on the background. Each audio query captures distinct semantic features: some attend to specific parts of the sounding object, while others capture the entire object. Across different frames, queries adapt by attending to different objects. For instance, a query might focus on the sounding object in one frame but shift attention to the background in a different context.

6. For Q.9, as described by the appendix of Ref1 and its open-sourced code, Ref1 is trained on 512x512 resolution, while most of the AVS models are trained and evaluated on 224x224 resolution. We did not include it, considering the need for a consistent training resolution. However, this work's main contribution lies in its novel training strategy, which uses a relatively common network architecture. This makes our innovation points orthogonal. We believe that better results can be achieved by combining the methods from both of our works.

7. For Q.10, from the perspective of tasks, S4 shows quite a different property from MS3. In the single source subset S4, we find that sounding objects are commonly at the saliency position. AVS models could still segment correct objects as a kind of saliency object segmentation, even without the help of audio (See Table 4. w/o QG). That is, audio-visual correspondence plays a minor role in the S4 task. While in the MS3 subset, multiple objects appear in a single image, some may be sounding while some may not, which requires better audio-visual distinguishing ability than S4. With attention dissipation, it is easier to distinguish multiple-sounding objects, which further harms the performance. As we tackle attention dissipation with PQG, the model gains the distinguishing ability and shows satisfied performance.