Rethinking Audiovisual Segmentation with Semantic Quantization and Decomposition

1. The details of the paper need to be re-checked, as shown in Figure 1 (a). It seems that the audio and video of the guitar and singing do not correspond correctly.
2. For ease of reading, some important conceptual terms need to be explained nearby, such as local ASD and global ASD in Figure 3.
3. The audio semantic decomposition via product quantization proposed by the author is mainly aimed at the scene of multi-source audio, so SQD has limited improvements over the AVS-Object dataset.

The paper did not cite all the papers published earlier this year (i.e., [1,2]). I would suggest that the author include these related works and, if necessary, make comparisons.

I believe that the core of the paper is quite similar to the prototype-learning method (i.e., [3,4]). In this method, prototypes are learned for each class during training, and during testing, similar metrics are used to determine the highest similarity of the input feature with respect to the prototypes. However, the paper lacks a discussion of the use of such a method in the literature review, and the novelty of the PQ-based multi-source semantic decomposition is somewhat overstated. Additionally, instead of the 'decomposition' of sound sources into a class-specific manner, the proposed method attempts to use these prototypes in a class-agnostic manner, which might hinder the visualization and understanding of the learned prototypes.

I am not sure about the "|" notation in Eq.6, does this equation mean we have two inputs for the network?

In Tab.1 of the experimental section, it is good to see that the author compares different backbones for the evaluation. However, it is unclear to me why the author decided to use Swin-Tiny and V-Swin-Tiny to compare with the PVT-v2-Base baseline TPAVI (Zhou et al., 2023). It looks like an unfair comparison in this case. Additionally, it looks strange to me that the proposed method only improved by 0.8% under AVS-Object-Single (Jaccard) but improved by 23.6% on the AVS-Semantic dataset (mIoU).

The proposed framework exhibits certain similarities with ReferFormer [5]. However, the paper does not include the results of ReferFormer, which could be placed in either Tab.1 or Tab.2.

Minors In Figure 1 the Guitar and Singing are misplaced from each other.

Reference [1] Li, K., Yang, Z., Chen, L., Yang, Y. and Xun, J., 2023. Catr: Combinatorial-dependence audio-queried transformer for audio-visual video segmentation. arXiv preprint arXiv:2309.09709. [2] Chen, Y., Liu, Y., Wang, H., Liu, F., Wang, C. and Carneiro, G., 2023. A Closer Look at Audio-Visual Semantic Segmentation. arXiv e-prints, pp.arXiv-2304. [3] Snell, J., Swersky, K. and Zemel, R., 2017. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30. [4] Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C. and Su, J.K., 2019. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32. [5] Wu, J., Jiang, Y., Sun, P., Yuan, Z. and Luo, P., 2022. Language as queries for referring video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4974-4984).

• The proposed method is just a pipeline of existing methods (transformer decoder, VQ, and feature modulation). Vector quantization for cross-model learning has been proposed in [1] and global-local audio-visual representations have been explored in [2]. Also, the proposed method is relying on pre-extracted features (if I understand correctly, the visual and audio feature backbones are not trained).
• The contribution of this work is narrow - a specialized architecture is proposed only for audio-visual semantic segmentation. I think this paper would be stronger if the method was evaluated on a large-scale dataset like AudioSet and VGGSound (similar to the evaluation in [2]) or if the method was evaluated on different modalities like [1] does (ie. image-text or speech-image).
• The paper claims an improvement over SOTA of 25.4 -> 46.6 mIoU for the AVS-Semantic task, while the authors' baseline model in Table 2 without the proposed modules already achieves 33.5 mIoU, so the impact is less than stated. Also, this doesn't report results for the F-Score on the AVS-Semantic task which was reported in prior results.
• For the AVS-Object task, the proposed method mainly improves on the mIoU (region similarity) and the improvement on contour accuracy F is less significant.
• The writing is unclear in several sections and there are distracting typos ("Sementic"). I did not understand part of the methods section. Figure 1 seems incorrect - the person is highlighted for "guitar" and the guitar is highlighted for "singing."

[1] Alexander Liu, SouYoung Jin, Cheng-I Lai, Andrew Rouditchenko, Aude Oliva, and James Glass. 2022. Cross-Modal Discrete Representation Learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3013–3035, Dublin, Ireland.

[2] Zeng, Zhaoyang, Daniel McDuff, and Yale Song. "Contrastive learning of global and local video representations." Advances in Neural Information Processing Systems 34 (2021): 7025-7040.