Segment, Associate, and Classify: Decoupled Audio-Visual Segmentation Framework

1. I have concerns about the novelty of the proposed method. In particular, the 'Segment' stage uses existing image segmentation models to obtain visual masks. Notably, this has also been employed in prior works, for example, Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024) ; The 'Classify' stage simply uses existing CLIP model to decide sound source categories; The proposed MSA-MICL is similar to existing EZ-VSL method.
2. One of the main contributions of this paper is the unsupervised contrastive learning. However, it seems that the authors utilize 144k videos from VGGSound for model training. Since the AVSBench datasets are collected using techniques similar to those for VGGSound, there may be a risk of testing data leakage. Moreover, the introduction of MSA-MICL contrastive loss is unclear; this loss will be influenced by the construction of synthetic data, about which details and discussions are not provided.
3. More questions will be given in the next part.

While the paper achieves impressive performance, it essentially stacks pre-existing modules (e.g., pretrained segmentation and vision-language models) in a decoupled framework, with limited architectural innovation, It feels more like a pipeline.

The figs focus on single-frame segmentation without evaluating time-based alignment in dynamic scenes. This limitation misses a critical aspect of audio-visual synchronization. Continuous multi-frame results or temporal metrics would help verify the framework's ability to manage complex time-dependent audio-visual correlations.

The model implicitly assumes every audio segment is linked to a visible object. In real-world applications, background music or unrelated sounds are common, which may lead to incorrect associations. Implementing a “null correspondence” mechanism or similar approach could help address this limitation.

1. Accumulation error in matching: Could you please provide more examples and statistics related to matching? Since the audio label and visual label may not always align perfectly. In AVS-bench, the segmentation labels can be quite ambiguous, such as "man" and "boy," "car" and "ambulance." Have any analyses been conducted on this issue?
2. Accumulation error in detection: The class-agnostic object detector often detects unwanted objects and assigns incorrect classes. Is there any further analysis on this matter?
3. Further analysis on mask-wise audio similarity: Let's consider the image in Figure 1 as an example. The man in the image can produce not only speech but also sounds like clapping and whistling. How can mask-wise audio similarity help address this issue? In other words, sounds like clapping and whistling can be emitted by multiple different objects, like men and women.
4. Dataset: In the paper, the authors claim that they divide AVSBench into two 5-second clips. Is it fair for other models that take 10s input? Why 5 seconds?
5. Fair comparison: Can authors provide the comparison of parameters and FLOPS?

– The paper relies primarily on pre-trained models - object detection, segmentation (SAM), CLIP, VGGish. While a workable system for sounding object segmentation can be created using these strong models, it does appear less interesting and as more of an assembled system with existing models. Moreover, with such an approach the labeling effort has been shifted from essentially labeling sounding objects to the ones used by these models (not such a bad thing, just less interesting).

– The global local embeddings are essentially obtained through a linear combination of embeddings the detected object and rest of the image. It’s not very intuitive if we are simply combining them through linear combination, then why would a straightforward CLIP embedding of the whole image not capture all of the same information ? Perhaps some experiment where a CLIP embedding of the whole image replacing f_n, with everything else the same, might shed some light.

– The approach seems to work primarily on a closed set, where the mast classification label set is known (and fixed) a-priori ? That might be restrictive given how reliant this approach is to large pre-trained models, where the model should be able to handle open-set.

– The performance seems to increase with the number of Input Masks, all the way to up to 50 masks. Given the experimental settings mostly has very few (1, 2 or so) sounding objects, this is a bit surprising that even though such large number of sounding objects are used, the performance does not plateau or deteriorate. Would be good to discuss this.

– The details of the MSA-MICL loss – why it’s needed, what’s the intuition and how it’s done etc., could be improved. It’s a bit hard to follow.

– For multi-source case in Fig 4, why does lambda > 0.6 leads to such a massive drop in performance, much more compared to single-source.

– Table 4, is not very informative, especially given that the training data scale is too narrow – from 50k to ~150k. More varied scale, say an order or more change in training data could actually be a bit informative. Otherwise, hard to see what we are trying to infer.

– Some more insights might be helpful. Example, what happens when there are multiple sounds, with one or more of them not in field of view. Is the model able to localize only the visible sounding object? Does it produce more false positives? Or Two objects in the scene which can produce the same sound.