Sounding that Object: Interactive Object-Aware Image to Audio Generation

We thank the reviewer for their comments and time.

Clarifying AVC.

In fact, we employed AVC to measure the semantic (instead of temporal) correspondence between audio and image, since our primary goal is to generate audio aligned with objects from image (not video).

Why images only?

Our paper’s main contribution is object-aware, user-driven audio generation based on static images. Focusing on images allows us to cleanly isolate object-to-audio relationships and provide more intuitive user control. While incorporating temporal dynamics (in videos) is a natural extension, it entails additional complexities such as motion tracking and scene changes, which lie beyond our paper’s current scope. However, we agree that expanding to video is an important next step.

Additional Synchformer-based metric.

We note that Synchformer specifically targets audio-visual synchronization for video-to-audio generation and does not directly apply to the image-to-audio task. However, inspired by Synchformer’s contrastive pre-training (like DeSync in MMAudio and AV-Sync in MultiFoley), we employ ImageBind [Girdhar et al., CVPR 2023] to measure audio-visual matching on static images. By extracting features from both modalities and computing cosine similarity, we show in the table below that our method consistently outperforms all baselines on this metric.

Comparison to SAM 2.

As suggested, we replace SAM with SAM 2 and evaluate our method on the test set. We show in the table below that this substitution leads to further gains in generation accuracy and quality, which confirms that more precise segmentation masks benefit our method and aligns well with Theorem 3.1 of our paper.

Balancing the volume of different objects.

We demonstrate in Figure 5 and the demo video of our paper that specifying each object separately tends to assign a similar volume to all sources. However, when multiple objects are selected, our method dynamically accounts for context. For example, if a large car dominates the scene, its siren may overwhelm subtle ambient sounds, creating a more realistic blend instead of flattening everything to equal volume. Moreover, we quantitatively confirm this context-driven behavior in Table 1, 7, and 8 of our paper, where our object-aware method better reflects how certain sources can overpower others or combine to create natural audio events.

Generating off-screen sound.

Our current method grounds audio into visible objects via segmentation masks. To incorporate off-screen events, we will add another textual cue (like background music) as extra conditioning for content that lacks a corresponding visual region. This requires no changes to segmentation since off-screen sources are not maskable. While off-screen sound lies outside our paper’s main scope, we agree it is a valuable direction for user-driven, interactive audio generation.

Missing references.

Thanks for pointing these out. We will include the suggested references in the revised version. We also clarify that, as noted on the right side of Line 73-95 of our paper, our method differs by enabling object-aware audio generation from static images rather than full video or text-centric approaches. We ground audio in user-selected objects, thus offering fine-grained controllability compared to broader scene-level or textual conditioning alone.

We thank the reviewer for their comments and time.

Generating audio from multiple objects.

In fact, we showed in Figure 5 and the demo video of our paper that our method accepts multi-object masks (including more than two objects) to generate an audio mixture that reflects each selected object. We also evaluated a multi-object subset in human perceptual study (the last four columns of Table 1 in our paper), further confirming that our model can handle multiple sources in a single scene.

Interactions among multiple objects.

We illustrated in Figure 6 and the demo video of our paper that a single image showing a stick in water or grass can generate splashing or rustling by leveraging full-image segmentation masks. These results indicate our method’s ability to handle basic multi-object interactions from static images. However, we also note that more intricate physical interactions often rely on temporal cues, which are generally studied in video and lie beyond our current single-image scope.

Linear vs. MLP.

Our method used a lightweight MLP to capture nonlinear cross-modal interactions more effectively than a simpler linear layer. As shown in the first table below, retraining with a linear projection yields lower performance. Importantly, our MLP added roughly 1% more parameters to the AudioLDM backbone (see the second table below), resulting in a minimal additional computation cost.

Comparison to CoDi.

As suggested, we evaluate CoDi (we do not compare CoDi 2 since it is not open-source) on our test set in the table below. We find that our model still holds better performance across all metrics. Unlike CoDi’s global any-to-any generation, our model explicitly allows users to select specific objects within an image for fine-grained control in audio generation.

Interaction satisfaction evaluation.

As suggested, we conduct a human study focusing on user-driven audio generation, comparing our method to text-based baselines (we exclude image- and video-based baselines, as they do not allow user prompting). We ask 5 experienced participants to generate "baby laughs and puppy barks" from a single image (Figure 2 of our paper), and we measure the average time taken (in minutes), the number of attempts required, and a 5-point subjective satisfaction score (with 95% confidence intervals). As shown in the table below, text-based baselines often miss one of the sounds and require multiple prompt adjustments, leading to higher time and lower satisfaction. Our method, by contrast, consistently requires fewer attempts, takes less time, and achieves higher satisfaction—even for participants already familiar with prompting. We will include these findings in the revised version.

Missing references.

Thanks for pointing these out. We will include CoDi and CoDi 2 in the revised version. We also clarify that, as noted on the right side of Line 73-95 of our paper, our method differs by enabling object-aware audio generation in static images rather than full video or text-centric approaches. We ground audio in user-selected objects, thus offering fine-grained controllability compared to CoDi’s global any-to-any generation.

We thank the reviewer for their comments and time.

Clarifying caption-based methods.

In fact, we have evaluated two caption-based variants in Table 1 & 7 of our paper. In Captioning, we generated a single caption from the entire image and fed it to a pre-trained AudioLDM 2. In Captioning & Mix, we generated separate captions for each detected object, synthesized individual audio clips with AudioLDM 2, and mixed them. As expected, both methods perform worse than ours, but interestingly Captioning outperforms Captioning & Mix, suggesting that caption-based methods do not fully address context or proportional mixing—an observation also noted on the right side of Line 45-48 in our paper.

Region-based audio correlation.

We demonstrate in Figure 5 and the demo video of our paper that naive region-based layering tends to assign similar loudness to all sources, leading to flat, unrealistic soundscapes. In contrast, our method dynamically accounts for context. For example, if a large car dominates the scene, its siren may overwhelm subtle ambient sounds, creating a more realistic blend instead of flattening everything to equal volume. Likewise, Figure  6 of our paper shows that our method captures interactions—like a stick splashing water—instead of generating only generic water flowing sounds. Moreover, we quantitatively confirm this context-driven behavior on AudioSet and VGGSound Sync in Table 1 & 8 of our paper, where our object-aware method better reflects how certain sources can overpower others or combine to create natural audio events.

Clarifying SSV2A.

SSV2A is an unpublished concurrent work and the code has not been fully released before the submission deadline, so we did not compare with it, per ICML policy. In fact, our method is quite different from SSV2A: (1) our training is self-supervised, while they require bounding boxes from external object detectors such as YOLOv8; (2) we allow fine-grained user control on objects, while their manifold pipeline lacks this capability. As suggested, we now evaluate SSV2A on our test set, showing that we generate more accurate and comparable high-quality audio:

Additional dataset evaluation.

The ImageHear dataset contains only a single object per image, which does not align well with our object-aware setting so we did not evaluate on it. However, as suggested, we now compare our method to both image- and video-based baselines using metrics from ImageHear. As shown in the table below, our method continues to outperform them:

Missing references.

Thanks for pointing these out. We will include these references in the revised version. We also clarify that, as noted on the right side of Line 73-95 of our paper, our method centers on user-selected object-aware audio generation in static images, allowing finer control than the concurrent work SSV2A’s bounding-box method (as discussed above) and the broader scene-level approaches like V2A-Mapper or Seeing & Hearing.

We thank the reviewer for their comments and time.

Reliance on SAM.

First, our method does not fundamentally rely on SAM for its performance, but instead benefits from it functionally for enhancing user interactivity. As shown in Table 2(iv), Theorem 3.1, and Figure 4 of our paper, comparable results can be obtained when SAM is replaced with text-image cross-modal attention at test time. Second, we respectfully disagree with the notion that leveraging strong pre-trained models weakens the contribution. Our use of SAM follows a common and productive paradigm in the community to build upon powerful existing components—examples include LLaVA (LLaMA + visual encoder), ControlNet (Stable Diffusion + extra conditioning), and Prompt-to-Prompt (Stable Diffusion + cross-attention control).

Limited novelty.

Our core novelty does not lie in proposing a new network architecture or method purely for state-of-the-art performance. Instead, our contributions are: (1) identifying and posing a surprising self-supervised learning signal—distinct objects and their associated sound—for audio generation, which is in the tradition of ICML papers like SimCLR [Chen et al., ICML 2020] that opened up promising research directions; (2) fundamentally overcoming the challenge of forgetting or binding sound events—by introducing multi-modal dot-product attention for audio generation, grounded in theory; (3) enabling a new interactive setup, where users can control which objects produce sound via simple mouse clicks.

Clarifying SSV2A.

SSV2A is an unpublished concurrent work and the code has not been fully released before the submission deadline, so we did not compare with it, per ICML policy. In fact, our method is quite different from SSV2A: (1) our training is self-supervised, while they require bounding boxes from external object detectors such as YOLOv8; (2) we allow fine-grained user control on objects, while their manifold pipeline lacks this capability. As suggested, we now evaluate SSV2A on our test set (the table below), showing that we generate more accurate and comparable high-quality audio.

Comparison to additional methods.

As suggested, we compare several video (or video + text) based methods (Seeing & Hearing, ReWaS, and FoleyCrafter), noting that ReWaS is concurrent, FoleyCrafter is unpublished, and SonicVisionLM has no public codebase so we do not compare to it. Since these works do not provide training codes, we instead run their public inference scripts on our test set. The table below shows that our method outperforms these baselines, largely because it introduces object-level specificity.

Constrained dataset evaluation.

We have provided quantitative evaluations on the commonly used AudioSet and VGGSound Sync (Tables 1 & 8 of our paper), which feature in-the-wild videos, and we include out-of-distribution examples (Section 4.4 of our paper) plus a demo video (Appendix A), using inputs from the internet or different datasets (Places & the Greatest Hits). These studies, conducted across diverse datasets at scale, demonstrate our model’s generalization capability.

Extending to video.

Thanks for pointing out this important future direction and broader impact. Our paper’s main contribution is object-aware, user-driven audio generation based on static images. Images allow us to cleanly isolate object-to-audio relationships and provide more intuitive user control, while temporal dynamics (in videos) entail additional complexities such as motion tracking and scene changes, which lie beyond our paper’s current scope.

Comparison to frozen-diffusion.

Neither SSV2A nor ReWaS reports experiments contrasting frozen vs. fine-tuned diffusion, so they do not quantify how full adaptation might improve cross-modal alignment. The ablation in Table 2(i) of our paper reveals that freezing simplifies training but hinders fine-grained alignment (especially at the object level), leading to higher FAD. By fully fine-tuning, we reduced FAD and improved ACC because the model can capture richer object-specific cues.

Clarifying video-based baselines.

As described on the right side of Line 288-289 of our paper, we randomly sampled a single frame from each video clip as input and fine-tuned them on our dataset.

Missing references.

Thanks for pointing these out. We will add comparisons (see above) and discussions (the right side of Line 73–95 of our paper) on the suggested papers in the revised version.