ZeroSep: Separate Anything in Audio with Zero Training

We thank the reviewer for highlighting our mechanistic analysis, open‑set capability, and the “paradigm‑shift” use of text‑guided diffusion for separation. We clarify and address each question below.

Q1. Latent inversion approximation and mitigation

DDIM inversion is fast but introduces stepwise drift. As the reviewer mentioned, MEDIC found DDIM inversion works for very small edits, yet it accumulates reconstruction error, breaks musical structure, and offers no fine‑grained control. MEDIC tackles all three pain points with a plug‑and‑play framework that needs no extra training. Therefore, MEDIC pointed out a direction to mitigate the latent inversion error, which will also apply to our method. Apart from MEDIC, there are more correction methods that exist in vision. ZeroSep is drop‑in compatible with each:

• Null‑text (embedding) optimization: optimizes the unconditional embedding along the trajectory to match the source mixture, compensating for score error.
• PnP‑style inversions: explicitly estimate and inject residual noise to correct drift.
• DDPM‑style stochastic inversion: by recovering per‑step noise latents, it better matches the forward process and reduces accumulation compared with purely deterministic DDIM inversion.

We currently use a DDPM‑based inversion variant precisely to mitigate drift; we will expand the discussion on inversion methods and possible mitigation.

Q2. Prompt sensitivity & hierarchical/open‑set behavior

We probe 11 instruments with two perturbations per class:

• Ambiguous (hypernym / family): violin → string instrument, flute → woodwind.
• Suboptimal (underspecified / distractor adj.): trumpet → bright instrument, cello → low tone.

Results (vs. canonical prompts).

Interpretation. Hypernyms keep the correct family, yielding small degradations; the slight C‑T rise suggests the classifier judges family‑level matches as more consistent, while C‑A (accuracy to ground‑truth class) drops. Suboptimal descriptors sometimes improve distributional proximity (FAD) yet hurt class accuracy, indicating plausible but less precise extractions. We will add this table and clarify metric roles.

Q3. Novelty vs. editing‑oriented methods [1‑4]**

Key differences:

1. Task – We target source separation, not waveform replacement. Separation demands preserving mixture consistency and yielding N disjoint sources, which editing pipelines cannot guarantee (see Table 1 AudioEdit row)
2. Training‑free and optimization‑free – Unlike AudioEditor’s per‑prompt null‑text fitting, ZeroSep is runtime‑constant.
3. Separation analysis – Our paper dissects the role of inversion, guidance weight, and prompt specificity—analysis particularly related to separation task, yielding different findings compared to [1‑4].

Nonetheless, we will add [1‑4] to Related Work to include a in-depth discussion.

We sincerely thank you for the thorough, constructive review. Below we (i) add the missing citations, (ii) clarify novelty and positioning, (iii) extend the evaluation with time‑aligned metrics, and (iv) correct wording/formatting issues.; we will incorporate all changes in the revised manuscript.

### Q1 Missing Citations We thank the reviewer for providing the paper list to help us make a more comprehensive bibliography. We have added all requested references as followed:

These changes ensure the paper accurately cites prior works.

### Q2. Novelty Clarification & Positioning

We agree that our original phrasing overstated novelty. We now state:

“ZeroSep is a training‑free audio source‑separation framework that repurposes pre‑trained text‑guided diffusion models…”

Difference to GMSDI. GMSDI (Postolache et al. 2024) performs zero‑shot separation, but via posterior sampling with a Dirac likelihood. ZeroSep instead performs latent‑space inversion plus text‑conditioned denoising. While GMSDI can indeed perform zero-shot separation given appropriate textual descriptions of the input during inference, it requires train a text-conditioned diffusion model on mixtures only, which is not as versatile as our method where any off-the-shelf text-guided diffusion model can work. Also, GMSDI's effectiveness heavily depends on the quality and relevance of the textual embeddings and the accuracy of the provided text descriptions. And the open-set ability is constrained due to the mixture dataset it train on. Moreover, as no open-source implementation or pretrained checkpoints for GMSDI are currently available, we were unable to include comparative results from this method in our evaluation during the rebuttal window. However, we will include a detailed discussion with GMSDI in the revised manuscript. We summarize the difference in the table below:

Difference to AudioEditor. AudioEditor targets editing; ZeroSep targets separation. Beyond task difference, ZeroSep introduces separation-based analysis that focus on at what situations a T2A diffusion model can be repurposed to a separation model. These design principles are critical for accurate multi‑source separation; AudioEditor does not address them.

Insight on CFG Weighting. We agree with the reviewer that generation-oriented CFG trade‑offs is well taken, where lower CFG reduce conditioning strength and higher CFG may degrade result quality. However we sweep CFG $\in$ [0, 2] and find CFG=1 yields good trade-off between separation and degradation (under-separation or over-generation, see Fig. 3(a)). This specific CFG choice is not well stuided as most papers focus on scenarios where CFG > 1. For example, Fig. 8 in (Karras et al. 2024) studies CFG=4 and Table 1 in (Postolache et al. 2024) stuides CFG=(3,7.5,15,24) We clarify this distinction and will add more discussion in the revised manuscript.

### Q3 Metric Suite Expansion

Citation of Jayaram et al. 2020 Thank the reviewer for providing the reference to better motivate our finding. We will cite Jayaram et al. 2020 in the Table 3.

Implementation of FAD In our paper, we use the embedding space of CLAP model to compute FAD.

Improve Table 1 with More Metrics: We thank the reviewer for the great suggestion! We agree that additional metrics like L1/L2 loss on Encodec embedding and MSE on mel-spectrogram can capture temporal fidelity better. Below we show the improved table:

Generally, on the newly added metrics, ZeroSep matches the supervised method FlowSep and outperform LASS-net on the MUSIC dataset, confirming genuine separation rather than prompt‑consistent hallucination.

### Q5 Minor Corrections & Wording We will incorporate the following edits verbatim:

Dear authors, thank you for adressing my concerns: 1. the novelty claim 2. adding missing references 3. adding L1 / L2 metrics on embeddings. While I am a bit skeptical on the L1 / L2 losses results (why NMF-MFCC gets such a good score on this being a relatively weak method?), I decided to raise the score to Borderline Accept.

We thank the reviewer for highlighting ZeroSep’s novelty and for the constructive questions. We clarify and address each point below.

Q1. Inference cost

Let $S$ be the number of network inference steps and $T_d$ the time per step. ZeroSep performs one inversion of the mixture (shared for all separation branches) and then runs one guided denoising per target source. Thus complexity is

For single‑pass discriminative models like LASS-Net and AudioSep, the runtime is ${N\,T_d}$ which only 1 step inference is required. For FlowSep, which also based on a rectified flow model and requires multi-step inference, its runtime is $N\,S\,T_d$ for multi‑target separation. In summary, we list the table below with the runtime complexiity for each method and the actual time cost on a A100 GPU.

Note that $S$ and $T_d$ for different methods is different. In our implementation, ZeroSep’s runtime is larger than LASS-Net and Audio-Sep while keeping similar as FlowSep. In practice, $S$ can be reduced substantially with advanced samplers or distillation: DPM‑Solver++ attains high quality in 15–20 steps, and consistency models reduce to 2–8 or even 1 step, which directly scales our runtime down.

Q2/Q4. Why a null inversion prompt outperforms non‑target prompts

A non‑target prompt biases the latent inversion toward irrelevant sources, constricting the representation available for the true target during guided denoising. A null prompt leverages the diffusion prior’s rich, unbiased mixture representation, after which the target‑specific prompt can act more effectively. Table 4 shows a consistent – albeit modest – drop when conditioning on non‑targets. We will make this intuition explicit.

Q3.  $c_{\text{rev}}$ ablation wording

Thank you for point out this issue. Indeed, the CLAP‑T score on AVE improves with a random prompt. We will replace “across all metrics” with “across all metrics except CLAP‑T on AVE”

Q5. Dataset choice

We clarify that our dataset choice is reasonable:

• First, the two datasets, MUSIC and AVE, are the de‑facto multimodal (including text query) source separation benchmarks [1, 2, 3, 4]. Therefore, they are suitable for assessing the performance of unsupervised or text-guided separation methods as shown in the Tab 1.

[1] The Sound of Pixels [2] CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos [3] iQuery: Instruments As Queries for Audio-Visual Sound Separation [4] Language-Guided Audio-Visual Source Separation via Trimodal Consistency

• Second, not evaluating on AudioSet can test open‑set generalization since all supervised baselines except FlowSep were trained on AudioSet yet are evaluated out‑of‑domain here. ZeroSep’s gains therefore reflect true open domain robustness.

Q6 │ Prompt length / descriptiveness

We ran a focused prompt‑descriptiveness study on the MUSIC dataset. Specifically, We expanded each of the 11 MUSIC instrument labels into a free‑form descriptive caption (~10x length). For example: "saxophone" to "Warm, breathy reed tone with slight rasp and expressive bends,"accordion" to "Reedy, wheezy sustained chords with gentle tremolo and slow swells." This create a challenging setting to test ZeroSep where the prompt is descriptive but potentially noisy.

We observe a moderate drop of ZeroSep yet still beat the strong supervised baseline AudioSep. We agrue that the degradtation likely stems from a mismatch between our expansion and the caption used to train Text-to-Audio diffusion model, which is often a simple audio description like "a man speak while the baby is crying".

Q7. Qualitative demo (Fig. 2)

Thank you for suggesting improvement on the choice of qualitative example. We have included additional qulitative examples in the supplement, and we will correct LASSNet’s spectrogram orientation in the revised manuscript plus adding more qualitative examples in the main paper.

We thank the reviewer for acknowledging the novelty and importance of our paper, and also for providing constructive comments to help improve the paper. Below, we address each point.

Q1. Why do T2A diffusion models exhibit zero‑shot separation ability?

Data perspective. Text‑to‑audio (T2A) diffusion models are usually trained on multi‑label, polyphonic audio corpora, so the model must learn both the acoustics of individual sources and how they combine. Thus, the model learns composable latent factors that align with text semantics (e.g., dog bark, rain) and their superposition (e.g., a dog bark and raindrops). This polyphonic nature exists in many T2A pretraining datasets. For example, AudioSet -- the predominant pretraining resource -- has 2.7 labels per 10‑s clip on average[1], indicating widespread mixtures, which encourages the model to encode disentangled source structure.

[1] Audio set: An ontology and human-labeled dataset for audio events

Score-function view. Classifier‑free guidance (CFG) combines an unconditional score $\nabla_{x_t}\log p(x_t)$ with a conditional increment that steers toward $\nabla_{x_t}\log p(c\mid x_t)$:

which is a standard derivation in CFG. Setting $\omega=1$ removes the unconditional term and retains only the conditional direction, effectively filtering the inverted latent toward the target source specified by text. Therefore, the zero-shot separation capability is an emerging property of T2A diffusion models trained with mixture audio. We will add this discussion to the revised manuscript.

Q2. Effect of $\omega$ and scheduled guidance

Large $\omega$ amplifies the conditional score. Because both scores are learned approximations, excessive scaling can move samples toward low‑probability regions (“off the data manifold”), improving prompt adherence and diversity of generated samples but risking artifacts -- an effect documented broadly in image/audio diffusion [2, 3].Therefore, it will opt for generation instead of accurate separation with large $\omega$. This potential artifact, along with large $\omega$, does not contribute to accurate separation, and empirically we show $\omega=1$ achieves a good trade-off between removing the other sounds and introducing generated content/artifact.

Static vs. scheduled guidance. Thank you for the great suggestion! Intuitively, the early stage of diffusion models builds the global structure, while the later denoising steps act as a refinement stage. We tested constant and scheduled $\omega$ on AudioLDM2‑Large backbone (DDIM, 50 steps). Results on the MUSIC dataset:

Observations. (i) Low-to-high, Starting at 0 (0→1, sine) harms separation (global structure lost of clean target audio). (ii) High‑to‑low (1→0) improves over constant 1.0, aligning with findings that guidance is most useful in the early to middle noise range and less beneficial at the end. This further enhances our claim that $\omega=1$ is beneficial for accurate separation, while dynamic scheduling based on this finding can also be promising in future exploration, which we will add discussion in the revision.

[2]: CLASSIFIER-FREE DIFFUSION GUIDANCE. NeurIPS 2021 Workshop

[3]: ELIMINATING OVERSATURATION AND ARTIFACTS OF HIGH GUIDANCE SCALES IN DIFFUSION MODELS. ICLR 2025.

Q3. Runtime and scalability for multiple sources

Let $S$ be the number of network inference steps and $T_d$ the time per step. ZeroSep performs one inversion of the mixture (shared for all separation branches) and then runs one guided denoising per target source. Thus complexity is

For single‑pass discriminative models like LASS-Net and AudioSep, the runtime is ${N\,T_d}$ where only 1 step inference is required. For FlowSep, which is also based on a rectified flow model and requires multi-step inference, its runtime is $N\,S\,T_d$ for multi‑target separation. In summary, we list the table below with the runtime complexity for each method and the actual time cost on an A100 GPU.

Note that $S$ and $T_d$ for different methods are different. In our implementation, ZeroSep’s runtime is larger than LASS-Net and Audio-Sep while remaining similar as FlowSep. In practice, $S$ can be reduced substantially with advanced samplers or distillation: DPM‑Solver++ attains high quality in 15–20 steps, and consistency models reduce to 2–8 or even 1 step, which directly scales our runtime down.

Q4. Latent inversion approximation and mitigation

DDIM inversion is fast but introduces stepwise drift. As the reviewer mentioned, MEDIC found DDIM inversion works for very small edits, yet it accumulates reconstruction error, breaks musical structure, and offers no fine‑grained control. MEDIC tackles all three pain points with a plug‑and‑play framework that needs no extra training. Therefore, MEDIC pointed out a direction to mitigate the latent inversion error, which will also apply to our method. Apart from MEDIC, there are more correction methods that exist in vision. ZeroSep is drop‑in compatible with each:

• Null‑text (embedding) optimization: optimizes the unconditional embedding along the trajectory to match the source mixture, compensating for score error.
• PnP‑style inversions: explicitly estimate and inject residual noise to correct drift.
• DDPM‑style stochastic inversion: By recovering per‑step noise latents, it better matches the forward process and reduces accumulation compared with purely deterministic DDIM inversion.

We currently use a DDPM‑based inversion variant precisely to mitigate drift; we will expand the discussion on inversion methods and possible mitigation.

Q5. Polyphonic mixtures

ZeroSep supports multi‑source separation naturally. We additionally include results on 3-source mixtures built from MUSIC dataset (e.g., violin+flute+trumpet), which shows moderate degradation as the number of mixing sources grows.

For the challenging cases the reviewer mentions, where sources from the same event class are mixed, we argue that adding attribute qualifiers in text (e.g., “soft violin arpeggio” vs. “long sustained violin note”) could help. We will include the new 3-source separation results and add the discussion on challenging instance-level separation in the revised manuscript.

Q6. Prompt sensitivity & hierarchical/open‑set behavior

To test the prompt sensitivity and hierarchical prompting performance, given the class labels from the MUSIC dataset, we probe 11 instrument class names with two perturbations per class:

• hypernyms:** violin → string instrument, flute → woodwind.
• underspecified descriptors:** trumpet → bright instrument, cello → low tone.

Results (vs. canonical prompts).

Observation. Hypernyms keep the correct family, yielding small degradations; the slight C‑T rise suggests the classifier judges family‑level matches as more consistent, while C‑A (accuracy to ground‑truth class) drops. Underspecified descriptors sometimes improve distributional proximity (FAD) yet hurt class accuracy, indicating plausible but less precise extractions. We will add this table and clarify metric roles.

Open‑set note. These results indicate good generalization from parent nodes (e.g., string instrument) to leaves (e.g., a violin articulation), but we avoid over‑claiming “strong open‑set separation.” We will frame it as hierarchical robustness and include success/failure cases in the revised manuscript.