SAO-Instruct: Free-form Audio Editing using Natural Language Instructions

We thank the reviewer for their valuable insights and address their feedback in the following:

Question 1: CLAP score is used in many places in the paper, including training data generation, Objective Function, and evaluations. Is there a risk of overfitting a specific CLAP model here? Any mitigations?

We acknowledge the risk of over‑reliance on a single CLAP model. In our pipeline, CLAP is one metric among several: Prompt-to-Prompt candidate search also uses a perceptual Gemini filter, the Bayesian Optimization objective includes a multi-scale mel-spectrogram distance, and evaluation includes reference-based metrics in addition to a human listening study. In our experiments we used different public CLAP models and checkpoints (e.g., LAION and Microsoft), and while absolute scores were slightly different, the produced samples and rankings were overall stable. While we did not observe overfitting to the chosen CLAP checkpoint, future work could explore aggregating scores across diverse CLAP models and checkpoints to further reduce model-specific bias.

Question 2: Manual edit - is this really manualy edit or agentic edit? It's not super clear to me how manual edit was implemented. It seems each manual edit is just from a list of predefined function calls with arguments. But how are the arguments determined? Using LLM or by human?

Yes, edits are randomly sampled from a predefined set of supported edit operations (e.g., ADD, PITCH, ...). Some of these editing tasks support additional parameters as outlined in Table 5. For example, the PITCH tasks allows to change the amount of pitch shift. Unless stated otherwise, parameter values are sampled uniformly from a fixed range of supported values and the edit is deterministically applied to the audio. The LLM does not choose the operation or its arguments but is responsible for creating a fitting edit instruction for the sampled editing task and its optional parameter. This is done in multiple stages: first, the initial edit instruction is generated, and then optionally rephrased and minimized to make it more natural. In the revised manuscript, we clarified the role of the LLM to avoid confusion.

Question 3: Does the authors plan to open source the generated data?

We will open source the model and our full codebase. We are also exploring releasing the generated datasets. For the semi-synthetic DDPM inversion and the fully real manual edits we will first need to verify potential licensing issues as they contain audio from FreeSound and AudioSet-SL.

Question 4: In Table 3, are all the models being compared here trained using completely different data? Clarification would be helpful.

We thank the reviewer for raising this point. AudioEditor uses Auffusion [1] as its underlying model, fine-tuned on several general audio datasets including AudioCaps, WavCaps, ESC50, and MACS. The ZETA DDPM inversion approach uses Stable Audio Open as the base model, trained on Freesound and FMA. We have updated Section 5.2 to make these training data differences more explicit.

A study of failure cases would be helpful. Also in the demo site, adding a section for failure cases or cases where the proposed model performs worse than baseline models will complete the story.

We thank the reviewer for this suggestion. We will update the demo page with relevant failure cases after the review period ends (NeurIPS rebuttal policy).

A discussion regarding the "safety" aspects of the proposed model might be helpful. E.g. what can we do to prevent evil or inappropriate use of the model?

We discuss broader impacts in Appendix G, including that we do not scrape data from arbitrary sources and rely on well-established datasets which have been widely adopted in prior research. According with the reviewer’s feedback, our revised manuscript will expand on this section to discuss possible safeguards against inappropriate use of the model. For example, embedding inaudible watermarks in the model’s output to help detect AI-generated content, similar to existing watermarking approaches [2, 3, 4]. Additionally, for our current model checkpoints, we will consider a more restricted release of the model weights under a research license, to further reduce the risk of misuse.

[1] Xue, Jinlong, et al. "Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation." IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024).

[2] San Roman, Robin, et al. "Latent watermarking of audio generative models." ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.

[3] Wen, Yuxin, et al. "Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust." arXiv preprint arXiv:2305.20030 (2023).

[4] Guo, Yiyang, et al. "FreqMark: Invisible Image Watermarking via Frequency Based Optimization in Latent Space." Advances in Neural Information Processing Systems 37 (2024): 112237-112261.

We thank the reviewer for their valuable insights and address their feedback in the following:

The “regenerated reference” is produced solely by sampling from Stable Audio Open using the target caption, with no conditioning on the original input clip. This isn’t a true “ground-truth” edit of the input audio, but rather a brand-new, synthetic sample that may lose all source-specific content.

We agree that a regenerated reference based on the target caption is not a ground-truth edit. Evaluating the edit accuracy is challenging as paired ground-truth edited general audio datasets are unavailable. This is why we report two complementary proxies in accordance with previous work [1]: (i) scores against the original audio clips to measure quality and content preservation to guard against over-editing, and (ii) scores against this regenerated reference conditioned on the target caption to approximate edit accuracy. These metrics should be interpreted jointly to assess the model’s ability to perform accurate and faithful edits. To compare our model with other baselines, we additionally perform a human listening study to reduce reliance on these reference-based metrics.

In Section 4.2 the authors define FD/LSD/KL (and IS, CLAP) as their objective metrics, but these are always computed and reported as a single number averaged over all instructions. I want to point out that using the same set of evaluation criteria for different editing operations is not reasonable. For operations like LOOP with instructions as “repeat the audio 5 times”, we want the model to repeat the exact same audio 5 times. I would expect the edited audio to be exactly the same as the ground truth to be considered as a successful edit. However, none of the metrics in Table 2 and 3 are able to provide information like this because the performance values are averaged over different operations. While LOOP is defined as “repeat the audio l times” in Table 5 (with the constraint len(result) ≤ 47 s), there is no mechanism in the evaluation to verify how many loops were actually produced. A perfect DSP-based loop would trivially get an exact-match score, but the paper’s FD/LSD/KL cannot distinguish “five perfect repeats” from “four repeats plus a cut-off,” nor do they introduce any operation-specific metric (e.g. loop-count accuracy). I don’t think simply relying on subjective evaluation for these kinds of requirements are suitable as they are not precise enough.

We agree that for deterministic edits such as LOOP operation-specific evaluation measures (e.g., loop-count accuracy) are more appropriate then our chosen reference-based metrics. Our work focuses on free-form instruction-based editing, where edits vary widely in type of strength and are therefore difficult to categorize and evaluate. This motivated our choice of using reference proxies to that measure (i) similarity to original audio clips to measure quality and content preservations, and (ii) similarity to a regenerated sample conditioned on the target caption to approximate instruction relevance. To evaluate aspects not captured by these reference proxies, we additionally conducted a subjective listening study. We will clarify in the revised manuscript that aggregate metrics are intended to be interpreted jointly and will note the value of task-specific validators (e.g., loop-count checks) for future benchmarks.

The scores are based on a 1 to 5 rating, while only 1 and 5 are provided with a criteria. 2, 3 and 4 are undefined. Human annotators do not have a clear criteria for the ratings. Please refer to AudioBox on how to define criteria for ratings of 1 to 5.

We thank the reviewer for this suggestion. All participants in our listening study were recruited from our lab and briefed prior to the study. While they were familiar with such listening tests, we acknowledge that providing text anchors for every point on the 1-to-5 scale can improve consistency and reduce potential confusion. We will adopt this practice in future listening tests.

The design of Faithfulness in Section 4.2 (Subjective Metrics) is not reasonable and very confusing. Faithfulness is defined as “the similarity between the reference and the edited audio” in Section 4.2, while mentioned in Appendix, it is defined as “How similar does the edited audio sound to the input audio?” By simply reading through these statements, one would consider the term “reference” referring to the ground truth edited audio, while “input audio” is the original audio intended to be edited.

We thank the reviewer for pointing out this ambiguity. We will revise the explanation of the subjective metrics in Section 4.2 to use the term “input audio” instead of “reference”. For clarity, we will explicitly state that “input audio” refers to the original audio provided for editing, as described in the instructions given to the listening study participants.

Even if there is no confusion in the term “reference”, setting the criteria of faithfulness as “How similar does the edited audio sound to the input audio?” makes no sense because isn’t the goal of audio editing intended to modify the input audio into something different? Why do we need a rating of 5 indicating that the edited audio is “same as input audio”, which means that a rating of 5 indicates that the model did not do edits, while this metric is the higher the better?

Because evaluating Faithfulness or Relevance in isolation can be misleading, we interpret them together: Faithfulness quantifies how many acoustic characteristics of the original clip survive the edit, whereas Relevance measures how accurately the edit instruction is carried out. For example, consider the input “birds chirping with water flowing” and the instruction “add dogs barking."

• If the output keeps the birds and water and adds dogs, both metrics are high.
• If birds disappear but dogs are added, Relevance stays high while Faithfulness drops.
• If dogs are never added, Faithfulness may remain high yet Relevance is low.

As shown in previous work [1], jointly examining these two metrics reveals whether the model both follows the instruction and preserves the essence of the source audio.

Instructions are generated via a templated LLM pipeline and evaluated only in English. The paper does not explore robustness to paraphrases beyond its few-shot LLM variations, nor to multi-step or compound edits (for example “speed up then repeat”).

We thank the reviewer for raising this point. Because our underlying model (Stable Audio Open) was mainly trained on English data, we restricted our pipeline and evaluation to English instructions. While our work focuses on single free-form edits, we agree that compound edits are an interesting direction for future work. We will note these limitations in the revised manuscript.

In the limitations sections, the authors mentioned their approach “can naturally extend to music”. The training data is drawn from event‐focused caption datasets (AudioCaps, WavCaps, AudioSet) and manually edited effects. None of these guarantee coverage of the complex patterns found in music. For example, you are not able to simply construct a regenerated reference of the instruction “add a guitar track to this song”. The regenerated reference pipeline ignores the source audio. It doesn’t know the melody, key, or ambience of your particular song, so it will produce a new guitar-backed track rather than a faithful edit of your input. Thus, I am strongly against to the statement that their approach “can naturally extend to music”.

The regenerated reference is used only as an evaluation proxy to measure edit accuracy. During training, SAO-Instruct receives the input audio by concatenating it to the model’s input channels. During inference, the input audio is additionally encoded into the latent space, with added Gaussian noise, and used as a starting point for the denoising process. The amount of Gaussian noise can be configured and controls how similar the edited audio should be to the provided input audio.

Regarding applicability to music, the same semi-synthetic triplet generation (input audio, edit instruction, output audio) can be applied on music datasets using a music-specific generative model. Editing music using DDPM inversion was already demonstrated in [2]. To avoid overstating the claim, we have revised our statement in the updated manuscript to: “While our work focuses on editing general audio, our approach could be extended to music editing tasks provided appropriate generative music models exist and triplets are constructed from music datasets.”

Based on our feedback and proposed improvements, would the reviewer be willing to reconsider their score?

[1] Jia, Yuhang, et al. "AudioEditor: A training-free diffusion-based audio editing framework." ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.

[2] Manor, Hila, and Tomer Michaeli. "Zero-shot unsupervised and text-based audio editing using DDPM inversion." arXiv preprint arXiv:2402.10009 (2024).

The reason I insist on using different sets of evaluation criteria for different editing operations is that some editing operations are easy, ex: loop. Averaging the metric results over all kinds of editing operations does not provide a clear picture of how each kind of operation performs. The results of this work are presented as an overall score across different edits. Simple editing operations can easily contribute to the overall score and make the metric values look good. In order to raise my score, I would need to see performance values for each different kind of editing operation. This cannot be simply addressed by listing the suggested items of the reviewer as future work.

The authors admit, "for deterministic edits, such as LOOP operation-specific evaluation measures (e.g., loop-count accuracy) are more appropriate than our chosen reference-based metrics." In Appendix B, the authors also explicitly define twelve editing tasks in total. This does not align with their claim in the rebuttal, saying "this work focuses on single free-form edits". Even if we pivot this work to be able to generalize to free-form edits, the concerns can still be simply addressed by reporting evaluation scores for each editing task they define.

Additionally, three of the concerns mentioned in the review are listed as "future work" in the rebuttal. Thus, I decide to keep my score at this stage if they do not address the main concerns.

We thank the reviewer for their valuable insights and address their feedback in the following:

(1) As the main efforts of this work should be considered in data synthesis, I would recommend the authors to provide more insights into this process. Specifically, for each of the three data synthesis methods, how do you monitor the intermediate results quality, what are the statistical numbers along this way, and how is data quality control achieved and impact the overall results.

We agree that the quality of our generated data is critical for our approach. This is why we use a two-stage pipeline for the fully synthetic sample generation using Prompt-to-Prompt. Candidates (seed/CFG values) are first filtered using a perceptual Gemini check and CLAP similarity before the final sample is generated. Because intermediate statistics can be misleading across very different free-form edits (e.g., a subtle “add a light echo” vs. a structural “remove people talking”), we relied on subjective listening during development to set suitable ranges for the parameters we optimize (e.g., attention injection strength). To have a proxy for human judgement of generated samples at scale, we defined an objective function used during Bayesian Optimization in Prompt-to-Prompt and DDPM Inversion that combines several metrics. We calibrated the metric weights in a small scale listening study where listeners compared samples picked by the objective function initialized with different weightings. Finally, we selected the top-ranked weight initialization based on an ELO rating as described in Section 3.2 (Objective Function). We clarified our reliance on subjective listening for design decisions and the selection of parameter ranges in the revised manuscript.

Since the model is only fine-tuned on 50k samples, I would expect the authors to run more ablation studies to dig out what factors during the data synthesis would have the most impact on the final performance.

We report an ablation study isolating the contribution of each proposed data generation method (Prompt-to-Prompt, DDPM inversion, and manual edits) and their combination, showing complementary gains (quality and edit accuracy) when all are used together. During development we conducted internal listening tests that informed our in-method design decisions (e.g., the perceptual Gemini filter during Prompt-to-Prompt). Exhaustive ablations for each data generation method, especially for the fully or semi-synthetic data generation methods, would require a significantly higher compute budget. Therefore, our ablation study focuses on comparing the three methods and their combination.

(2) The author claims their model can follow the free-form instructions. However, when using the manual edition method alone (note this is implemented with a closed set of operations), it shows close performance with the other two methods. That's to say, under the evaluation paradigm provided by the authors, the ability to follow free-form instruction is not well justified.

We thank the reviewer for this observation. The model fine-tuned on manual edits scores well on metrics computed against the original audio indicating high content preservation and quality. However, it underperforms on metrics computed against the caption-conditioned regenerated reference, which indicates weaker edit instruction adherence. This trade-off is also evident in the MOS for the ablation study that we provide in this response. While the manual-only model can still follow some free-form edit instructions, mainly due to the Stable Audio Open pretraining and our diverse edit instruction generation, it cannot follow the same broad range of free-form edits covered by the Prompt-to-Prompt and DDPM pipelines (e.g., “make the car drive on gravel instead”).

(3) When claiming "achieving both accurate edits and faithfulness" in Table 2, do the authors want to mention the MOS scores like Table 3?

We provide the MOS for the ablation in the following table:

Fine-tuning on the Prompt-to-Prompt dataset achieves the highest edit relevance but at a slight cost in quality and faithfulness to the input audio (it can over-edit and introduce perceptual artifacts). DDPM inversion improves quality and faithfulness, but tends to follow instructions less precisely. Using only manual edits achieves the best faithfulness and quality but suffers from low edit relevance due to the closed operation set. The combined setting balances the advantages of these methods and achieves high relevance while maintaining strong quality and faithfulness to the input audio characteristics.

(4) This is optional, but is recommended: can the authors provide some generated samples for demonstration?

The manuscript on page 2 (Line 57) includes a link to our online demo page with generated samples. We will make it more prominent in the revised manuscript.

Some recent peer-reviewed works are missing from the related work. Especially, Fuggato has claimed the free-form instruction-following and editing.

We thank the reviewer for noting this. We will expand the related work section in our manuscript to include these recent works and briefly clarify how our setting and contributions differ.

Based on our feedback and proposed improvements, would the reviewer be willing to reconsider their score?

We thank the reviewer for their valuable insights and address their feedback in the following:

The reliability of the objective metrics is unclear. The authors suggest using both the original and generated audio as references; however, for complex edits, measuring KL or LSD against the original audio may not be meaningful, as the event distribution can change significantly.

We thank the reviewer for this remark. The main challenge we face during evaluation is the lack of ground-truth edited general audio pairs. Following AudioEditor [1], we use two complementary reference proxies: (i) scores against the original audio clips to measure quality and content preservation to guard against over-editing, and (ii) scores against a regenerated sample conditioned on the target caption to approximate edit accuracy. These scores reflect similarity to their respective references rather than absolute correctness and should be evaluated jointly to assess the model’s performance. As future work, datasets exploring edited general audio pairs would enable a more direct evaluation of edit accuracy.

Prompt-to-prompt sample generation uses 100 steps (line 200) and DDPM uses 70 (line 218).

We use different step counts because these methods have different goals and compute profiles. For Prompt-to-Prompt, we found that generating the final selected pair with 100 denoising steps results in high-quality synthetic samples. The DDPM inversion approach using ZETA consists of two stages: (i) the input audio is inverted to an initial latent up to T_start, (ii) this latent is conditioned on the target caption and denoised for 70 steps. As this two-stage process results in a longer inference time compared to Prompt-to-Prompt, 70 denoising steps gave a good quality-compute trade-off for ZETA. We clarified these choices in the revised manuscript.

Lines 247-250 describe the data composition for Prompt-to-prompt, DDPM, and manual edits without much discussion around the choices or sampling strategy if any.

We use a random subset of AudioSet‑SL and Freesound, plus the full AudioCaps train split. We then compute the number of audible elements per caption using the procedure in Section 3.1 (Prompt Generation) and discard any sample with >2 elements, as mentioned in Appendix A.

"We evaluate on 1k 10-second samples from the AudioCaps test subset..." — the composition of the test set is unclear. Given that manual and prompt-to-prompt edits may differ in distribution, this could influence conclusions. Would the authors consider releasing the test set?

We thank the reviewer for raising this point. We are considering releasing the prompt test set for reproducibility which contains input captions from AudioCaps. However, AudioCaps states that their dataset may only be used for academic purposes.

Minor Comments / Suggestions

In regards to the minor comments and suggestions: We agree that mel-spectrogram based vocoders have achieved great results and have revised Section 2.1 to provide a more balanced view. We also expanded Section 2.2 to mention current work in masked in-filling editing, including Fugatto.

The authors mention fine-tuning SAO on AudioCaps to improve prompt adherence for general audio. It may also be useful to report metrics such as KLD, IS, and FAD, which are commonly used in the literature.

We thank the reviewer for this suggestion. In the current manuscript, we compared SAO with its version fine-tuned on the AudioCaps training subset using CLAP similarity. We now also report FD, KL, and IS to better motivate our choice to fine-tune SAO, with all metrics evaluated on the same AudioCaps test subset for consistency. We report FD computed with PANNs features rather than FAD with VGGish, as the PANNs classifier provided more reliable results in our experiments and in prior work [2].

Lines 259–260: "During inference, we encode the input audio into the latent space of the diffusion model and add Gaussian noise..." Why is the noised latent used during inference, this seems to be different from training time? Did the authors compare this with sampling from noise directly? Adding those results will be useful.

We have added an ablation table comparing two inference modes for SAO-Instruct: sampling from pure-noise versus sampling from the Gaussian-noised latent of the encoded input audio. Metrics with * are computed against the regenerated reference, while the others use the original audio. Starting from the noised latent substantially improves metrics computed against the original audio clips, which indicates better preservation of the input audio's characteristics. It also achieves a higher CLAP score and lower FD/KL relative to the target caption-conditioned regenerated reference, showing accurate instruction following. Overall, sampling from the noised latent preserves more input audio characteristics while still providing enough flexibility to perform the required edits. We will include these results in the Appendix of the revised manuscript.

[1] Jia, Yuhang, et al. "AudioEditor: A training-free diffusion-based audio editing framework." ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.

[2] Liu, Haohe, et al. "AudioLDM: Text-to-audio generation with latent diffusion models." Proceedings of the 40th International Conference on Machine Learning. 2023.