FLAM: Frame-Wise Language-Audio Modeling

We sincerely thank Reviewer Rbvm for their thoughtful and constructive review. We are especially grateful for your recognition of our theoretical and mathematical contributions. As you noted, “the theoretical claims around bias correction… [are] broadly correct,” and your review confirmed that you “reviewed the appendix and broadly found the theoretical claims to be correct.”

We also appreciate your positive assessment of our methodological choices—“the method is simple and elegant”—as well as your acknowledgment that our bias correction and synthetic dataset construction are “well-justified.” Your remark that “the focus on bias correction and robustness is admirable and should be standard in the contrastive literature” particularly resonates with our goals. We are hopeful that our logit-adjusted contrastive loss formulation will inspire future work in other domains such as vision and language.

Below, we address your primary concern and introduce a new alignment-specific evaluation metric, as requested.

---

The improvement in alignment of the audio events, the whole point of doing frame level SED, is only shown through examples, and not robustly checked and justified. This is the paper's major weakness, because, without summary statistics and robust justification, it is not clear if the models improvements in accuracy come from better alignment or simply better classification accuracy. While it is probably that the improvements come from better alignment, I'd like to see this measured better than via examples and diagrams.
Could the authors come up with a metric that focuses purely on alignment and compare the various models on it?

We fully agree with the reviewer that robust evaluation of alignment quality is essential to justifying the benefits of frame-level supervision. In our original submission, we report frame-level AUROC and PSDS, both of which reflect the model’s ability to distinguish when an event occurs, given a caption. AUROC is computed over all frame-caption pairs, including negative ones, and thus directly reflects alignment fidelity. PSDS (Polyphonic Sound Detection Score) is a DCASE challenge metric that integrates precision and recall across thresholds with a focus on time-sensitive detection.

In Table 1 of the main paper, we showed that FLAM significantly improves both AUROC and PSDS across open-vocabulary and closed-set datasets, which we argued reflects improved temporal alignment. However, we agree that these classification-based metrics only measure alignment indirectly.

To address this, we propose a new diagnostic metric specifically designed to directly evaluate alignment: Spearman correlation between the model’s frame-text similarity scores and the ground truth label mask. This measures how well the model’s similarity predictions track the actual occurrence of audio events over time, independent of decision thresholds or absolute similarity magnitude.

Definition: For each captioned event, we compute the Spearman rank correlation between:

• the model’s similarity scores across frames (either pre-sigmoid logits or dot products), and
• the corresponding binary ground-truth label mask indicating event presence over time.

A higher Spearman correlation indicates that the model more faithfully aligns audio with text on a frame level.

---

Results: Alignment Correlation (Spearman ρ)

Note: for all Spearman ρ, the p-value is smaller than 0.01.

---

These results confirm that FLAM achieves substantially higher alignment correlation than both MGA-CLAP and FLAM-Global on both ASFX-SED and synthetic datasets. This supports our hypothesis that FLAM’s improvements stem from better temporal alignment, not merely improved classification.

We will consider including this new metric, its formulation, and the results above in the revised manuscript. Thank you again for your insightful suggestion—it has helped strengthen the empirical justification for our frame-level approach.

We sincerely thank Reviewer GGdQ for their thoughtful and constructive feedback, and for recognizing several key strengths and contributions of our work. We are encouraged by your positive assessment of our model and contributions—for instance, your remark that "FLAM mostly outperforms baselines in some audio-text tasks", your recognition that FLAM is the "first SED system trained with sigmoid loss in a contrastive learning setting", and your appreciation of our "proprietary high-quality sound dataset". We also value your comment that "the proposed architecture and training scheme make sense", and your confirmation that "the datasets and metrics used in the experiments are standard in this field". We found your comments particularly insightful and have revised the manuscript to incorporate your suggestions, which we address point-by-point below.

---

It would be insightful to see a comparison in closed-set SED tasks between other SED methods not designed for open vocabulary SED.

We appreciate your suggestion to compare FLAM against strong closed-set SED models. In response, we conducted additional evaluations of FLAM on closed-set SED benchmarks alongside state-of-the-art closed-set systems. Due to space constraints, we summarize these results here: https://flam-model.github.io/response_reviewer_ggdq.html

While FLAM performs competitively, especially considering its generalization capabilities, there remains a performance gap compared to highly specialized closed-set models. This result is expected, as closed-set SED methods are explicitly trained on a fixed, predefined label set and can thus heavily optimize for those specific categories. However, this design inherently limits their applicability to new or unseen sound events, since they cannot recognize or adapt to events outside their training vocabulary. In contrast, FLAM supports open-vocabulary detection by leveraging natural language descriptions, enabling broader generalization and better alignment with real-world, evolving sound taxonomies.

From a downstream application perspective, this flexibility makes FLAM more broadly applicable, particularly in dynamic or user-driven scenarios where defining a fixed vocabulary in advance is infeasible. We believe these findings motivate further research into bridging the gap between open and closed-set SED and enhancing generalization across domains.

---

Ablation studies to confirm the strength of sigmoid loss, its low computational cost and its insensitivity to batch size, should be performed under the SED scenario

Thank you for encouraging a deeper investigation into the role of sigmoid loss. We conducted the following ablation studies to evaluate its effectiveness under the SED setting, with results in https://flam-model.github.io/response_reviewer_ggdq.htm

• We trained a version of FLAM without sigmoid loss (please see the exact object in link). This version fails to converge and results in substantially degraded SED performance, highlighting the importance of the sigmoid loss in stabilizing training and enabling convergence in the frame-level contrastive setup.

• We trained FLAM using two smaller batch sizes: 256 and 128 (original batch size = 512). As shown in our results, FLAM remains robust across batch sizes, with only marginal drops in performance at smaller sizes. We attribute this to fewer negative examples being sampled during training.

These results support our claim that sigmoid loss is both effective and computationally efficient for frame-level SED. While it is not entirely insensitive to batch size, its formulation allows us to scale to larger batches under the same memory constraints—unlike InfoNCE-based losses, which require global softmax computation and scale poorly with batch size. This makes sigmoid loss particularly well-suited for the resource-intensive frame-level SED setting.

---

Typos:
Thank you and Reviewer bqpP for catching the remaining typos! We've corrected the following:

• Sec. 3.3
  Original: "we allocate up t slots perup to five times the audio in a batchd pad any unused sloaudio or text ts with placeholders"
  Fixed (Lines 273–274, Page 5, left column):
  “Since each audio clip may contain a varying number of events, we allocate text slots equal to five times the number of audio clips in a batch, padding any unused audio or text entries with placeholders.”

• Sec. 5
  Original: "How does FLAM fare on downstream tasks typically used to evaluate contrastive ALMs?"
  Fixed: "How does FLAM perform on downstream tasks typically used to evaluate contrastive ALMs?"

• Sec. 5.2
  Original: "ustic"
  Fixed (Line 360, Page 7, left column): "Acoustic events"

• Table 2 (A2T experiment on Clotho):
  We fixed the bolding of the correct model.

• Line 244 (Page 5, left column):
  We removed the duplicated phrase: “the loss $\mathcal{L}_p =$”

We thank reviewer d9oS for reviewing our manuscript and recognizing the novelty of our frame-wise contrastive learning, data augmentation, and FLAM’s strong open-vocabulary SED performance. We respectfully clarify and address the concerns you raised. Notably, these concerns were not shared by the other reviewers. For example:

• Reviewer Rbvm commented that “the experiments are fair, with no clear advantages given to the proposed method in comparison to the baseline,” and further stated that “the supplementary material... [is] complete and satisfactory.”

• Reviewer GGdQ noted that “FLAM mostly outperforms baselines in some audio-text tasks” and that “the datasets and metrics used in the experiments are standard in this field.”

• Reviewer bqpP emphasized that “FLAM builds on previous audio-language models by introducing a frame-level representation, which (as the paper proves) is really important for event detection,” and remarked that “the theoretical analysis and the results... both support [the main claims].”

We hope this provides context for why we believe the raised concerns do not substantially weaken the validity of our contributions. We now respond point-by-point:

The paper lacks supplementary material, which could provide additional details on experiments, datasets, and implementation. The paper does not present any formal theoretical proof.

We’d like to clarify that our submission does include detailed supplementary material from page 11 to page 20, covering:

• A full derivation of the FLAM frame-wise contrastive objective
• Sound event detection results on open-vocabulary tasks
• Detailed descriptions of our training procedure, dataset construction, and architectural hyperparameters

In addition, we provide a demo website at https://flam-model.github.io/, which showcases FLAM's localization results on unseen real-world samples.

Both Reviewer Rbvm and Reviewer GGdQ explicitly confirmed they reviewed the supplementary materials and found them complete and satisfactory. We invite Reviewer d9oS to revisit this part of the submission.

---

Regarding baseline comparison:

The baseline comparisons seem weak, as the primary baseline (MGA-CLAP) is re-trained on the authors' dataset, potentially limiting its effectiveness.
Why did you choose the specific baselines for comparison, and how do you justify the selection given the availability of other state-of-the-art models in the field?

We chose MGA-CLAP as the primary comparison for open-vocabulary SED because it is the most recent and relevant baseline in this domain. Specifically:

• MGA-CLAP was accepted as an oral paper at ACM Multimedia 2024 (top 3.97% of submissions), indicating strong peer recognition and technical merit.
• Its training objective and model design also aim at audio-text alignment, making it a natural benchmark for FLAM.
• Since our training set comprises licensed proprietary sound effects, we retrained MGA-CLAP on the same dataset to ensure a fair comparison. This avoids domain shift effects that would unfairly disadvantage the baseline.

To help readers interpret both in-domain and cross-domain generalization, we also include the original MGA-CLAP performance on public datasets. Additionally, on retrieval and classification tasks, we compare to other baselines like CompA and LAION-CLAP, where FLAM remains competitive.

As Reviewer Rbvm stated: “The experiments are fair, with no clear advantages given to the proposed method.” Reviewer GGdQ similarly noted that “FLAM mostly outperforms baselines in some audio-text tasks,” reinforcing that our baseline selection and evaluation strategy are reasonable.

---

The paper could benefit from a clearer discussion of the limitations and potential future work.

Thank you for this helpful suggestion. We added a paragraph at the end of Section 7 discussing limitations and future work:

“FLAM represents an initial step toward large-scale open-vocabulary sound event detection, but several aspects remain to be improved. The current training corpus, while diverse, is still limited in scale; future work could explore larger and more diverse corpora, potentially by synthesizing additional labeled mixtures or leveraging web-scale audio. The lightweight model could benefit from scaling or more expressive architectures. Additionally, FLAM uses a fixed 10-second audio input and a coarse frame resolution, which constrains its ability to handle longer or more temporally nuanced recordings. Future efforts could focus on supporting variable-length audio and adopting encoders with finer temporal granularity. Beyond architectural and data improvements, future work could explore the use of real-world frame-level annotations, better evaluation protocols, KL penalty to align frame-level outputs with global model, and generative augmentation strategies to further enhance open-vocabulary localization.”

We sincerely thank the reviewer bqpP for their detailed and thoughtful review. We appreciate your recognition of FLAM’s contributions and have revised the manuscript to address your comments and questions. Below, we respond point-by-point, reordered for clarity. Due to space constraints, we include additional results and responses on the supplementary webpage:
https://flam-model.github.io/response_reviewer_bqpp.html

---

Questions Related to Dataset

Dataset size and audio length
As noted in Sec. 4.1, the training set includes 1.1M samples. The 1M augmented samples are 10 seconds each, matching FLAM’s fixed-length input (line 317). Original text-audio clips are variable-length; we sample 10-second segments during training.

How many types of events are covered in the captions?
The captions span most categories in the Universal Category System (UCS). These include nature sounds (e.g., thunder, rainfall, bird calls), urban and human-made sounds (e.g., car engines, speech), and professionally designed sound effects (e.g., gunshots, lightsabers).

License of dataset
All proprietary datasets are fully licensed. We clarified this in the revised opening of Sec. 4.1:

“We gather a large mix of licensed proprietary sound effect datasets and ...”

Will the training dataset for FLAM be made available?
Due to licensing, the full dataset cannot be released. However, we will release the ASFX-SED dataset generated by our augmentation pipeline. We hope this will provide a valuable benchmark for future research in open-vocabulary SED.

How effective are the dataset augmentation techniques?
Prior to our work, no training dataset existed for open-vocabulary SED. We show how to scale the construction of such a dataset via data augmentation techniques. This enables frame-level training, since we know frame-level event labels by construction. This makes a crucial difference for significantly outperforming MGA-CLAP (Table 1), which is trained without explicit frame-level supervision.

---

Additional Technical Questions

How large is FLAM? How long does it take to train?
FLAM has ~150M parameters. FLAM-Global trains in ~12 hours; full FLAM in ~24 hours. It shares LAION-CLAP backbones but adds:

1. A 1024-dim projection head
2. Two MLPs predicting the per-text bias ($\beta^t$) and per-text scale ($\alpha^t$)

In comparison, CompA is larger, using HTSAT-large and FLAN-T5. We updated line 400 to clarify:

“Relative to larger-scale ALMs like CompA, FLAM remains competitive, particularly on VGGSound.”

Effect of varying L (number of frames)
We thank the reviewer for this suggestion. Due to limited time, we trained a variant of FLAM with L=128 and observed a trade-off between SED performance and retrieval/zero-shot performance. See the supplementary link above for results.

In table 1, is the gap between MGA-CLAP* and MGA-CLAP (reported) explained by the different training datasets?
Yes, this gap is due to dataset differences. MGA-CLAP was trained on large-scale web data, including YouTube, which aligns with DESED and AudioCaps. MGA-CLAP* was retrained on our proprietary dataset, explaining the lower performance.

There seems to be a trade-off between frame representation and global representation [...]. Could this difference be due to model drift when training FLAM with the final loss? And if so, could a KL-penalty term with respect to the predictions of FLAM - Global be helpful (similar to the KL penalty used in DPO [1])?
Excellent observation! Indeed, your observation reflects a trade-off between local and global representation alignment. To investigate further, we conducted additional experiments training FLAM without the global loss. Removing global loss results in a slightly better SED performance but a significant drop in retrieval and zero-shot performance. Please refer to the results in the link above. We mitigate this trade-off via joint global contrastive optimization. As the reviewer suggested, incorporating a KL penalty to align frame-level outputs with the global model is a promising direction. We’ve noted this in the discussion for future work.

The impact and intuition about logit scale
Intuitively, a smaller logit scale increases the cosine distance between negative frame and text embeddings for the same loss effect. This helps the model capture finer distinctions in cosine similarity. Due to space constraints, please refer to the link above for more discussion.

---

Other Comments

Missing impact statement
Many thanks for pointing this out. We have added an impact statement at the end of the paper. See the link above for the updated version.

Typos
We thank the reviewer for highlighting the typos—these have been corrected. Due to space constraints, full details are in our response to reviewer GGdQ.