XAttnMark: Learning Robust Audio Watermarking with Cross-Attention

We sincerely thank the reviewer for their meticulous and constructive feedback. We will revise the manuscript by improving the introduction and mentioning the CAI initiative and the C2PA standard in the body of the paper. Our responses are as follows:

Q1. On the statistical significance of the results, and concerns on the #validation set.

We want to clarify that, while our test set is indeed composed of 100 audio files, we follow AudioSeal's protocol by hiding 100 messages per file, corresponding to a total of 10k watermarked audio samples. We then apply 16 audio transformations to each one of them. So the total number of samples contributing to our scores is 160k. To further validate the statistical significance of our results, we perform McNemar's test and Wilcoxon signed-rank tests across edits with 1e4 users. The results are reported in Tables A and B, which shows that the results on our setup are statistically significant in both attribution performance and perceptual quality.

Q2. On the definition of attribution accuracy, report of false attribution rate, and the results discrepancy from the two baseline papers.

In our work we define the attribution accuracy in a different way, as the fraction of correct attributions among the detected audio inputs, which equals to $1 - FAR$ (False Attribution Rate reported in AudioSeal). This definition decouples the attribution performance from the detection performance. To show a direct comparison between the different metrics, we report the results on both MusicCaps and VoxPopuli setups in Table C.

Q3&W.1 On the motivation of the proposed perceptual loss, the comparison with the TF-Loudness loss in AudioSeal, and the justification of additional complexity with score gain.

In Appendix C.8, we discussed the advantage of our proposed masking loss compared to the TF-loudness loss. The TF-loudness loss employs a coarse approach based on loudness differences within each tile, neglecting sophisticated auditory masking effects, such as the interactions between masker and maskee across tiles. Additionally, we found that using loudness difference as a discrepancy measure provides only weak supervision. In contrast, we have designed a more sophisticated TF-weighted MSE loss, which simulates a two-dimensional energy decay in the temporal-frequency domain, effectively identifying masker-maskee pairs, leveraging psychoacoustic principles. Furthermore, we utilize mean-square error in the mel-spectrogram domain as our discrepancy measure, providing more fine-grained guidance (see our qualitative comparison in Figure 10 in the appendix). To quantitatively justify the effectiveness of our proposed loss, we evaluate the attribution accuracy under different watermark strengths (controlled by PESQ ranges) in Figure A. These results clearly indicate that our method consistently achieves significantly higher attribution accuracy at each imperceptibility level.

Q4. On the lack of results comparing computational efficiency.

We additionally report the results on computational efficiency in Table D.

W2. On the localization capabilities of XAttnMark.

Thanks for this insightful point. Although we have not explicitly discussed the localization capabilities of XAttnMark, our model can be easily extended to have this ability with sliding window detection. Specifically, since our model includes shifting-robust transformations and operates on 1s segments, we can distribute the per-segment detection probability to the per-frame level with multiple overlapping detection windows as the BFD in WavMark does. We have implemented this and report the results in Figure B. Results show that XAttnMark can achieve comparable localization performance to AudioSeal and significantly outperforms WavMark.

Q5. On the comparison with stronger gradient-based attacks in the semi-black box and white box settings and fairness on the HSJA.

We additionally report the robustness against the white-box, semi-black-box attacks in Figure C. Results show that XAttnMark is slightly more vulnerable to these two attacks compared to AudioSeal (might be attributed to our smaller detector). On the fairness concern on HSJA, we clarify that we use the same attack budget for the two methods, and the higher perceptibility score in ours is because HSJA fails to successfully find more adversarial samples within the given budget compared to AudioSeal's case.

Q6. On the contribution of the cross-attention and the conditioning module.

Regarding the interpretation of Fig. 4, we clarify that our claim is that, when considering a single module in isolation, the cross-attention module (acc. of 62%) is more effective than the temporal conditioning module (acc. of 50%) in enhancing efficiency.

We sincerely appreciate the reviewer's insightful attention and precise feedback. We have addressed the reviewer's concerns as follows:

Q1: How about the comparison of model's parameters, training speed and inference speed?

Response: We additionally report the model size, training speed, and inference speed comparison with AudioSeal in Table D. Our model uses fewer parameters and has a smaller size for both the generator and the detector. Although our generator has higher FLOPs and a slightly increased inference time per segment (~0.3 ms/segment), our detector significantly reduces FLOPs while maintaining similar inference efficiency overall. In training speed, our model achieves a similar second-per-iteration rate (around 1.15s/iter) as AudioSeal, with faster convergence speed in learning message decoding. Specifically, as shown in Appendix C.2, our model takes ~4k steps to reach perfect detection accuracy, and ~10k steps to reach perfect attribution accuracy, while AudioSeal takes ~32k steps to reach perfect detection accuracy, and 50k steps to reach around 70% attribution accuracy. This demonstrates that XAttnMark achieves 5 to 8 times better training efficiency than AudioSeal.

---

Q2: As discussed above, the paper focuses on illustrating technical details while lacking more deeper insights for robust audio watermarking.

Response: Thank you for this valuable point. We will refine our presentation to include more design insights. Due to the space limit, we put a significant part of the details on the design motivation in the appendix sections. In the appendix, we have provided more analysis and discussion on the proposed modules, including the cross-attention architecture and the proposed temporal-frequency perceptual loss. Specifically,

• In Appendix C.2. (Analysis of the Training Dynamics of Models with Different Architectures), we analyze the training dynamics of different architectures under a controlled experimental setup to better understand their inherent learning capabilities.
• In Appendix C.8, we provide a comprehensive comparison with the TF-loudness loss of AudioSeal.

During the revision of the paper, we will add these deeper technical insights to the main text for better readability.

We sincerely thank the reviewer for taking the time to review our manuscript and providing valuable feedback. We have carefully considered each point raised and provide our detailed responses below:

Q1. The subjective listening test involves a relatively small number of participants.

Response: We launch our subjective listening test with 18 participants initially following the ITU-R BS1534-1 [1] standard and practice in related audio publications [2,3]. Specifically, ITU-R BS1534-1 suggests that when the conditions of a listening test are tightly controlled on both the technical and behavioral side, experience has shown that data from no more than 20 subjects are often sufficient for drawing appropriate conclusions from the test. Our internal test with unified software/process and participants from a similar background profile satisfies this control requirement.

During the post-screening process, we further filtered out 6 participants who missed the reference audio to ensure the result's validity, resulting in 12 valid evaluators. Similarly, SilentCipher[3] also performs post-processing on the test group results with 12 valid evaluators in total. While these references support our setup, we acknowledge that the population size for our MUSHRA test is relatively limited. If needed, we will expand the test population and update our results in the final version of the paper.

[1] Method for the subjective assessment of intermediate quality level of coding systems (Recommendation ITU-R BS.1534-1), International Telecommunication Union. (2003).

[2] Davidson, G., Vinton, M., Ekstrand, P., Zhou, C., Villemoes, L., & Lu, L. (2023). High Quality Audio Coding with MDCTNet. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP).

[3] Singh, M. K., Takahashi, N., Liao, W., & Mitsufuji, Y. (2024). SilentCipher: Deep Audio Watermarking. In Proc. Interspeech 2024 (pp. 2235-2239).

---

Q2. The ablation study on the adaptive bandwidth (constant weight $\gamma$ = 1) is limited to a single configuration.

Response: Our ablation study on the adaptive bandwidth mainly focuses on the two most representative cases, the constant weighting and the adaptive weighting for per-mel masking radii, which we propose in our work. In our design, the per-mel-bin masking radii $r^m$ are adjusted based on the frequency instead of being set as a constant across all frequencies. Adjusting different $\gamma$ values can be viewed as a hyperparameter tuning process on the base radius $r^m_b$, which still assigns the same radii across all the mel-bins and conceptually belongs to the same class of constant weighting. Due to the time constraint, we leave the exploration on this hyperparameter searching for constant weighting as a future work.

We sincerely thank the reviewer for their valuable feedback. We have addressed the reviewer's concerns as follows:

W1. The model still struggles with extreme transformations like speed changes (acknowledged in the text).

In the paper we show that the model is able to effectively perform the detection task under speed changes transformations. We also acknowledge that the model struggles against speed changes (and other challenging transformations like generative edits) for the attribution task. However, in the Appendix C.3.1, we show that, for challenging speeding operation, we could build up a simple speed reversion layer, that greatly improve the attribution performance without significant overhead (as shown in Table 7 in appendix).

W2. Attribution performance under generative edits was not deeply analyzed; only detection was reported.

As mentioned earlier, we acknowledge that our model is still limited in attribution robustness against generative edits. However, we believe that this still marks a significant step forward in watermarking against generative audio edits. To the best of our knowledge, we are the first to report non-trivial detection robustness (90%+) against generative editing in a zero-shot manner. With additional specialized training on those transformations, the attribution performance might be further improved. We leave this as future work.

W3. The paper does not evaluate robustness under white-box adversarial attacks.

We additionally report the robustness against white-box adversarial attacks in Figure C.

C1. The paper would benefit from clarifying the decoding pipeline under attribution evaluation with large user pools (e.g., scalability of Hamming decoding).

Please refer to the 'Evaluation Setup' part of Appendix A and Table 6, where we have provided a detailed discussion on the Hamming decoding process used in the attribution evaluation and also the scalability aspect.

C2. Consider releasing code/models to improve reproducibility and adoption.

Thank you for the suggestion. We will consider releasing the code and models upon the paper's publication.

Q1: Can the attribution method be extended to a continuous space (e.g., using embeddings) to improve robustness under generative edits?

This is an interesting idea. Currently, the existing watermarking methods are mostly designed for embedding discrete bit-strings (e.g., 0s and 1s). However, our experiments show that, under the challenging generative editing, previous methods fail to succeed at both detection and attribution. One potential reason is that we treat the source as discrete informationless bit-strings, without leveraging the semantic information that could help the attribution task (e.g., style attribution [1]). For example, in the audio domain, a copyrighted timbre might have countless reference audio files, which could be leveraged as attribution anchors for more robust attribution. This is an orthogonal direction to our research, which we leave as future work.

[1] Wang, Sheng-Yu, et al. "Evaluating data attribution for text-to-image models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Q2: How sensitive is the performance to the architecture of the embedding table and temporal conditioning module?

We additionally provide the sensitivity study on the embedding hidden dimension and temporal conditioning architecture in Figure D. For the embedding table, we found that the embedding dimension $H$ will affect the convergence speed of the watermark model in both detection and message decoding (more sensitive to $H$ in the message decoding part). With grid search over $[\frac{b}{2}, b, 2b, 4b, 8b]$, where $b$ is the bit-length of the secret message ($b=16$), we found that $H$ values ranging from $H=\frac{b}{2}$ to $H=4b$ both yield fast convergence except for $H=8b$. For the temporal conditioning module, we additionally provide an ablation study with different numbers of MLP layers (linear, 2-layer MLP, and 3-layer MLP). Results show that the linear projection proposed in XAttnMark is the only one that converges, indicating that the convergence is sensitive to the architecture choice of the temporal conditioning module.

Q3: The paper does not evaluate robustness under white-box adversarial attacks. How would the proposed method perform under white-box adversarial attacks compared with existing methods?

We additionally report the robustness against white-box, semi-black-box, and Gaussian noise attacks in Figure C. The results show that XAttnMark is more robust than AudioSeal in Gaussian noise attacks. In the white-box and semi-black-box attack scenario, we observe that XAttnMark is slightly more vulnerable to white-box attacks compared to AudioSeal, which might be due to the smaller model size of the detector module (XAttnMark is 7.59M, while AudioSeal is 8.65M).