An Implicit Watermark Framework for Adversary Identification

• The writing quality of the introduction and methodology sections is rather low, and some of the key concepts were difficult to interpret on a first pass. The writing quality improves as one delves deeper into the paper. There were some confusing word choices throughout the manuscript (e.g., water used interchangeably with watermark). The clarity of the early sections is low.
• Much of the notation could be cleaned up, particularly around Equation 2. The main issue is the number of repetitive parentheses. It also isn't clear why the authors choose $f_{att}$ when the model copies are denoted as $g$.
• The framework is designed with the assumption that multiple customers are performing the same classification task on the same training data, but this seems unrealistic to me. I would've liked to see the performance when different data is used to fine-tune customer models.
• The evaluation against adaptive attacks seems limited, as the authors only checked existing attacks and later adapt the watermarking hyperparameters, rather than consider the attack which targets the hyperparameters explicitly (see questions for the authors).
• Some design choices are unclear, such as needing the true label of the adversarial sample, or relying on the confidence delta of the output logits (see questions for the authors).
• The proposed framework is less effective against black-box adversaries, particularly attacks based on single-point gradient estimation (HSJA), which are the most realistic within the threat model.

Weak experiment against backdoor defenses

To me, Section 4.3 is the most important in the paper; it addresses the largest concern of this empirical method. In particular, the assumption is that the adversary is (1) unaware of the backdoor pattern and is (2) unable to remove the backdoor. However, the experiment still falls a bit short for multiple reasons.

1. There is a rich literature on solving both of these problems as a defense against backdoor attacks: backdoor removal [1, 2, 3, 4] and backdoor detection [5, 6, 7]. It looks like the paper only covers one removal method and one detention method but does not explain why they were chosen instead of more recent and more powerful methods.
2. Backdoor removal. Generally, I would like to see more results to support the claim that the removal is ineffective when random erasing is used as data augmentation. This result on its own seems surprising and particularly important not only to this work but also to the broader backdoor attack/defense community. A few suggestions on how to strengthen this experiment:
   1. Explain why random erasing is effective.
   2. Do other data augmentation methods also lead to more robust backdoors?
   3. Does this conclusion hold against more sophisticated or more aggressive removal methods?
   4. Results on more datasets and models would make the argument more convincing as well.
3. Backdoor detection. What makes Neural Cleanse ineffective at detecting the backdoor? Why does using $|\mathbf y_i|=4$ instead of $2$ solve the problem? Have other detection methods been tried? Similarly to the backdoor removal result, I cannot find any table or figure that describes the quantitative results in this section at all.
4. If the state-of-the-art backdoor detection method still does not work, I may recommend a naive backdoor detection in this case: compute gradients of the loss w.r.t a test input (similar to how one generates adversarial examples), and then select the pixels with the top-k largest absolute gradients as the mask $m_i$. I would love to see the “accuracy” of multiple detection methods including this naive one.
5. Building on the previous point, it might be worth considering what the adversary could do to avoid being fingerprinted if he/she knows (1) both $m_i$ and $t_i$, (2) only $m_i$, and (3) only a noisy (or a subset) version of $m_i$. How effective would the adversary’s strategy be in each of these cases?
6. Lastly, the threat model should be stated more clearly, and there should be a discussion on the possibility of adaptive attacks, what their costs are, etc.

Adaptive adversarial attacks

I believe that it is also important to consider adaptive attacks that do not involve removing or detecting the backdoor. Especially, the authors raise an interesting point on page 6 that “identification result on the black-box attack is not as good as white-box attack tested because of the noise gradient estimation used in the black-box attack.” This immediately suggests an adaptive attack that modifies any white-box attack by adding noise to the gradients. More generally, I wonder if methods for generating physically robust adversarial examples (i.e., expectation over transformation or EoT [8]) that apply random transformations to the input could be an effective adaptive attack. This is rather promising because attacks of this type are good at making the adversarial perturbation exploit more diverse and more robust features (i.e., lower frequency).

Comparison to Cheng et al. [2023]

The authors note the reason for not comparing their scheme to Cheng et al. [2023] as “Cheng et al. [2023] is specifically designed for black-box attack identification which makes explicit modifications on the architectures and the white-box attacker could create strong adaptive attack to make the identification totally fail…” I do not believe that this reason does not invalidate the comparison under no adaptive attacker. If anything, this comparison could highlight the strength of the proposed scheme.

False positive rate on attacks from non-watermarked models

I wonder what happens if the adversary uses a different (perhaps public) model not among the watermarked ones to generate adversarial examples. Since this paper formulates the identification task as a classification problem, one of the models will always be “blamed” for any given adversarial example. This can be extremely problematic so there must be an additional mechanism for determining if an adversarial example really comes from one of the fine-tuned models (e.g., detection threshold on the score).

Other solutions to the same problem

As I understand, this paper focuses on fine-tuning services as an application where the adversary may use one of the many very similar copies of the model to mount a transfer to the other model copies. This is indeed an interesting and practically well-motivated setting, but I think there may be other defenses to solve the same problem.

For instance, if the transfer attack is the main concern here. The service provider may opt to deploy standard defenses against adversarial examples such as “adversarial training.” Other solutions include training methods that try to diversify each copy of the model [9, 10, 11]. These defenses against (transfer) attacks directly should be preferred to methods for identifying the culprit after the fact. I would like to see a detailed discussion that compares the pros and cons of different solutions.

Clean accuracy vs identification accuracy trade-off

It should be possible to strengthen the watermark efficacy at the cost of the clean accuracy of the watermarked models. What are the knobs the service providers can use to tune this trade-off? Currently, the authors simply state the hyperparameter choices but do not clearly justify them. This experiment would be very interesting and should help strengthen the paper.

Cost

I can see two main overheads of the proposed technique.

1. The score has to be computed over all the fine-tuned models so the cost scales with the number of models. In other words, the number of forward passes needed is $NM$ where $N$ is the number of models and $M$ is the number of adversarial examples being identified. $N$ can be anywhere from thousands to millions.
2. The ratio metric requires the ground-truth label. This means that there must be a human in the loop to provide this label.

While these costs should not completely prevent the deployment of this technique, I would still like to see these limitations mentioned and discussed in the paper.

Ablation study

1. $T$ should be treated as a hyperparameter, and the result should be reported with the optimal choice of $T$. This can be tricky because a good value of $T$ also depends on the attack algorithm. There should exist a trade-off of identification accuracy among the attacks. There should be more experiments on how to choose a good value of $T$ in practice.
2. In the “effect of watermark design” paragraph, $T$ should be optimized for each strategy (discrete and square) instead of fixing it to the same value. There is no reason to believe that one value of $T$ will be good across different schemes.
3. I would like to see an ablation study on the choice of the identification metrics (KL and ratio). What is the accuracy if only one of the metrics is used at a time? What is the accuracy if other simpler schemes are used to average the two metrics?

• [1] https://arxiv.org/abs/1906.10842
• [2] https://arxiv.org/abs/2101.05930
• [3] https://proceedings.mlr.press/v139/shen21c.html
• [4] https://proceedings.mlr.press/v139/hayase21a.html
• [5] https://ieeexplore.ieee.org/document/9710346
• [6] https://arxiv.org/abs/2212.09067
• [7] https://arxiv.org/abs/2306.17441
• [8] https://arxiv.org/abs/1707.07397
• [9] https://arxiv.org/abs/2009.14720
• [10] https://arxiv.org/abs/2104.00671
• [11] https://proceedings.mlr.press/v162/rusu22a.html

$\bullet$ The scenario and threat model of this work are not clear. In particular, what are the objectives and capabilities of both attackers and defenders? why are such attack and defense assumptions reasonable? Indeed, from the defender's perspective, a white-box attack assumption is more challenging than the black-box setting employed in previous research, however, we should question whether it is necessary and reasonable to give attackers such extensive capabilities. The authors should elaborate more on scenarios where public access to the weights and architectures of the victim's or defender's models is necessary and justified. In other words, what is the rationale for requiring the architecture of the models to be publicly available? Moreover, in cases where the defender has identified the specific model being targeted for adversarial example generation, what should be the subsequent course of action? Without addressing these real-world issues and establishing a clear threat model, this work may lack practicality in real-world applications. The authors need to clarify the attack scenarios, describing their rationality and practical implications; clarify and formulate the goal of the attacker (e.g., to increase the attack success rate on a single model or to increase the transferability across multiple copies of a model), and the goal of the defender/recognizer.

$\bullet$ From a technical perspective, the approach proposed in this paper bears resemblance to the concept of a "Honeypot" employed for capturing adversarial attacks[Ref-1]. In [Ref-1], the authors deliberately introduce trapdoors, causing attackers' optimization algorithms to converge towards these trapdoors. The fingerprints developed in this paper represent another form of backdoors, which share similarities with the trapdoors described in [Ref-1]. Therefore, I find that the novelty within the technical aspect is somewhat restricted.

$\bullet$ How to ensure that watermarking has a negligible impact on the performance of each model replica? Table 1 shows that after fine-tuning, the classification accuracy of the model replicas decreased by 2%-5%.

$\bullet$ Why does the proposed method use a linear combination of two similarity metrics as the final similarity metric? Specifically, what is the physical significance of the Ratio metric proposed in this paper, and what is the physical significance of the linear combination of the KL distance and the Ratio metric?

$\bullet$ Experiments lack comparisons with baseline methods. For example, Table 1-4 only presents the performance of the proposed method without comparison with baseline methods.

$\bullet$ Some of the experimental settings are confusing. For example, in Section 4.1, why was it applied using 0.9% of the total number of images, and in Section 4.2, why was the threshold T set to 7?

[Ref-1] Shan, Shawn, et al. "Gotta catch'em all: Using honeypots to catch adversarial attacks on neural networks." Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security. 2020.

(1) The assumption of the threat model is strong. The proposed framework is only useful when there is one and only one adversary among all model copies. This is a strong limitation. It is possible that adversarial examples come from multiple adversaries, in which case the framework can still identify only one of them.

(2) More evidence is needed to support the claimed assumption. (i) For example, the assumption for the KL metric, “when the model predicts x_adv with high confidence on the fingerprint class y_j”, then“the generated adversarial perturbation would be very similar with the sampled pixels t_i” is not well supported. (ii)Also, the assumption for the ratio metric, “we observed that there would be a significant change on the model prediction distribution after applying t_i for the model that x_adv is based”, is not well supported. After applying t_i to the clean model, wouldn’t there also be a change in the distribution? How do g_att and clean model g_i differ from each other?

(3) Claims for the robustness of the proposed watermark are not well supported. For the fine-tuning-based methods, it’s claimed that simple data augmentation will make the proposed watermark more robust. However, when users fine-tune their copy with their own data and various data augmentation methods, how are original watermarks ensured not to be removed? A study of the impact of fine-tuning a clean model is needed. Even if there is only one adversary, the normal users’ copies of the model can also be fine-tuned, but the impact is not studied.

(4) For the reverse-engineering-based methods, it’s claimed that increasing the number of fingerprint classes will make these methods unable to detect watermarks. However, a study on the sensitivity of the outlier's threshold for the picked method, Neural Cleanse[a], is needed. Also, the claim, “the number of fingerprint class |y_i| has limited effect on the identification accuracy and it can even further improve identification accuracy” needs more experiments to be supported. Only the results with 4 fingerprint classes are reported in Table 3 and Table 4.

(5) Some notations are not well defined. Such as m_{a,b,c} in section 3.2.

[a] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707–723. IEEE, 2019.