Provable Watermarking for Data Poisoning Attacks

Thanks for your insightful comments and valuable suggestions. We provide responses below to address your concerns.

1. The overall setting feels disconnected from practical use cases. Can you clarify the practical scenarios in which your watermarking scheme would be deployed, and who the intended users are (e.g., researchers, data providers, regulators)?...

Any copyright owners can deploy our watermarking when releasing their original datasets to a third party (e.g., AI training platforms, academic institutions and copyright certification systems). We have already talked about the real-world application of Glaze and NightShade in our introduction. To make the threat model more concrete, we will provide a detailed deployment scenario below.

A company (called Alice) that collects a large proprietary dataset for autonomous driving research (e.g., dash cam video frames). She wants to open source a part of her dataset to promote innovation for community (e.g., Non-profit research organization), but also want to prevent unlicensed users to train a machine learning model on it successfully to protect her intellectual property.

To achieve the above goals, Alice runs our poisoning+watermarking algorithm on every instance of their dataset, publishing the perturbed (i.e., protected) dataset which is unlearnable by standard models and obtains a secret, key dependent watermark signal. She publishes this on her GitHub under a permissive license, accompanied by a SHA256 hash so any recipient can verify integrity.

A research lab (called Bob) registers on Alice’s portal and agree to a standard agreement for legal use of the dataset. After approval by Alice, Bob receives a secret key (e.g., a 128 bit seed) provided via Alice’s portal’s secure HTTPS channel. Furthermore, Bob also gains a pipeline (e.g., Python pre processing package) from Alice such that he can run the watermark detection to verify his identity and ensure that no file corruption. After the verification, Bob can run an algorithm designed by Alice (e.g., directly adding inverse unlearnable noise for each data) to remove the unlearnable poisons. If the pipeline receives wrong key or tampered file, the detection fails and the poisons cannot be removed to ensure the unlearnability.

For a malicious user (called Chad), first, Chad can download the same public poisoned and watermarked dataset, but cannot train a good model on it because the dataset is unlearnable. If Chad tries to remove or tamper watermarks and unlearnable poisons without knowing the secret key, detection will fail.

For key management, Alice can rotate keys per month and publish on her portal only to approved accounts (i.e., trusted users). Alice can also add a HMAC scheme to prevent potential forgery risks, which are raised by Reviewer RfGy, please see our rebuttal to Reviewer RfGy for details.

Compared with licensing+hash approach (proposed by Review RfGy), our poisoning+watermarking approach can deter unauthorized use of the data in model training, giving data owners capability for prevention without inducing heavy overhead. Licensing+hash can only prove file integrity, fail to prevent a malicious user Chad to train a well-performed model when the data is leaked.

2. The writing and presentation could be improved for clarity and accessibility…

Thanks for your suggestion. $d$ and $\epsilon$ denotes the data dimension and the perturbation budget respectively, we will add the description in our Abstract and Introduction in the next version. For mathematical symbols, notations or definitions, according to the suggestion of Reviewer 9bm6, we will add a dedicated symbol table in the next version of our paper to enhance the readability. For experimental setup, we will add a detailed description in the Appendix to make it clearer.

3. The scope of the experimental evaluation is too limited…

Beyond CIFAR-10, we have already provided additional experiments on CIFAR-100 and TinyImageNet in Appendix C.1. To further validate the effectiveness of our method, for text dataset, we implement watermarking ($\epsilon_w=16/255$) in a backdoor attack on SST-2 dataset with BERT-base model [a], observing similar trends (Section 6) for this NLP task.

Moreover, according to the suggestion of Reviewer 91mv, we also conduct a preliminary experiment on WikiArt Dataset [b], a historical artists dataset to test our watermarking under Glaze protection [c]. Following the setting of Glaze, we finetune Stable Diffusion 2.1 to mimic victim artists’ style under Glaze protection with our post-poisoning and poisoning-concurrent watermarking respectively. We standardize the image of WikiArt to 768*768 and set watermarking dimension $q=5000$ with perturbation budget $\epsilon=8/255$. For evaluation, beyond AUROC, we follow [c] to test the protection performance using CLIP-based genre shift. Results show that our watermarking achieves high detection performance while keeping the protection of Glaze.

Furthermore, for multi-modal data, following the experiment of [d], we use the MS-COCO dataset to generate text-image pairs, and conduct invisible backdoor attack in the image modality. We test both post-poisoning and poisoning-concurrent watermarking with $q=10000$ and $\epsilon=2/255$. Results demonstrate perfect detection performance (higher AUROC), decent backdoor performance (higher Benign Accuracy, higher ASR), and good image quality (higher PSNR, higher SSIM, lower MSE)

Beyond these, according to the suggestion of Reviewer 9bm6, we further test our post-poisoning and poisoning-concurrent watermarking on ViT-B model [e] under UE, AP availability attacks and Narcissus, AdvSc backdoor attacks with different watermarking length $q$. The patch size is change from 16 to 4 in order to meet with the size of CIFAR-10. Similar conclusions can be summarized from the following results compared with discussions in Tables 1 and 2 in Section 6.2 on ResNet-18. Therefore, we believe our watermarking work well on modern architectures like Transformer.

4. data preprocessing, augmentation, and noise, none of which are accounted for in the current experiments.

We have considered many types of data augmentations, including Random Flip, Cutout, Color Jitter and Grayscale, as well as some regeneration attacks including VAE-based attack and GAN-based attack in Appendix D. Results in Table 10 show quite high robustness of our proposed watermarking. Furthermore, we also evaluate differential privacy noises with both Gaussian and Laplacian mechanism in Table 11, the poisoning utilities have been completely destroyed. Therefore, DP-based defenses are not applicable in our context. For data preprocessing like Normalization, our results (both theoretical and empirical) can be directly expanded with just some affine transformation. For other types of preprocessing, as our watermarking creator can be also served as poisoner (especially in poisoning-concurrent watermarking), they can just add watermarking after preprocessing to ensure the theoretical guarantee of our proposed watermarking.

Ref:

[a] Gan et al. Triggerless Backdoor Attack for NLP Tasks with Clean Labels. NAACL 2022.

[b] Saleh and Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv:1505.00855.

[c] Shan et al. Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models. USENIX Security 23.

[d] Zhang et al. BadCM: Invisible Backdoor Attack Against Cross-Modal Learning. IEEE TIP 2024.

[e] Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021.

Thanks for your positive feedback and valuable comments. We provide responses below to address your concerns.

1. Limited to imperceptible clean-label attacks; Could the framework extend to noisy-label or targeted poisoning attacks?

Thanks for your advice. Beyond clean-label attacks, we extend our experiments to dirty-label backdoor attacks in order to strengthen our findings. We evaluate three dirty-label attacks, including BadNet attack [a], Trojaning attack [b] and Invisible attack [c] on CIFAR-10 dataset. Following the original setting, in BadNet attack, we set the poisoning rate be 0.1, in Trojaning attack and Invisible attack, we set the poisoning rate be 0.05 and poisoning budget be $8/255$ under $L_{\inf}$ norm. We test both post-poisoning and poisoning-concurrent watermarking with length $q=1000$. Results shown that our watermarking algorithms embed well on these dirty-label backdoor attacks, achieve high test performance (AUROC) without affecting the poisoning utility (Acc and ASR).

For targeted poisoning attacks, we further conduct our experiments to two well-known attacks, Poison Frogs [d] and Witches Brew [e] on CIFAR-10 dataset with post-poisoning and poisoning-concurrent watermarking under $q=1000$. Similar trends shown in our results demonstrate the effectiveness of our watermarking under targeted poisoning attacks.

2. Assumes large N and d for theoretical guarantees. How does watermarking perform on low-dimensional data (e.g., tabular) where d is small?

Although in theoretical results, it may require larger sample number $N$ and data dimension $d$, in practice, we find that our watermarking schemes work well for relatively low-dimension dataset. For instance, we test our watermarking under BadNet backdoor attacks on MNIST dataset ($d=768$) and USPS dataset ($d=256$). We set $q=400$ for MNIST and $q=150$ for USPS. Evaluation provided below demonstrate that, even for relatively low-dimensional dataset, both post-poisoning and poisoning-concurrent watermarking still display good detection performance as well as keeping backdoor utilities.

For tabular data, since most of them contain a large number of discrete/categorical variables (e.g., gender, regional code, risk level, etc.), the setting of imperceptible poisoning noises on such features often destroys semanticity. Therefore, it may not exist the scenario that using small perturbed data for copyright protection under tabular data. We may leave the watermarking of tabular data poisoning as the future work.

3. Defense Mitigation: Could adaptive watermarking (e.g., dynamic watermark) improve robustness against purification attacks?

Thanks for your valuable suggestion. For potential adaptive watermarking, we may train a surrogate network (encoder) to transfer universal watermarking into sample-wise version while keep the detection key be single. In this case, watermarks can be more elaborate and robust against stronger removal attacks, e.g., purification attacks. However, as our main focus on this paper is to provide a provable watermarking for data poisoning attacks, similar theoretical guarantees under more elaborate and adaptive watermarking may require more detailed analyses and rigorous proofs. We will add these discussions into the limitation and leave them as the future work.

Ref:

[a] Gu et al. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv:1708.06733.

[b] Liu et al. Trojaning Attack on Neural Networks. NDSS 2018.

[c] Li et al. Invisible Backdoor Attacks on Deep Neural Networks via Steganography and Regularization. arXiv:1909.02742.

[d] Shafahi et al. Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. NeurIPS 2018.

[e] Geiping et al. Witches’ Brew: Industrial Scale Data Poisoning via Gradient Matching. ICLR 2021.

Thanks for your positive feedback and valuable comments. We provide responses below to address your concerns.

1. Unrealistic threat model.

The poisoning considered in our paper are more refer to copyright protection, which has been discussed in the Introduction. Concretely, poisoner itself is a data provider. Watermarking does not consider whether the data provider is trusted, but rather whether users trustworthy. If they are trustworthy, watermarked dataset will inform users that the data is protected from poisoning, allowing them to reverse the operation to extract a clean dataset for normal use. For an malicious and untrusted user, he cannot obtain authorization as no detection key holds. Therefore, he can only get an unlearnable dataset. To make the threat model more concrete, we will provide a detailed deployment scenario below.

A company (called Alice) that collects a large proprietary dataset for autonomous driving research (e.g., dash cam video frames). She wants to open source a part of her dataset to promote innovation for community (e.g., Non-profit research organization), but also want to prevent unlicensed users to train a machine learning model on it successfully to protect her intellectual property.

To achieve the above goals, Alice runs our poisoning+watermarking algorithm on every instance of their dataset, publishing the perturbed (i.e., protected) dataset which is unlearnable by standard models and obtains a secret, key dependent watermark signal. She publishes this on her GitHub under a permissive license, accompanied by a SHA256 hash so any recipient can verify integrity.

A research lab (called Bob) registers on Alice’s portal and agree to a standard agreement for legal use of the dataset. After approval by Alice, Bob receives a secret key (e.g., a 128 bit seed) provided via Alice’s portal’s secure HTTPS channel. Furthermore, Bob also gains a pipeline (e.g., Python pre processing package) from Alice such that he can run the watermark detection to verify his identity and ensure that no file corruption. After the verification, Bob can run an algorithm designed by Alice (e.g., directly adding inverse unlearnable noise for each data) to remove the unlearnable poisons. If the pipeline receives wrong key or tampered file, the detection fails and the poisons cannot be removed to ensure the unlearnability.

For a malicious user (called Chad), first, Chad can download the same public poisoned and watermarked dataset, but cannot train a good model on it because the dataset is unlearnable. If Chad tries to remove or tamper watermarks and unlearnable poisons without knowing the secret key, detection will fail.

For key management, Alice can rotate keys per month and publish on her portal only to approved accounts (i.e., trusted users). Alice can also add a HMAC scheme to prevent potential forgery risks, which are raised by Reviewer RfGy, please see our rebuttal to Reviewer RfGy for details.

Compared with licensing+hash approach (proposed by Review RfGy), our poisoning+watermarking approach can deter unauthorized use of the data in model training, giving data owners capability for prevention without inducing heavy overhead. Licensing+hash can only prove file integrity, fail to prevent a malicious user Chad to train a well-performed model when the data is leaked.

2. No evaluation on other model types. How the watermark performs on other model families, such as transformers or GNNs?

We have evaluated many well-known architectures including ResNet, VGG, DenseNet, MobileNet and Wide-ResNet in Appendix C.2. We further test our post-poisoning and poisoning-concurrent watermarking on ViT-B model [a] under UE, AP availability attacks and Narcissus, AdvSc backdoor attacks with different watermarking length $q$. The patch size is change from 16 to 4 in order to meet with the size of CIFAR-10. Similar conclusions can be summarized from the following results compared with discussions in Tables 1 and 2 in Section 6.2 on ResNet-18. Therefore, we believe our watermarking work well on modern architectures like Transformer.

Furthermore, according to the suggestion of Reviewer 91mv, we conduct a preliminary experiment on WikiArt Dataset [b], a historical artists dataset to test our watermarking under Glaze protection [c]. Following the setting of Glaze, we finetune Stable Diffusion 2.1 to mimic victim artists’ style under Glaze protection with our post-poisoning and poisoning-concurrent watermarking respectively. We standardize the image of WikiArt to 768*768 and set watermarking dimension $q=5000$ with perturbation budget $\epsilon=8/255$. For evaluation, beyond AUROC, we follow [c] to test the protection performance using CLIP-based genre shift. Results show that our watermarking achieves high detection performance while keeping the protection of Glaze.

Moreover, according to the suggestion of Reviewer aXke, we further conduct experiments for multi-modal data. Following the experiment of [d], we use the MS-COCO dataset to generate text-image pairs, and conduct invisible backdoor attack in the image modality. We test both post-poisoning and poisoning-concurrent watermarking with $q=10000$ and $\epsilon=2/255$. Results demonstrate perfect detection performance (higher AUROC), decent backdoor performance (higher Benign Accuracy, higher ASR), and good image quality (higher PSNR, higher SSIM, lower MSE).

3. There is significant notation overload. Many symbols are introduced before being properly defined, which impairs readability. A dedicated symbol table or an overview figure early in the paper would help readers more effectively.

Thanks for your valuable suggestion. We will add a dedicated symbol table in the next version of our paper to enhance the readability.

Ref:

[a] Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021.

[b] Saleh and Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv:1505.00855.

[c] Shan et al. Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models. USENIX Security 23.

[d] Zhang et al. BadCM: Invisible Backdoor Attack Against Cross-Modal Learning. IEEE TIP 2024.

Thanks for your positive feedback and valuable comments. We provide responses below to address your concerns.

1. How does your theoretical framework apply to the complex perturbations of poisoning-for-good tools like Glaze and Nightshade?

Thanks for your valuable suggestion. We conduct a preliminary experiment on WikiArt Dataset [a], a historical artists dataset to test our watermarking under Glaze protection [b]. Following the setting of Glaze, we finetune Stable Diffusion 2.1 to mimic victim artists’ style under Glaze protection with our post-poisoning and poisoning-concurrent watermarking respectively. We standardize the image of WikiArt to 768*768 and set watermarking dimension $q=5000$ with perturbation budget $\epsilon=8/255$. For evaluation, beyond AUROC, we follow [b] to test the protection performance using CLIP-based genre shift. Results show that our watermarking achieves high detection performance while keeping the protection of Glaze.

2. The experiments are conducted on small and traditional architectures (e.g., ResNet-18, VGG-19). This limits the generalizability of the findings to the large-scale scenarios that motivate the work.

We have evaluated many well-known architectures including ResNet, VGG, DenseNet, MobileNet and Wide-ResNet in Appendix C.2. We further test our post-poisoning and poisoning-concurrent watermarking on ViT-B model [c] under UE, AP availability attacks and Narcissus, AdvSc backdoor attacks with different watermarking length $q$. The patch size is change from 16 to 4 in order to meet with the size of CIFAR-10. Similar conclusions can be summarized from the following results compared with discussions in Tables 1 and 2 in Section 6.2 on ResNet-18. Therefore, we believe our watermarking work well on modern architectures like Transformer.

Furthermore, for text dataset, we implement watermarking ($\epsilon_w=16/255$) in a backdoor attack on SST-2 dataset with BERT-base model [d], observing similar trends (Section 6) for this NLP task.

Moreover, according to the suggestion of Reviewer aXke, we further conduct experiments for multi-modal data. Following the experiment of [e], we use the MS-COCO dataset to generate text-image pair, and conduct invisible backdoor attack in the image modality. We test both post-poisoning and poisoning-concurrent watermarking with $q=10000$ and $\epsilon=2/255$. Results demonstrate perfect detection performance (higher AUROC), decent backdoor performance (higher Benign Accuracy, higher ASR), and good image quality (higher PSNR, higher SSIM, lower MSE).

3. The lack of error bars.

Thanks for your advice. We run both post-poisoning and poisoning-concurrent watermarking with $q=1000$ under UE, AP availability attacks and Narcissus, AdvSc backdoor attacks for five times on different random seeds. Results shown below demonstrate that our watermarking are not sensitive to random seeds.

4. The DP noise defense invalidates the model's utility. Can you discuss a more realistic adaptive attack where an adversary tries to remove the watermark while preserving model utility?

Our watermarking algorithms are either robust to existing attacks, or induce attacks to degrade poisoning utilities, which means our watermarking under poisoning are effective and without negative impact. For potential adaptive attacks, we can consider a scenario that requiring higher ability for attackers: Assuming that attackers have both watermarked and non-watermarked images, and they can detect them effectively (e.g., obtaining the secret key). Our watermarks can be removed by simply subtract the average of watermarked images and which of the non-watermarked images as our watermarking are universal typically. For more finely designed adaptive attacks, we will take them as the future work.

Ref:

[a] Saleh and Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv:1505.00855.

[b] Shan et al. Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models. USENIX Security 23.

[c] Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021.

[d] Gan et al. Triggerless Backdoor Attack for NLP Tasks with Clean Labels. NAACL 2022.

[e] Zhang et al. BadCM: Invisible Backdoor Attack Against Cross-Modal Learning. IEEE TIP 2024.

Thanks for your insightful comments and valuable suggestions. We provide responses below to address your concerns.

1. Practical Threat Model and Real-World Use.

Any copyright owners can deploy our watermarking when releasing their original datasets to a third party (e.g., AI training platforms, academic institutions and copyright certification systems). To make the threat model more concrete, we will provide a detailed deployment scenario below.

A company (called Alice) that collects a large proprietary dataset for autonomous driving research (e.g., dash cam video frames). She wants to open source a part of her dataset to promote innovation for community (e.g., Non-profit research organization), but also want to prevent unlicensed users to train a machine learning model on it successfully to protect her intellectual property.

To achieve the above goals, Alice runs our poisoning+watermarking algorithm on every instance of their dataset, publishing the perturbed (i.e., protected) dataset which is unlearnable by standard models and obtains a secret, key dependent watermark signal. She publishes this on her GitHub under a permissive license, accompanied by a SHA256 hash so any recipient can verify integrity.

A research lab (called Bob) registers on Alice’s portal and agree to a standard agreement for legal use of the dataset. After approval by Alice, Bob receives a secret key (e.g., a 128 bit seed) provided via Alice’s portal’s secure HTTPS channel. Furthermore, Bob also gains a pipeline (e.g., Python pre processing package) from Alice such that he can run the watermark detection to verify his identity and ensure that no file corruption. After the verification, Bob can run an algorithm designed by Alice (e.g., directly adding inverse unlearnable noise for each data) to remove the unlearnable poisons. If the pipeline receives wrong key or tampered file, the detection fails and the poisons cannot be removed to ensure the unlearnability.

For a malicious user (called Chad), first, Chad can download the same public poisoned and watermarked dataset, but cannot train a good model on it because the dataset is unlearnable. If Chad tries to remove or tamper watermarks and unlearnable poisons without knowing the secret key, detection will fail.

For key management, Alice can rotate keys per month and publish on her portal only to approved accounts (i.e., trusted users). Alice can also add a HMAC scheme to prevent potential forgery risks (please see our response to Q3 for the detailed answer).

Compared with licensing and hash approach as you suggested, our poisoning and watermarking approach can deter unauthorized use of the data in model training, giving data owners capability for prevention without inducing heavy overhead. Licensing and hash can only prove file integrity, fail to prevent a malicious user Chad to train a well-performed model when the data is leaked.

2. Use of Separate Watermark and Poison Perturbations.

In practice, the poisons (both backdoor attacks and availability attacks) are sample-wise noises, that means every sample is injected with a specific poisoning noise. However, practical watermarking proposed in our scheme becomes universal, which makes it easier for key management, using a single detection key. Therefore, if we try to use the combined poison–watermark perturbation, we also need to make poisons become universal noises. Unfortunately, if we want to design universal poisons for the dataset, the poisoning utility is completely destroyed. As the result demonstrated, Universal UE and AP will recover the test accuracy to almost 90%, completely violate the goal of availability attacks: make dataset unlearnable. Similar trends also have been shown for universal backdoor attacks, the ASR drop to about 10%, resulting in the failure of backdoor triggers.

For potential direct watermarking with backdoor attacks ([arXiv:1802.04633v3], [arXiv:2208.00563v2], and [arXiv:2305.12502]), we highlight that these works are model watermarking, which embed watermarking signals into model parameters to achieve model ownership verification. However, our paper considers the provable watermarking under poisoned dataset (rather than model), to discern legitimate users and avoid unauthorized use for potential malicious users. Even if model parameters are unavailable, a legitimate user can still detect watermarking of the dataset with their secret key.

3. Key Management and Forgery Risk.

As our answer in Q1, Alice can rotate keys per month and publish on their portal only to approved accounts (i.e., trusted users). Alice can also add a HMAC scheme to prevent potential forgery risks. Specifically, we can separate keys into generation key $k_{gen}$ and authentication key $k_{auth}$, where $k_{gen}$ is completely same with our paper and correlate with injected watermarks $w_i$ for every data $x_i$. For each perturbed $\hat{x_i}$ with $w_i$, we can compute an additional tag $t_i$ by HMAC under $k_{auth}$, i.e., $t_i = HMAC_{k_{auth}}(id_i, \hat{x_i})$, where $id_i$ is a unique identifier for the image $x_i$ (e.g., index). After that, we store the ($id_i, t_i$) pair (e.g., through a sidecar JSON) for latter detection. In watermarking detection, beyond traditional detection using $k_{gen}$ and $\hat{x_i}$, we also verify the tag with $t_i$ and $t_i = HMAC_{k_{auth}}(id_i, \hat{x_i})$ to avoid the potential forgery attacks. In this case, even if the generation key $k_{gen}$ leaks, an attacker cannot forge a new valid $(x_i, t_i)$ pair as they lack the authentication key $k_{auth}$. We can keep $k_{auth}$ in a secure enclave and rotate it independently with $k_{gen}$ to enhance the security.

4. Adaptive Removal Risk Due to Poison–Watermark Separation.

Thanks for your suggestion. We conduct additional experiments on UE and AP with direct masking of the known watermarking dimensions (Masking), as well as the adversarial noising proposed by [arXiv:2309.16952v2]. We test both post-poisoning and poisoning-concurrent watermarking under $q=2000$. As the results shown below, although the detection performance (AUROC) drops, the utility of UE and AP also degrade significantly. The underlying reasons may be that availability attacks are designed with potential linear shortcut features [a,b], the masking of watermarking dimensions somehow destroy these linear features, undermining the unlearnability (low Acc). Adversarial Noising further destroys the poisoning utility as availability attacks are theoretically removed by perfect adversarial training [c]. Therefore, these adaptive removal attacks fail to maintain the poisoning utility, making them not applicable in our cases, similar with DP noises and purifications discussed in our Appendix D.

Ref:

[a] Yu et al. Availability Attacks Create Shortcuts. KDD 2022.

[b] Zhu et al. Defense of Unlearnable Examples. AAAI 2024.

[c] Tao et al. Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training. NeurIPS 2021.