Black-Box Privacy Attacks Against GANs via Detector Networks

Major Comments:

Q1: The proposed one-way distance-based attack is the same as Section 5.2 of GAN-leaks (Chen et al. 2020) The proposed two-way distance-based attack is the same as Section 5.2 + Section 5.6 of GAN-leaks (Chen et al. 2020). Please correct me in the rebuttal if I misunderstood it.

R1: The distance-based attacks are similar to the attacks proposed in GAN-leaks, which we state in the contributions section: ``We compare our Detector-based methods to an array of distance and likelihood-based attacks proposed in prior work Chen et al. (2020).'' These distance based attacks are implemented to provide comparison to our other black-box attacks. We also note that a small contribution of this work is executing these distance-based attacks with different distance-metrics, like LPIPS (https://arxiv.org/abs/1801.03924). We also provide the most extensive evaluation of these distance-based attacks across tabular and image data.

Q2: The proposed detector-based attack is the same as case 1 of the discriminative setting in Section 4.4 of LOGAN (Hayes et al. 2018).

R2: We thank the reviewer for bringing this to our attention. We have updated the discussion in the draft to reflect this relationship to prior work. Although this diminishes the algorithmic novelty of our approach, we actually think that this strengthens the impact of this particular paper. There are two components to this, the algorithmic novelty, and the experimental finding.

Experimental Findings relative to LOGAN: LOGAN studies training a detector on generated samples on CIFAR10 as we do in our Image GANs section and finds: “In Fig. 7, we plot the accuracy results for both settings, showing that the attack fails with both datasets when the attacker has only test set knowledge, performing no better than random guessing.” So the verdict on these black-box detector attacks from LOGAN is that they do not work. This stands in contrast to our findings, which is that in some settings, and particularly on genomic data, they do work very well. One reason is that since the LOGAN paper in 2018, our understanding of how to properly evaluate MIAs has evolved significantly since https://arxiv.org/abs/2112.03570 — rather than evaluating the accuracy of the MIA as in LOGAN, recent work including ours focusing on plotting the full ROC curve, and evaluating the TPR at low fixed FPR, which corresponds better to actual privacy risk for a small subgroup. By this metric on genomic data, the ADIS attack achieves a 10x improvement over the random baseline in some settings! On CIFAR10 our findings match LOGAN; although we do beat the random baseline at low FPRs for some detector-based methods, our improvements are modest, and worse than distance-based methods. Since prior work concludes detector based attacks are not effective, the fact that our work shows they work very well on tabular data adds importance nuance.

Novelty relative to LOGAN: In addition to the important experimental findings above, relative to the LOGAN paper, we introduce the ADIS variant of the detector based attack, which performs much better in practice on genomic data. We also provide theoretical analysis of the detector success in Theorem 1 and in Equation 1, which may provide some insight on why the detector-based attacks perform worse in certain settings. Finally, we evaluate on new types of GAN architectures: BigGAN, Progressive GAN, and ContraGAN, and find differences in attack success across architectures.

We thank the reviewer for their comments, we will address all minor comments and typos found, and update the related work to more accurately reflect LOGAN 2018, and GAN-leaks, both of which we do already cite. In the next comment we discuss the major points raised.

Minor Comments:

Q1: Many plots in Figure 1 and Figure 2 show that the proposed approaches can perform worse than random guesses in some settings (e.g., Figure 1a, 1b, 1d, 1e, 1f, 1g, 1j, 1k). Do you have explanations for that?

R1: On the genomic data, only the one-way distance attack really seems to perform worse than random guessing.

Recall that the one-way distance attack thresholds based on the distance between $x$ and the generated samples. We have added discussion of these results to Section 5.2: "Interestingly, the only method that consistently fails to beat the random baseline (AUC $< .5$) in all settings is the One-way distance-based attack. Paired with the moderate success of two-way distance attacks, this shows that for training points while the relative difference between their distance to generated samples and their distance to test samples is larger than the difference in distances for test points, but the actual distance to the generated samples alone is not uniformly larger."

Q2: It claims that "... ADIS consistently outperforms the Detector ...". This is not the case in Figure 1j.

R2: We have updated the language to reflect that occasionally the detector outperforms ADIS.

Q3: Because many of the lines in Figure 1 and 2 have significant overlap and the rankings of the methods are not consistent across the entire range of FPR, it would be more convincing to conduct the experiments multiple times and report the average and std of the scores.

R3: All reported results are averaged over 11 training runs, and in Figure $1$ we report the AUC $\pm$ the standard deviation in the legend. We've added this to the caption of the figure to make this more clear. We also added table $7$ to the Appendix that also contains the average AUCs and standard deviations.

We thank the reviewer for their excellent suggestions. We have run a new battery of experiments to address their main critiques, which we feel has strengthened the paper. We first discuss the major comments, and then the minor ones.

Major Comments:

Q1: How would the proposed attack perform if the distribution the adversary has access to is shifted away from the true distribution $\mathcal{P}$?

R1: We note that although almost all prior work on MIAs assumes that adversary has access to the true distribution P, we agree that in some situations where data access is very limited for privacy reasons, it is reasonable to assume the adversary only has access to a proxy distribution. We’ve added new experiments evaluating how the attack success of our detector attack changes as the proxy distribution that the adversary has access to shifts away from $\mathcal{P}$. We find that under both models of distribution shift that we study, as the distribution shifts way from $\mathcal{P}$, attack success decreases. Surprisingly we find that attack success decreases less than we might have thought, showing that the detector is somewhat robust to distribution shift. We summarize theses results in the distribution shift section of the draft.

Q2: Why is it reasonable to assume that $G$ is a convex combination of $P$ and $T$, $G = \alpha \mathcal{P} + (1-\alpha)T$.

R2: First we remark that Equation 1 still holds for arbitrary $G$, which says we can bound the error of the detector attack by $W^1(G, T)$. Now Theorem $1$ attempts to sharpen this result by showing how even if $G$ is actually quite far from $T$, and $W^1(G, T)$ is quite large, if $G$ is a convex combination of $\mathcal{P}, T$, the attack is still approximately optimal. We are not arguing that in practice $G$ is actually a convex combination of $\mathcal{P}, T$. We are more arguing that under this simplified representation of the generator, the attack is approximately optimal. While we do not expect that generators trained in practice will be an exact mixture distribution, the convex combination has some merits: (i) it reflects the fact that if properly trained G learns the true distribution $\mathcal{P}$ a $1-\alpha$ fraction of the time. Generative models have been observed to memorize their training data (e.g. https://arxiv.org/abs/2301.13188) and so the $\alpha$ component represents this memorization probability. Without a simplified model of the generator, obtaining theoretical results on attack optimality seems out of reach. We have updated the draft to provide this discussion in Section $4.1$.

Minor Comments: Thank you for the found typos, we will make sure to fix all of the above prior to the camera ready, and we have fixed the typo with the lipschitz constant in the Appendix (it was correct in the main draft) thank you!

Thank you very much for your feedback, and the great suggestion about training a private GAN. We are incorporating all of the found typos to improve the readability of the paper, including unifying some notation between the draft and the Appendix, and adding more explanation of the Detector attack from the Appendix to the main draft.

Main Comment: The main question that I have is with respect on how protection measure such as differential privacy would impact the conclusions drawn from the experiments. It would be good if the authors could do additional experiments to evaluate this issue.

Response: In response to this suggestion, we trained two DP GANs, one as per https://arxiv.org/abs/1802.06739 which uses DP-SGD to privately update the discriminator. We track the privacy loss using the RDP accountant in tensorflow (https://github.com/tensorflow/privacy/blob/979748e09c416ea2d4f85e09b033aa9aa097ead2/tensorflow_privacy/privacy/analysis/compute_dp_sgd_privacy.py). We also trained a DP GAN using PATE-GAN (https://openreview.net/forum?id=S1zk9iRqF7) although this did not work as well as the DP-SGD GAN, since the sample splitting required resulted in either too few samples to train the target GANs, or too much noise when privately aggregating. As a result we report results for the DP-SGD GAN, at epsilon = 1.7 in the new DP GAN training section of the draft.

On 805 dbGaP the Detector attack achieves AUC of .57 against the non-private WGAN. Against the DP-WGAN trained using DP-SGD, the attack AUC is only .50, which is the same as random guessing. We note that the fact training with DP destroys MIA attack success isn't that surprising, as differential privacy provides theoretical defense guarantees against $*any MIA*.$ See Theorem 3.1 in https://arxiv.org/pdf/2010.12112.pdf which says that if the training algorithm is $(\epsilon, \delta)$-DP, then for any attack the $\text{TPR} \leq \text{FPR}+ (e^\epsilon - 1 + 2\delta)/(e^\epsilon + 1)$. The difficulty in use DP as a defense is that historically training private classifiers that achieve competitive accuracy has been challenging, let alone DP generative models. While this result is encouraging, we note that it is possible a more complete battery of GAN convergence/utility tests may find that our private GAN is less useful than the non-private GAN even at $805$ SNPs, and we anticipate difficulty scaling these results to higher dimensions. We leave this as an intriguing direction for future work!

Thank you very much for your comments.

Minor Comments: We will add the relevant citations on black-box MIAs, move the description of ADIS up to the main draft, and added a description of how the detector pipeline works, so that Figure 4 is not essential to read the paper, as it is in the Appendix. Train and test sizes for the detector attack were chosen so that there are enough training points to train (i) a target GAN that converges (ii) a Detector that is accurate on a test set and (iii) points left over to accurately evaluate the MIA attack success. Typically we need a few thousand points each to train the generative models and the detector, and then at least $500$ points to evaluate MIA attack success.

Major Comments:

Q: Why is it reasonable to assume that $G$ is a convex combination of $P$ and $T$, $G = \alpha P + (1-\alpha)T$.

R: First we remark that Equation 1 still holds for arbitrary $G$, which says we can bound the error of the detector attack by $W^1(G, T)$. Now Theorem $1$ attempts to sharpen this result by showing how even if $G$ is actually quite far from $T$, and $W^1(G, T)$ is quite large, if $G$ is a convex combination of $P, T$, the attack is still approximately optimal. We are not arguing that in practice $G$ is actually a convex combination of $P$, $T$. We are more arguing that under this simplified representation of the generator, the attack is approximately optimal. While we do not expect that generators trained in practice will be an exact mixture distribution, the convex combination has some merits: (i) it reflects the fact that if properly trained G learns the true distribution P a $1-\alpha$ fraction of the time. Generative models have been observed to memorize their training data (e.g. https://arxiv.org/abs/2301.13188) and so the $\alpha$ component represents this memorization probability. Without a simplified model of the generator, obtaining theoretical results on attack optimality seems out of reach. We have updated the draft to provide this discussion in Section $4.1$.

We have also made additional improvements to the paper during the rebuttal period, in response to the other reviewers, including: (i) an investigation of attack success when the adversary does not have access to reference samples, and instead must use samples from a shifted distribution (ii) the impact of training a DP GAN on MIA attack success