Privacy Attacks on Image AutoRegressive Models

The method is primarily constructed from components of prev. work.

Beyond the proposed method, our contributions are:

1. First empirical privacy leakage evaluation of IARs. We develop the strongest model-specific attacks, and perform comprehensive analysis across publicly available models.

2. Privacy-utility trade-off. We show IARs are fast and performant, but substantially less private. We highlight that DMs are comparably performant, while leaking significantly less information about the training data.

The authors propose an attack specifically tailored for IARs, it is misleading to claim that IARs inherently have higher privacy leakage. It would be more accurate to state that using the tailored attack, higher privacy leakage is observed in IARs compared to DMs. Privacy leakage measurements should reflect an upper bound across possible methods, not just the results from a specific, targeted attack.

We do not use IAR-tailored MIAs/DI against DMs–they are not applicable to DMs. We use SOTA DM-specific attacks against DMs, thus the observed privacy leakage for DMs and IARs is an empirical upper bound across possible methods, since we use the strongest possible attacks. Effectively, our claims hold. We improved the wording in the manuscript.

The method [...] lacks sufficient emphasis on the differences from previous work. [...] In the introduction: [...] a citation to CLiD should be included.

We now include the citation to CLiD (Zhai et al. 2024) in the introduction.

Our MIA for VAR/RAR provides the following innovations over CLiD:

1. Difference between logits, not model loss. CLiD uses $\mathcal{L}(x,c)-\mathcal{L}(x,c_{null})$. Since MIAs we use work with logits, we compute $p(x|c)-p(x|c_{null})$.

2. Parameter-free method. CLiD needs a sweep through a hyperparameter $\alpha$ to achieve its high performance, as well as Robust-Scaler to stabilize the MIA signal. We provide a more generalized approach.

2. In Sec. 5.3: “[...] builds on elements of data extraction attacks [...].” It is not clearly stated which parts align with previous work. [...] Fixing and knowing the first i tokens mirrors the setting in LLMs.

Our attack consists of the following:

1. Efficient candidates selection. We do not simply generate millions of images. Instead, we identify promising images for further generation. This improves over Carlini et al., 2023.

2. Fixing the prefix. We directly follow the approach by Carlini et al., 2021.

3. Generating from the prefix. We follow (Carlini et al., 2021), and use greedy sampling for VAR/RAR starting from the prefix. For MAR, we do not alter the generation process.

4. Final assessment. In contrast to LLM extraction (Carlini et al., 2021), we focus on the images, not sequences of tokens. Samples we extract do not match the training samples in the token space; instead, we classify a sample as extracted in the image domain. To this end, we use SSCD (Pizzi et al., 2022), following Wen et al., 2024.

[...] the unclear presentation of prior work makes the paper’s contributions less apparent. [...] the attacking method is not particularly innovative [...].

Our methods include novel designs:

1. Improved MIA against VAR/RAR. While we are building on CLiD, the adaptation to IARs is non-trivial. We improve the performance by up to 69%.

2. MIA specifically tailored against MAR. We exploit model vulnerabilities (the DM module, training specifics) to boost the performance of our attack.

3. Efficient data extraction. While we build on previous work, we address two drawbacks: 1) focus in the sequence space, 2) high cost. We suggest an improved method enabling quicker, more successful extraction from the IARs, extracting up to 698 images.

4. Improved DI. We improve feature aggregation (replacing scoring function) and the feature extraction pipeline. We achieve an improvement of up to over 90% compared to the baseline.

The experimental section is solid, but [...] it would be beneficial to include more insights.

We appreciate that the Reviewer considers our experiments solid. We agree that insights would improve our work. In the answer to the Reviewer RYDX we provide intuitions to why IARs are more vulnerable than DMs. We add them to the paper.

The approach in Sec. 5.1 feels like parameter tuning [...] rather than offering deeper methodological innovations.

Our main innovation stems from modifying the DM module of MAR to increase the leakage. We are not aware of any prior work that modifies the inference stage of an AR model to design a more potent attack.

[...] unclear articulation of the contributions gives the impression of claiming a larger contribution than is warranted.

Do our above answers articulate our contributions clearly enough? We are happy to incorporate any additional feedback.

We thank the Reviewer for the feedback.

Attacks tailored for LLMs, DMs, IARs

The proposed [MIA and DI] are specifically tailored to exploit [ARs], raising questions about whether the comparisons with DMs are conducted under balanced conditions. [Selecting baselines derived] from language model research raises questions about their suitability for [DMs]. [...] The experimental setup appears to favor IARs. [...] The baselines [...] may not [be] rigorous benchmarks for [DMs].

We use MIAs that are tailored to unique characteristics of a given model, including DMs (see Tab. 13, App. F). For DMs we use the strongest MIA available at the time of writing the paper, namely CLiD (Zhai et al., 2024), and we do not use LLM/IAR-specific MIAs against DMs. Similarly, for IARs we build on the strongest MIAs that are suitable for the AR architecture of these models. Overall, we use the strongest attacks for a given model, following [1].

[...] developing equally specialized attacks for [DMs] or applying the same, more generic attack methods [...]. The authors should clarify why LLM-attack approaches were selected over methods designed for DMs.

We do not use LLM/IAR-specific MIAs/DI against DMs. Instead, we use SOTA DM-specific attacks, as explained in the previous answer.

Why [MIAs and DI] tailored [for ARs show that] IARs are more susceptible to privacy breaches than DMs. This design choice seems to favor ARs.

To perform DI against DMs, we use CDI (Dubiński et al., 2024)–a method explicitly created for DMs. For IARs we build upon LLM DI (Maini et al., 2024), which we adapt to IAR specifics (see Sec. 5.2) to ensure a fair comparison.

Different MIA strategies are tailored for MAR and for VAR/RAR [...] (Should?) a single, unified MIA strategy be used [...]

Empirical privacy leakage analysis should be carried out with respect to the worst case [1], thus the strongest known attack. Unified MIA for all IARs would not allow such comparison.

[...] whether the performance differences are due to inherent model vulnerabilities or inconsistencies in the attack strategies [...] These strategies are combined into a single tab. without clearly distinguishing how each approach is evaluated. [It is unclear if] the metrics for MAR should be compared directly to those for VAR/RAR.

MAR and V/RAR are distinct in design, inference, and training–and the attacks differ too, as they exploit unique vulnerabilities of the models. Some design choices allow for stronger attacks; it is reflected in the results (Tab. 1). Evaluation protocol stays consistent; the attacks vary.

Baselines

The authors select certain baselines [...], but some vital experimental settings are not explicitly detailed. For instance, the term “baseline” appears repeatedly in [Tab. 1-3] without fully describing the precise settings, assumptions, and hyperparameters [...]

All MIAs assume gray-box access to the model, i.e., output and model loss. Some MIAs for DMs require white-box access to the model. We expanded the description provided in App. B. For all MIAs, we use the default hyperparameters from the respective MIAs. Following the literature we report the TPR@FPR=1% only for the best hyperparameter in Table 9, 11. In Table 1, for baseline and our methods, we report the best MIA for each model, as we strive to compare only the strongest attacks.

​​In Tab. 1-3, “baseline” denotes a naive use of LLM-tailored MIAs and DI to attack IARs. We revise App. B to include experimental details, relevant configurations and parameter choices.

A clearer justification of baseline [...]

For IARs we use MIAs and DI for LLMs as “Baseline” (Table 1-3) as we can directly apply them to IARs, and no IAR-specific attacks exist in prev. work.

Other

Methodological descriptions (e.g., the integration of CLiD into MIAs) and the explicit definition of experimental settings [...] require improvement. [For instance] “incorporate [this] into our MIAs by building on CLiD”, [...] how the tailored MIA approach is employed within the DI [...].

We improved the clarity of those aspects. For details, see answer to the Reviewer ZL49.

Fig. 1 and 2

We improve Fig. 1,2 accordingly.

TPR@FPR

We clarify what TPR@FPR (True Positive Rate at False Positive Rate) stands for in the abstract section.

Please elaborate on the unique contributions or key design elements of your methods. The paper lacks a clear theoretical contribution beyond statements such as “incorporate CLiD in our methods” (line 224) and “simply summing” (line 289).

Please refer to the answer to the Rev. 1YbK.

Tab. 9

We thank the Reviewer for the suggestion. We will move Tab. 9 to the main text if accepted, given the 1 extra page allowed.

Ref.:

[1] Carlini et al., Extracting Training Data from [DMs], USENIX 2023.

Emerging DMs (e.g. flow-matching)

We extend evaluation to 1) latent flow matching (LFM) (Dao et al., 2023), 2) sparse DM (DiT-MoE), 3) flow matching transformer (SiT) (Ma et al., 2024), We report:

We observe that while LFM and DiT-MoE display privacy leakage comparable to other DMs, SiT is more vulnerable. However, the leakage is still smaller than for IARs.

Factors vs MIA efficacy

We compare training duration, model size, and a binary “Is the model an IAR?” factor against MIA and DI performance metrics, reporting Pearson’s correlation:

1. Duration influences MIA/DI against DMs the most.
2. Model size influences leakage in IARs more than in DMs.
3. Is IAR factor has the strongest correlation to DI performance.

We cannot isolate duplicates as a factor without re-training models from scratch. However, all evaluated models (DMs/IARs) are trained on ImageNet-1k (same duplicates).

Temper claim

We adjust our claims to be more precise and state that the privacy risks for IARs are empirically more severe than DMs, given the state of current privacy attacks.

Dataset - scalability

We acknowledge this concern, however:

1. IARs trained on >1M images (Han et al., 2024) do not specify their training data. Thus, a sound MIA/DI evaluation is impossible, as they need a) train data (members), b) IID non-members. Failure to satisfy b) leads to dataset detection (Das et al., 2024), and without a) we have no data to perform the attacks.

2. These are far from “toy models”, as ImageNet-1k allows for high quality, diverse generation.

3. The dataset is widely used as a benchmark; most cutting-edge DMs and IARs are trained on it.

We believe our setting is useful for practitioners, while ensuring full methodological correctness.

Novelty

We gladly clarify the novelty:

1. First empirical privacy leakage evaluation of IARs. We employ the strongest model-specific attacks, and perform comprehensive analysis across publicly available models.

2. First IAR-tailored MIA. We combine LLM-like properties of IARs with ideas from attacks against DMs to craft our MIA, improving TPR@FPR=1% by up to 69% over the naive baseline.

3. First IAR-specific DI. We decrease the number of samples needed for DI for IARs by up to 90% compared to baseline.

4. Successful extraction attack. We are the first to recover training data from IARs, leaking up to 698 images.

5. Privacy-utility trade-off. IARs are fast, but less private. Next, we explain why:

What makes IARs leakage different? DMs vs. IARs

Inherent causes for higher privacy leakage in IARs:

1. Access to p(x) boosts MIA (Zarifzadeh et al., 2024). DMs do not expose it at inference–they learn to transform N(0,I) to data. IARs are trained to output p(x) directly. It is reflected in distinct MIA designs for DMs and IARs–the former exploit the noise, the latter–p(x), via logits. MAR does not output p(x), and is less prone to MIA (Tab. 1).

2. AR training exposes IARs to more data per update. RAR outputs 256 distinct sequences to predict a sample. DMs operate only on a single, noised image. At fixed training duration, leakage is stronger for IARs. VAR outputs 10 sequences of tokens, and is less prone than RAR to MIA (e.g., VAR-d20 vs. RAR-L of similar size).

3. Multiple independent signals amplify leakage. Each token predicted by IARs leaks a unique signal, as it is generated from a different prefix. DMs’ outputs are tightly correlated, and the aggregated signal is weaker.

Architectural design choices for DMs and IARs differ for every model; it makes single-point conclusions unsound.

Limited IARs. Real world: text-to-img models, on messy, large datasets

Due to the reasons highlighted in our answer on scalability, we cannot soundly evaluate larger models due to lack of access to train data and lack of IID non-members.

Still, we added experiments on VAR-CLIP (Zhang et al., 2024), a text-to-img VAR trained on a captioned ImageNet-1k, reporting:

We compare VAR-CLIP to VAR-d16, as these models have the same size (300M). Notably, text-to-img model exhibits greater privacy leakage, on the level of a model twice as big, VAR-d20.

Retelling DMs story

Our work goes beyond mirroring findings in DMs. We introduce the strongest privacy attacks available, and evaluate many public SOTA models. We empirically show that IARs are significantly more vulnerable than DMs; we explain why in the updated version of the paper. Thereby, our paper offers a valuable insight on the privacy of novel generative models.

We thank the Reviewer for the insightful comments. We address individual points below one by one:

[...] authors are claiming that IARs are now the gold standard for image generation, while it has not been so widely adopted.

We clarify that we position IARs as novel model family that can perform on par or slightly better than DMs according to the established benchmarks. Given this, we find investigating the privacy leakage of IARs at the early stage of adoption valuable for the community to support responsible adoption.

The proposed MIA/[DI] method is very succinctly explained in page 5. It would have been nice to have a more detailed explanation. [...] I have found that the description of the proposed MIA/DI method is too succinct [...]. An additional figure to explain, or equations, would have made things clearer.

To address the Reviewer's suggestion, we further expanded and clarified our methods in the revised manuscript, including visual diagrams and procedular steps. Here, we provide a summary :

1. MIA for VAR/RAR: For IARs the output token probabilities are additionally conditioned, e.g., on class labels, yielding $p(x|c)$. We follow CLiD (Zhai et al., 2024) to exploit the conditional overfitting of IARs and provide $p(x|c) - p(x|c_{null})$ as input to MIAs methods (described in more detail in App. B).

2. MIA for MAR: we select the optimal diffusion timestep and mask ratio, perform multiple inferences (to limit the variance of the diffusion process), and obtain per-token losses per pass. We average them across inferences, and input these per-token losses to MIAs (App. B). We use losses, as logits are unavailable for MAR which outputs continuous tokens.

3. DI improvement: LLM DI [2] uses a scoring function $s$ to aggregate signals from the features. We note that this increases $P$ (number of samples required for DI), since a subset of samples is used to fit $s$. We replace $s$ with a summation of normalized features instead. Additionally, instead of using the original MIAs from [2], we substitute them with our improved versions (points 1. and 2 above).

MIA needs members and non members from the same distribution. The authors do not detail how the non-members are sampled, which is very important in practice if one wants to do an MIA in a realistic setting.

We agree members and non-members have to be from the same distribution for the MIA/DI results to be sound. In Sec. 4, we state “For MIA and DI, we take 10000 samples from the training set as members and also 10000 samples from the validation set as non-members.” (lines 235-237). Because the validation set of ImageNet-1k was selected randomly from the full dataset, members and non-members are IID, which satisfies the requirement.

“IARs can achieve better performance than their DM-based counterparts.”—> Its not that clear that IARs will take over the world

We fully understand the Reviewer’s concerns and tuned down this sentence to “IARs are an emerging competitor to DMs”.

The authors define memorization as verbatim memorization

We explore the worst-case memorization, following the setup from [1].

“We provide a potent DI method for IARs, which requires as few as 6 samples to assess dataset membership signal.”-> This depends on model size, overfitting etc.

Indeed, we presented the strongest result. Following the Reviewer's idea, we provide a comparison between two factors: model size and a binary “Is the model an IAR?” factor and $P$ (DI metric) as a Pearson’s correlation:

1. Model size influences leakage in DMs more than in IARs.

2. Is IAR is the factor with the strongest correlation to DI performance.

Should be clearer if its a realistic case and that comparison is done apple to apple in terms of FID compared to [DMs].

Fig. 1 (left) and Fig. 2 show direct comparison between DMs and IARs in terms of FID (y-axis), where IARs exhibit greater privacy leakage than DMs, for similar values of FID. For example, in Fig. 1 (left), we observe that VAR-d24 (second blue dot from the right) has a FID of ~2.0, but the TPR@FPR=1% for this model is ~22%. In comparison, SiT achieves FID of also ~2.0, while maintaining the MIA performance of ~6% TPR@FPR=1%. We acknowledge that we do not compare privacy leakage at a fixed FID, but we believe these plots serve as privacy-utility trade-off curves.

“Interestingly, we find that t = 500 is the most discriminative, differing from the findings for fullscale DMs, for which t = 100 gives the strongest signal.” —> no figure or table to refer to for these results?

We apologize for the imprecise formulation. We base the claim about fullscale DMs on [1]. We added the citation to the manuscript.

References:

[1] Carlini et al., Extracting Training Data from [DMs], USENIX 2023.

[2] Maini et al., LLM Dataset Inference [...], NeurIPS 2024.