Fragments to Facts: Partial-Information Fragment Inference from LLMs

Classifier is considered as the baseline for experiments. Are there any other methods in the literature that you can make the comparison with? / The paper should compare its attack success rate against existing MIA or extraction attacks on the same dataset/model.

This is an excellent question / comment. To our knowledge, no directly comparable baselines operate under the same weak adversarial assumption as PIFI. Traditional extraction and membership inference attacks require full, ordered samples and much more information (e.g., around 200 tokens), whereas PIFI only assumes access to a small set of unordered fragments (approximately 20). For this reason, we include a classifier -- leveraging ground truth labels -- as a strong, data-aware baseline for performance, against which we compare our data-blind methods (LR-Attack and PRISM). Although we reference likelihood ratio–based methods (e.g., Carlini et al. 2022a) for context, those approaches target a fundamentally different threat model. For instance, Carlini et al. report a 1.4% TPR at 0.1% FPR using 256 shadow models.

While their success rates are in a similar range, direct comparisons are challenging due to significantly differing assumptions. Our goal is to demonstrate that successful extraction is possible even under much weaker assumptions, rather than to claim that PIFI is inherently stronger. We agree that further contextualization alongside prior MIA work would be valuable and will expand on this discussion in the revised paper.

(Related question from Reviewer zpv2) Why do we observe that legal setting attacks are more challenging?

We’d be happy to expand on the legal results in the revised paper, drawing from Appendix B.3 (T6). In the legal summarization task, target fragments often include more common language, including crimes or legal terms that are likely present in general pre-training data. This dilutes fragment-specific signals compared to the medical setting, where terms like “daunorubicin” are more domain-specific and comparatively rare. As a result, attack performance is lower in the legal domain, though still above chance. Interestingly, LR-Attack performs best here, likely due to its heightened sensitivity to rare fragments, which helps in a setting where most targets are otherwise common.

But how do we know the model isn’t just capturing domain-wide patterns (like “cold” usually co-occurring with “cough”) rather than genuine memorization at the individual level? / Clarify whether the proposed algorithm leverages the frequency of different conditions?

These are good questions. Our experiments indicate that the attack isn’t simply leveraging generic domain-wide associations but is also capturing signals specific to individual samples. For example, PRISM’s performance nearly matches that of a data-aware classifier with ground-truth labels. If the model were only exploiting common co-occurrence patterns (e.g., “cold” with “cough”), then the target and shadow models would yield very similar probabilities, and the likelihood ratio would not offer the discriminative power we observe. Moreover, our memorization tests (see Appendix Section E, Table 6) further confirm that fine-tuning leads to the memorization of individual samples.

At the same time, our approach does take advantage of statistical co-occurrence: if an adversary knows that a record contains a fragment like “hypertension,” and the fine-tuned model was trained on data where “hypertension” frequently co-occurs with “osteoporosis,” the model assigns a higher probability to that associated fragment. LR-Attack captures this by comparing the target and shadow model probabilities, while PRISM refines the inference by incorporating a general “world” probability to discount associations that are common across the domain. This combination of mechanisms ensures that our attack can distinguish genuine individual-level memorization from mere generic co-occurrence patterns.

(Related question from Reviewer zpv2) In Sec. 4.2, how unique is the given fragment s?

In the medical setting, we targeted 4,302 fragments, 1,034 of which were unique. Common fragments included "pain" (124), "fever" (69), "shortness of breath" (55), "diarrhea" (47), and "chest pain" (44). However, 75% of fragments appeared fewer than five times, and 47% were targeted only once, indicating that most were highly specific. For instance, single-targeted examples include "Vincristine," "colitis," "daunorubicin," "Naprosyn," "Xalatan," and "lumbar spinal stenosis."

...assumptions about prior distributions may not always hold in real-world...

We appreciate the comment. PRISM does make an assumption about the usefulness of a prior world model probability in adjusting the likelihood ratio (assumed to be correlated with sample membership), a premise grounded in statistical principles (see Section 4.2) and supported by prior work [e.g., Carlini et al. 2022a]. Additionally, we have empirically validated this assumption through controlled experiments with a fully specified trigram model (Appendix C) and large-scale LLMs in both medical and legal settings (Section 7, Appendix B). In practice, PRISM reduces false positives compared to LR-Attack, showing that even an approximate prior can provide meaningful signal. Moreover, PRISM performs competitively with, and sometimes better than, a data-aware classifier baseline, suggesting its potential to generalize beyond our setup.

Missing citations.

We discuss all three of these key prior works in our paper already. Carlini et al. (2022a) is cited when introducing membership inference and extraction attacks and motivating our LR-Attack. Nasr et al. (2023) is referenced in our review of scalable extraction methods, supporting our focus on fragment-based extraction in fine-tuned models. Shokri et al. (2016) is acknowledged as foundational in membership inference and helps contextualize our extension of prior threat models. We’d be happy to highlight these works more prominently in the revision.

...deeper analysis of why certain fragments are more extractable...

This is a great question. As noted in our response to Reviewer zpv2, for the medical setting, we targeted 1,034 unique fragments. 47% of those fragments occurred only once. Below, we provide results for the subset of fragments that occur only once, compared to the subset that occurred multiple times:

Results on target fragments occurring only once:

Results on target fragments occurring multiple times:

This result highlights how incorporating the world_model prob with PRISM significantly improves the sensitivity to more common medical terms, whereas the LR-Attack is very sensitive to rare fragments. We’ll make sure to highlight this result in the revised version of our paper, and add a more in-depth discussion.

How does PIFI perform against differential privacy, etc.?

Thank you, this is an excellent question. Please see our response to Reviewer m6fY, where we report results on our attack under differential privacy.

How to differentiate between ID and OOD data?

The LR-Attack could be susceptible to ID vs. OOD data, as you suggest. This motivated the PRISM approach, which uses the world model probability as a prior to adjust the likelihood ratio (target versus shadow), discounting fragments that are common in general knowledge. In principle (see Appendix C for the Synthetic data results, validating this heuristic), PRISM ensures that only fragments with an unusually high likelihood -- beyond what is expected from generic associations -- are flagged as memorized from the training data.

How does fine-tuning strategy impact effectiveness?

Our paper explores this along two dimensions: number of fine-tuning epochs and fine-tuning strategy (full fine-tuning vs. LoRA). We find that more epochs increase TPR at fixed low FPRs, showing that repeated exposure heightens memorization and vulnerability (Section 7.2). While LoRA models are less vulnerable than fully fine-tuned ones, they still leak information, indicating that parameter-efficient methods reduce but don't eliminate privacy risks (Section 7.3). We’re happy to emphasize these findings more in the revision.

How does exposure affect attack performance?

Our experiments show that increasing the number of fine-tuning epochs consistently increases the risk of fragment inference, as demonstrated in Section 7.2. We have observed that attack performance improves continuously with additional epochs. We will add a more in depth analysis to the revised version of the paper, where we step across epochs in greater granularity.

Practical implications for deployed LLMs?

Our work shows that fine-tuning increases memorization risks, even when only partial fragments are exposed. Organizations must weigh the performance gains of fine-tuning against the risk of fragment leakage. Mitigations include using privacy-preserving techniques (e.g., differential privacy), restricting model access, and monitoring for leaks. Ultimately, domain experts should guide these privacy-performance trade-offs through informed risk assessments.

In Sec. 4.2, how unique is the given fragment s?

Please see our response to Reviewer rGXA, where we give exact details on uniqueness.

Why does LR-Attack and PRISM sometimes outperform classifier baseline?

This is a great question and we’ll include this discussion in the revised paper — while the classifier baseline benefits from ground-truth labels and can, in theory, exploit signals more effectively, its performance may degrade due to overfitting, especially when fragment distributions are highly variable. In contrast, LR-Attack and PRISM use data-blind decision rules based on probability ratios (and a Bayesian update in PRISM), without learning from labeled data. This makes them more robust to overfitting. As a result, in settings with high distributional variation, these rigid decision rules can sometimes outperform the classifier.

Attacks transferable to other open- or closed-source LLMs?

We evaluated our attack on a range of open models and found it consistently effective across architectures and parameter scales (see Appendix B1, Figure 7). The transferability to closed models, however, is currently limited by the need for echo functionality on logprobs — previously available in some APIs (e.g., OpenAI’s) but now removed due to security concerns. Future work could explore alternatives, such as methods proposed by Finlayson et al. (2024) to obtain logprobs from closed models.

Is this threshold in algorithm 1 different for different samples in a given dataset?

Good question, we’ll clarify this in the revised paper. No, the threshold is fixed for each strategy based on the desired sensitivity, similar to prior membership inference attacks [Carlini et al. 2022a; Duan et al. 2024]. For instance, on a converged Llama 8B model, the PRISM threshold is 0.081, and the LR-Attack threshold is 1.78 when targeting a 2% FPR.

Test on models fine-tuned with privacy constraints?

Thank you for the excellent suggestion. Please see our response to Reviewer m6fY, where we evaluate our attack under a differential privacy.

Clarify whether the proposed algorithm leverages the frequency of different conditions?

Thank you for this observation. Please see our response to Reviewer rGXA, where we discuss frequency / co-occurrence

…test using small datasets…

While the test datasets are relatively small (312 samples for medical and 235 for legal), our attack evaluated over 4,302 target fragments, providing strong evidence for the feasibility of the PIFI threat model. We plan to expand our evaluation to larger datasets in future work to further validate these results.

…what is utility of the proposed attack on larger LLMs?

Thank you for the suggestion; we have evaluated our attack on a 70B-parameter Llama-3.3 model fine-tuned for 10 epochs, which is the largest model we could finetune given our compute constraints. The results below show that the attack remains effective even at this larger scale:

We will include this result in our revisions.

Why do we observe that legal setting attacks are more challenging?

Please see our response to Reviewer rGXA, where we discuss the legal setting in detail.

…do not provide LightGBM accuracies on the classification task + range of performant classifiers...

We'd be happy to report accuracies for the LightGBM classifier baseline + other classifier baselines (e.g., XGBoost, FT-Transformer etc.). For example, the Llama-3 8B model attack had accuracy (0.77) and F1 score (0.25); however, we believe in our focus on ROC AUC and TPR (which we do report on the classifier baseline) as it reflects the utility of our LR-Attack and PRISM methods in low-FPR regimes (consistent with [Carlini et al. 2022a]).

…Figures 4-5, LR-Attack obtains an AUC lower than random guess?

This typically means the scoring function is inversely correlated with true membership, and happens when assumptions about target vs. shadow model behavior break down, such as when both assign similar probabilities or the fragment is very common, skewing the ratio. This is one motivation for PRISM.

…ensemble of large open-sourced models as the world model?

Great suggestion. We have now experimented with using DeepSeek-v3/r1 world probabilities (averaged), and found that this higher-quality model did improve performance. These results suggest that adding more high-quality open-source models to the world model ensemble could further enhance our approach.

Below is the performance of the Llama 3 8B model with DeepSeek world probabilities (10 epochs):

…discuss whether the attacks work on general-purpose LLMs

Thank you for raising this important point. Our work specifically focuses on sensitive domains (e.g., hospitals or legal) where models are adapted on private data, and fine-tuning is often necessary because organizations cannot or do not wish to train a full-scale foundational model on sensitive data. Consequently, the vulnerabilities we expose are most pertinent to these fine-tuned models. Additionally, prior work (e.g., [Duan et al. 2024]) shows that membership inference attacks on foundation models are significantly more challenging. as such they’re trained on broad datasets and memorize less. Additionally, our method depends on building a shadow model, a step that is often impractical for large, non-fine-tuned models due to their scale and complexity. We also note, though, that our fine-tuned models remain performant on general-purpose benchmarks like MMLU, suggesting they can be used in manner similar to other general-purpose models, as discussed further in our response to reviewer wf5m.

…compare its attack success rate against existing MIA or extraction attacks...

Thank you for the comment; some other reviewers brought up similar points. Please see our response to Reviewer rGXA, where we discuss MIA performance relative to our PIFI threat model.

…unclear whether the paper accounts for trivial guesses… does attack outperform a "most likely completion" heuristic?

Thank you for the suggestion. To clarify, by “most likely completion” do you mean a baseline that considers a greedy or Monte Carlo generation of the next token? We’d happy to include another baseline, please clarify and we will add it in the revised paper.

…explicitly evaluate common privacy-preserving techniques.

This is a great suggestion. We have added experiments evaluating differentially private (DP) finetuning on the Llama 3.2 3B Instruct model using sample-level DP (via the dp-transformers package with ε = [0.3, 1, 3, 9, 27] over 10 epochs). We include only the ε = 3 results here due to space constraints, but will add full results in the revised paper. This table shows task performance for the base model, the DP fine-tuned (FT) model, and the non-private FT model:

DP FT slightly improves performance over the base model but does not reach the performance of non-private FT, which aligns with expectations [Lukas et al. 2023].

We further evaluated the PIFI threat model and attacks on the DP FT LLM:

Here, DP fine-tuning reduces the success of the LR attack (0.9% TPR at 2% FPR), indicating the DP mechanism protects against this attack. However, both the classifier baseline and PRISM still achieve roughly twice the TPR at fixed low FPRs. We consider a potential explanation:

Sample level DP guarantees for LLM finetuning ensure $\sum_{i=1}^T \ell_i(x_{1:i}) \leq \epsilon$ where $\ell_i(x_{1:i}) = \log \frac{\Pr(x_i \mid x_{1:i-1}, D)}{\Pr(x_i \mid x_{1:i-1}, D')}$ (Yu et al., 2021). However, note that this guarantee of privacy loss can be unevenly distributed: for example, consider a two-token case with $\frac{\Pr(x_1 \mid D)}{\Pr(x_1 \mid D')} = e^\epsilon$ and $\frac{\Pr(x_2 \mid x_1, D)}{\Pr(x_2 \mid x_1, D')} = 1$; the total privacy loss $L(x_1, x_2) = \epsilon + 0 = \epsilon$, satisfying the guarantee, but the entire privacy budget is focused on a single token! In summary, with standard DP finetuning individual tokens can incur nearly $\epsilon$ loss if others compensate. Further analysis + other DP approaches would be promising for future work.

…PRISM attack is not completely clear to me…more intuitive explanation would help

Thank you, we’ll include this discussion in the revised paper if helpful. PRISM builds on the standard LR attack by asking: how surprising is it that the target fragment appears, given what we expect in general? LR attack compares the target model (which has seen the sample) and shadow model (which hasn’t) probabilities to detect memorization of a fragment. However, this ratio can be high simply because a fragment is common in a domain. PRISM corrects by incorporating a “world model” that estimates the general likelihood of a fragment. In essence, PRISM performs a Bayesian update — it adjusts LR score with a prior that reflects how likely the fragment is in any sample. If "smoker" frequently co-occurs with "osteoarthritis," a high LR score might not indicate memorization; by using the world model probability, PRISM tempers the score -- if "smoker" is common overall, the adjusted score is lower, reducing FPs. We validate this with a small, controlled synthetic setup (Appendix C), where PRISM outperformed the basic LR attack.

Concerns about practical impacts, given the “modest” results.

We appreciate the concern about practical impact. While a 10% TPR at 2-5% FPR may seem modest, it can still pose a significant privacy threat (e.g., it equates to an attacker being correct 4 out of 5 times), especially scaled to thousands of individuals. Moreover, membership inference attacks also face challenges in achieving high TPRs, yet are considered to be a significant privacy concern (see e.g., Duan et al. 2024)

The practicality and prevalence of the assumed attacker capabilities remains speculative. How plausible is it for an attacker to reliably obtain the fragment sets you assume (e.g., 10–30 specific keywords)?

We agree that the threat model’s practical impact depends on how feasible it is for an attacker to gather necessary fragments. In domains like healthcare, such fragment data is increasingly accessible. A 2020 study [Seh et al., 2020] found that healthcare data breaches between 2005 and 2019 affected over 249 million individuals. Additionally, self-health data collection (e.g., Lupton, 2016) and sharing on social media make such fragments more easily available. Given something as simple as a public record or leaked insurance info, combined with a reliable set of fragments derived from otherwise benign medical data shared on social media, an attacker could construct an effective set S of tokens to enable targeted attacks, as demonstrated by our PIFI threat model. We will elaborate on this in the revised paper.

Generalization results on standard NLP benchmarks (MMLU) pre- and post-finetuning

Thank you for the suggestion. We ran preliminary utility evaluations (MMLU). The Llama-3-8B model fine-tuned for 10 epochs reached an average MMLU score of 0.565, compared to 0.487 after one epoch and 0.413 for the baseline model. In medical areas (e.g., clinical knowledge, medical genetics), fine-tuning improved performance over a non-fine-tuned baseline, suggesting the model remains capable and general. Note that further fine-tuned models can outperform generalist models on some benchmarks due to “better behavior” (they naturally follow a stricter format), which DeepEval docs specifically mention as a limitation of MMLU.

We will add exhaustive MMLU results to our revised paper.

Add qualitative examples of successful and failed attacks to aid interpretability.

Here are several examples of successful and failed attacks under PIFI from our data. We will update the paper with several of these in accordance with your recommendation.

1. An attack with the fragments “anxiety disorder, arthritis, Morton's neuromas, migraines” infers that an individual has hypothyroidism.
2. An attack with the fragments “shortness of breath, Coumadin, lightheadedness, chest pain, Cardizem, pain, vertigo” infers that an individual has atrial fibrillation.
3. An attack with the fragments “toxicity, breast cancer, Ixempra, tumor, neuropathy, Avastin, cancer, Faslodex, Zometa, ixabepilone” infers that an individual takes Aromasin.

Failed Inferences:

1. An attack with the fragments “cramping, trauma, pain, shock, numbness” fails to infer that an individual takes Naprosyn.
2. An attack with the fragments “diabetes, redness, swelling, itchiness, pain, skin infection” fails to infer that an individual also has cellulitis.

False Positives:

1. An attack with the fragments “myelofibrosis, swelling, diarrhea, hydroxyurea, J A K, steroids, polycythemia vera, lenalidomide, smoking, smoke, pain, numbness” incorrectly infers that an individual takes Prednisone.
2. An attack with the fragments “lung disease, shortness of breath, pneumonia, oxygen” incorrectly infers that an individual has COPD.

Could the attack work without knowing the exact fragments from the sample, i.e., based on approximate knowledge?

Yes, the attack remains effective when the attacker’s knowledge is approximate. In our ablation studies (see Appendix, Figure 9), we show that replacing 25-75% of true fragments with random unrelated ones leads to only modest performance drops. This indicates the method is robust even when the adversary’s information is imperfect.

How sensitive are the results to the prompt(s) template used?

We experimented with various prompt templates (e.g., “Consider a patient’s medical summary includes…”; “Suppose a patient’s medical summary includes conditions such”; etc.) and found that attack performance did not change substantially. While format influences raw output probabilities slightly, the overall ranking of candidate fragments (and hence the attack’s effectiveness) remains consistent. We will add a sensitivity analysis in the revised paper.