Can We Infer Confidential Properties of Training Data from LLMs?

We sincerely thank the reviewer for taking the time to review our paper and provide thoughtful comments. We provide our responses to your concerns and questions below.

MIA leveraging word frequencies in literature (W1): Thank you for highlighting these related works on membership inference attacks (MIA). They complement our findings and help explain why word frequencies or n-grams are strong signals in the context of LLMs. Consistent with prior work, we also observe that loss-based scores—effective in MIA and property inference for earlier models like ConvNets and linear classifiers—are less informative for LLMs. We will incorporate this discussion in the revised version.

Details for perplexity attack baseline (Q1): The perplexity attack baseline follows the set-up in [1]. We have two hold-out dataset $S_0$ and $S_1$ where the property ratios are 0% and 100%, respectively. Then we calculate the average perplexity scores for each dataset for each model $f$, yielding a two-dimensional vector that serves as the shadow feature. The perplexity attack baseline defines the mapping from the model $f$ to this two-dimensional vector as the shadow feature function. These details are presented in the Appendix A.2 and we will move these details to the main paper for better understanding this baseline!

[1] A. Suri and D. Evans. Formalizing and estimating distribution inference risks, 2022.

More details about the keywords in word-frequency attack (Q2): Yes, we are happy to provide additional details on how keyword frequencies correlate with the property ratio in the fine-tuning dataset. Please see our response “Intuition of word-frequency signals on L230–231” in the reply to Reviewer nLBK. For example, when the target dataset has a higher Female ratio, the word "female" appears more frequently in the outputs. We also report the list of selected key words from the shadow model with the corresponding correlation coefficients with the ratio in that response.

Word-frequency attack needs more samples than BB generation attack (Q3): We would like to answer two questions separately.

1. We have studied the impact of query budget on the BB generation attack. As shown in Figure 7 (Appendix), the performance for LLaMA-1 and Pythia converges around 2k samples. Therefore, increasing the query budget to 100k is unlikely to significantly improve the results.

2. Why does word-frequency attack need more than $100k$ samples, much more than the BB generation attack? Unlike the BB generation attack, which directly predicts the property ratio, the word-frequency attack relies on the frequency of specific keywords—many of which occur much less frequently than the property ratio itself. For instance, when the target female ratio is 0.7, the word "female" appears in only 2.5% of samples (the female property can be indicated by other words too). A query budget of 2k introduces an estimation error of around 0.01, which may still allow the BB attack to accurately predict a ratio close to 0.7 (e.g., 0.69). However, this level of noise significantly disrupts the shadow model's ability to learn from small word-frequency variations—e.g., "female" frequencies are 0.016, 0.0219, and 0.025 for target ratios 0.3, 0.5, and 0.7, respectively. Many other informative keywords have even lower frequencies, making the attack highly sensitive to sample size. This necessitates a larger query budget to preserve the correlation signals needed for effective property inference.

Other confidential properties (Q4, W2): Thanks for bringing up the new practical examples. By considering a dataset of political news and the property of any political topic, the property inference from a news model that is finetuned on such dataset can be another practical scenario. We will add this discussion in the revision!

We sincerely thank the reviewer for taking the time to review our paper and provide thoughtful comments! We provide our responses to your concerns and questions below.

The effect of bias from pre-training on property inference (Q1): Yes, intuitively, pre-training bias may affect the effectiveness of property inference, depending on both the pre-trained model and the target property ratio. In our experiments, we evaluated multiple pre-trained models across various gender ratios. We first examined the ratio of the target property (female) indicated by the pre-trained models, as shown in the following table:

The pre-train female ratios among all three models are between 60%-70%. We check the results in Table 1 and have the following observations:

1. In Chat-Completion mode, both the BB generation attacks and shadow model attack perform effectively across target female ratios ranging from 30% to 70%.

2. In Q&A mode, we observe that the BB generation attack yields significantly higher error at a target female ratio of 30%, which has the greatest deviation from the pre-train ratio, compared to the errors observed at 50% or 70%. On the contrary, the shadow model attack based on word-frequency is more effective in the QA mode.

These observations suggest that (1) In Chat-Completion mode, the models learn the full distribution of the fine-tuning data and pre-training bias has minimal impact on property inference. (2) In Q&A mode, where fine-tuning does not optimize over full text but only answers, pre-training bias may have a greater influence on subsequent property inference. Additionally, the shadow-model attack with word-frequency has stronger ability than BB generation attack to overcome this bias.

Evaluating more models (Q2, W1): We evaluate our PropInfer benchmark on the Qwen-2.5-7B-Instruct model and observe similar trends. The attack results for the QA mode (MAE) are shown below.

Gender property

Medical diagnosis property

The observations on Qwen-2.5 are consistent with those from other models presented in the paper. For example, in the case of the Female property—which is more likely to appear in the question (input)—the shadow model using word-frequency features performs best when the model is fine-tuned in QA mode. In contrast, for medical diagnosis properties, which are more commonly found in the answer (output), the BB generation attack achieves strong performance. We will include the full results in the revised version.

Intuition of word-frequency signals on L230-231 (Q3-1, W2): We begin by examining how word frequencies vary with different female ratios in the fine-tuning dataset. The tables below show the number of occurrences of selected words across 150k i.i.d. samples generated by the fine-tuned LLaMA-1 model under varying target female ratios.

We observe that the occurrences of both words increase monotonically with the target female ratio.

Furthermore, for each word, we compute the correlation coefficient between its occurrence frequency and the corresponding female ratio. We present the list of selected key words from the shadow model with the corresponding correlation coefficients with the ratio.

-- Model: Llama-1 (CC mode): The top 10 keywords are (his, himself, her, he, female, him, prostate, she, son, daughter) and the correlation coefficients of −0.956, −0.934, 0.947, −0.950, 0.960, −0.933, −0.931, 0.954, −0.929, and 0.932, respectively.

-- Model: Pythia (CC mode): The top 10 selected keywords are (scrotum, he, penis, foreskin, male, glans, female, masturbate, masturbation, tip) and the correlation coefficients of −0.941, −0.916, −0.944, −0.934, −0.928, −0.940, 0.968, −0.919, −0.921, and −0.917, respectively.

-- Model: Llama-3 (CC mode): The top 10 keywords are (penile, erect, penis, scrotum, female, scrotal, males, masturbating, erection, erectile) and the correlation coefficients of −0.940, −0.935, −0.939, −0.935, 0.980, −0.941, −0.935, −0.945, −0.945, and −0.934, respectively.

-- Model: Llama-1 (QA mode): The top 5 selected keywords are (female, reluctant, spotting, scanty, ovaries) and the correlation coefficients are 0.785, 0.774, 0.787, 0.821, and 0.782, respectively.

-- Model: Pythia (QA mode): The top 10 selected keywords are (pelvic, recurring, football, bland, indigestion, bothering, uti, point, presenting, smear) and the correlation coefficients of 0.738, 0.730, −0.740, 0.760, 0.740, 0.822, 0.746, 0.743, −0.751, and 0.753, respectively.

-- Model: Llama-3 (QA mode): The top 10 selected keywords are (nifedipine, readings, yielding, squats, analogs, smoke, particular, cigarette, quit, regularly) and the correlation coefficients are −0.786, −0.788, −0.765, −0.788, −0.791, −0.773, 0.847, −0.774, −0.810, and −0.832, respectively.

A correlation coefficient close to 0 indicates no correlation, while values near 1 or -1 indicate strong positive or negative correlation, respectively. We find that many selected keywords exhibit strong correlations with the target female ratio, supporting the effectiveness of word frequency as a signal. Notably, the correlations are generally stronger when the target model is in CC mode compared to QA mode—aligning with the performance differences observed in Table 1, where word-frequency attacks perform better under CC mode than QA mode.

The choice of the number of key words d (Q3-2, W2): When training the meta attack model $g$, the parameter $d$ represents the number of features. From a learning perspective: when $d$ is too small, the features are not sufficient enough to predict the label (target ratio) correctly; conversely, when $d$ is too large, the limited number of shadow models—due to the high cost of training them—may lead to overfitting and does not learn useful mapping that generalize well.

The relationship between property inference attack and data extraction attack (Q4, W2) As noted in [1], data extraction attacks on large language models tend to recover training examples that appear multiple times, introducing a repetition bias. This means the extracted data may not represent the full training distribution, especially for rare or non-repetitive samples. This highlights a key distinction between data extraction and property inference: while data extraction is considered successful if any training samples are recovered, property inference aims to recover global properties of the training set. Therefore, strong performance in data extraction does not imply effectiveness in property inference, which motivates our decision to study property inference as a distinct task.

[1] Carlini, Nicholas, et al. "Extracting training data from large language models." 30th USENIX security symposium (USENIX Security 21). 2021.

Human study (Q5): Besides the authors, we invite one general audience to help evaluate the labeling performance of ChatGPT.

We sincerely thank the reviewer for taking the time to review our paper and provide thoughtful comments! We provide our responses to your concerns and questions below.

Poor performance to infer childbirth rate (W1): We would like to elaborate more details to explain the relatively poorer performance for the childbirth ratio than the other properties. We checked the exact ratios of three medical properties from the pre-trained models, which are presented in the following table:

We observe that the pre-train ratio for Childbirth is notably lower than the pre-train ratio for the other two properties. This may help explain the weaker property inference performance on fine-tuned models for Childbirth: the pre-trained models may not have enough prior knowledge about Childbirth property (possibly due to some safety training that removes the sensitive topics such as childbirth); hence, the model may struggle to effectively learn Childbirth-related content during fine-tuning, making property inference for this attribute more difficult.

Potential defenses (W2): we agree that evaluating the attack under potential defenses is valuable. As part of the rebuttal, we implemented a simple defense: since the attack depends on model generations, we adjusted the temperature parameter $T$ in the final softmax layer. Note that $T>1$ makes the model 's output more balanced among all tokens and $T<1$ makes the model's output more concentrated on the high-probability tokens.

We empirically evaluate this defense on the BB-generation attack using LLaMA-3 fine-tuned in CC mode. The table below reports the average predicted ratios inferred by the attack under different temperature settings, given fine-tuning datasets with varying female ratios.

We observe that without defense ($T=1$), the attack predict the target female ratio mostly correctly. When the temperature is altered $(T \neq 1)$, the predicted ratios have larger error, indicating that adjusting decoding settings can serve as a simple yet effective defense against sampling-based attacks. However, such a defense may break down if the adversary is aware of the default temperature setting or has access to the model weights. A deeper investigation into the limitations of this defense, as well as the development of more robust mitigation strategies, is an important direction for future work. We will include the additional results and the above discussion in our revised version.

Clarification on Figure 1 (Q1): The “Female” property refers to the gender of the patient. In Figure 1, the patient is the daughter, as indicated by the parent seeking medical advice on her behalf. We will revise the text to define the “Female” property more clearly and avoid confusion.

Clarification on sample sizes for black-box attacks (Q2): Figure 2(c) in the main paper shows the ablation study on the effect of sample sizes on the performance of BB generation attack when targeting LLaMA-3 on the gender property, and Figure 7 in the appendix shows the ablation study on other models. We can observe that the attack performance converges around 2k samples for LLaMA-1 and Pythia, and shows near-convergence for LLaMA-3.

Study on the sensitivity of prompt wording and the number of prompts (Q3): We conduct an ablation studied on the effect of individual prompts on the performance of BB-generation attack in Table 3 (appendix). The results show that while there is some variance in attack performance across different prompts, the attack remains consistently effective—indicating that its success is not sensitive to prompt choice. In addition, we observe that aggregating three prompts is better than a single prompt for most cases across different models and different target ratios.

Clarification on shadow attack (Q2): The auxiliary dataset has 14791 data, enabling the subsampling of size 6500 (same as fine-tune size) with different desired ratios. As for selecting the keywords, we first construct an $k_1k_2\times |V|$ feature matrix $X$ where $X_{i, j}$ indicate the $j$th word’s frequency in the $i$th shadow model, and a label vector $y$, where $y_i$ indicates the property ratio of the $i$th shadow model. Then we apply the F-regression (https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html) for this regression task (X, Y), which assigns the score to each feature. Then we are able pick the top-$d$ features (i.e. words) according to the scores. We will add more details in the revision about this feature selection procedure.

We sincerely thank the reviewer for taking the time to review our paper and provide thoughtful comments! We provide our responses to your concerns and questions below.

Justification for the gray-box assumptions (W1, Q1): The gray-box setting we adopt is standard in prior work on attacks like membership inference [1, 2] and is intended to assess the upper bound of a realistic adversary’s capability. We agree that relaxing these assumptions offers deeper insight into the practical effectiveness of the attack; hence, in our rebuttal, we further examine assumptions about access to the pre-trained model and target dataset size.

To test the assumption of access to the same pre-trained model, we evaluate a setting where the adversary uses LLaMA-3 for training shadow models but targets a LLaMA-1 model. MAE results for both CC and QA modes are shown in the table below.

CC mode (MAE)

QA mode (MAE)

We observe that the knowledge of the pre-trained model has small effects in CC mode, but notable effects in the QA mode. This might be because with the CC mode, the model is learned to fully mimic the texts in the fine-tuning dataset, disregarding what the pre-trained model is. However in QA mode, while the model learns to answer questions based on the fine-tuning data, the generation of the questions themselves may still rely heavily on the pre-trained model, leading to greater divergence in the resulting question–answer pairs.

To validate the dataset size, we additionally test our word-frequency attack by training shadow models with the dataset size 3000 – the ground truth fine-tuning dataset size is 6500. The inference results (MAE) towards the target Llama-1 model trained with CC mode are reported below:

Similar to the observation with pre-trained model architecture, we find that knowledge of the fine-tuning dataset size has minimal impact on the performance of the property inference attack. This may be because with the CC mode, the model is trained to closely mimic the fine-tuning data, so even if the fine-tuning dataset size varies, the model’s output distribution remains consistent.

We will add the results to the revision as an analytical study for our word-frequency attack!

[1] N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), pages 1897–1914. IEEE, 2022.

[2] Watson, Lauren, et al. "On the importance of difficulty calibration in membership inference attacks." arXiv preprint arXiv:2111.08440 (2021).

Details about the prompt construction (Q2-1, W2-1): The prompts used in both the BB-generation and word-frequency-based attacks are listed in Appendix A.4. We choose three generic prompts, such as “Hi, ChatDoctor, I have a medical question.” without any prompt engineering or cherry picking.

We conduct an ablation study on the effect of individual prompts on the BB-generation attack: as shown in (Table 3 Appendix), we observe that while there is some variance in attack performance across different prompts, the attack remains consistently effective—indicating that its success is not sensitive to prompt choice.

The word-frequency feature versus existing features such as loss or perplexity (Q2-2, W2-2): Our perplexity-based baseline adapts the existing shadow model attack from [1] to the LLM setting by computing average perplexity over two hold-out datasets with 0% and 100% property ratios, forming a two-dimensional shadow feature (see Appendix A.2 for details). As shown in the results (Table 1), this baseline is consistently outperformed by our shadow-model attack based on word-frequency, indicating that perplexity is not a strong signal for property inference in LLMs.

We provide additional justification for using word frequency as a feature for the shadow model attack (see our response on Intuition of word-frequency signals on L230-231 to Reviewer nLBK). To support this, we report correlations between selected keywords and property ratios to test the hypothesis that certain properties are closely linked to specific word frequency in the generated text.

[1] A. Suri and D. Evans. Formalizing and estimating distribution inference risks, 2022.

Other potential dataset (Q3, W3): In our experiments with the ChatDoctor dataset, we varied three gender ratios and selected three additional medical attributes as targets for property inference. Meanwhile, we would like to discuss other practical scenarios which might support the evaluation – such as the one suggested by Reviewer 3qjY—where a model fine-tuned on political news might reveal the distribution of political topics in its training data.

The extension to property inference on pre-trained models (Q4, W4): Property inference on fine-tuned models has its own practical scenarios as fine-tuning datasets are often application- and user-specific, containing sensitive properties that should remain private. While we agree that the property inference on pre-trained models represents an interesting threat model, studying this setting requires either access to ground truth properties of existing public LLMs or substantial computational resources to pre-train models from scratch on controlled datasets–making it computationally infeasible for us.