Semantic Membership Inference Attack against Large Language Models

We thank the reviewer for their feedback. We provide responses for the weaknesses and questions raised below:

(w1)-Evaluation datasets measure distribution shift, not MI

To address the potential issue of our evaluation datasets measuring distribution shifts rather than true membership inference performance, we have added Section C.6 to our paper. In this section, we implement two blind attacks from Das et al. (2024): (a) greedy rare word selection and (b) bag-of-words classification, which operate without any signal from the target model. Our results, summarized in Table 13, show that these blind attacks achieve high AUC-ROC scores on the WC split (58.30% and 83.6%, respectively), indicating that WC is susceptible to distribution shift exploitation.

As we mention in Section 5.1, WC contains distribution-shifted data, and we discuss why different MIAs achieve good results on this dataset due to the distinct distributions of members and non-members. Our main evaluations focus on the WT dataset, which provides a realistic and challenging scenario where members and non-members come from closely aligned distributions. Evaluations on WC are included for comprehensiveness but are not the primary basis for our conclusions.

On the WT split, the same blind attacks perform no better than random guessing, with AUC-ROC scores around 52% and TPRs at low FPRs near zero. This stark contrast emphasizes the limitations of datasets like WC for evaluating MIAs, as their success is largely driven by distributional differences rather than true privacy leakage. Importantly, our proposed SMIA method maintains strong performance on the WT dataset (as shown in Tables 7 and 8), effectively detecting membership where blind attacks fail. This indicates that SMIA is not merely capturing distribution shifts but is robustly inferring membership information based on the model's learned representations, thereby addressing true privacy leakage concerns.

(w2)- AU-ROC alone is insufficient to evaluate MIAs

We agree that relying solely on AU-ROC can be insufficient and potentially misleading for evaluating membership inference attacks. In our paper, we have provided TPR at low FPR values for our main results in Tables 4 and 5 and have now added TPR@1% FPR to Table 4 for a more comprehensive evaluation; while previous works like Min-K and Min-K++ did not report TPR for FPR lower than 5%—possibly due to the low absolute values—we recognize the value of including these metrics to facilitate a thorough comparison; from Table 4, it is evident that SMIA achieves the best results in most scenarios, especially at low FPR thresholds, demonstrating its effectiveness over existing methods. We will update other tables to include TPR@1%FPR values in the main paper to enhance transparency.

(w3)- Complex approach without ablation

We have added Section C.7 to present additional ablation experiments that demonstrate the robustness of SMIA under variations in both embedding models and classifier network sizes. Specifically, Table 10 includes results using the E5-mistral-7b-instruct embedding model alongside the original Cohere v3 model, showing consistent performance across different embedding models. Additionally, Table 11 explores the impact of varying classifier network sizes by comparing the original network with a smaller linear classifier and a larger network with additional fully connected layers. These results collectively highlight SMIA’s adaptability to different model configurations.

(w4)- SMIA requires a known member and non-member subset from the training data distribution. This is a non-trivial assumption that requires some discussion.

While SMIA does require access to known member and non-member subsets from the training data distribution, we believe this assumption is realistic in several practical scenarios. In privacy auditing or unlearning verification, for instance, auditors or red team members often have access to portions of both member and non-member data to evaluate a model's compliance with privacy standards. Even in general attack settings, adversaries can leverage knowledge about the model's training cutoff date to infer likely members and non-members—for example, assuming that data published before the cutoff (e.g., Wikipedia articles, arXiv papers, GitHub repositories) are included in the training set, while newer data are not. We discuss this in Section 3.2, illustrating how an adversary might collect such data for SMIA. Therefore, while the requirement is non-trivial, it reflects practical conditions under which membership inference attacks are relevant and actionable.

After reading the other reviewers' comments and due to no response from the authors, I keep my verdict and recommend rejecting this paper. The authors have now responded; see below.

Consider the weaknesses raised by other reviews and the authors' inactivity. I would like to adjust my score to 3.

We thank the reviewer for their feedback. We provide responses for the weaknesses and questions raised below:

(w1 & q1)- Could the authors provide some examples of grey-box models?

In our work, the grey-box scenario refers to a threat model where the adversary has access only to the loss values returned by the target model for texts of their choosing. This setting is practical for models accessed via APIs that provide limited feedback, such as loss or perplexity scores, without exposing internal parameters or detailed outputs like per-token logits. Examples of grey-box models include cloud-based language services and commercial language models that offer summary metrics but restrict deeper access to prevent reverse engineering or misuse. Our attack, SMIA, is designed to operate effectively under these conditions, relying solely on loss values. This contrasts with prior works that require white-box access (e.g., [AA, BB]) or exact per-token logits (e.g., MinK and MinK++). In all our experiments, we simulate this grey-box scenario by using only the loss values from the target models, demonstrating the practicality and applicability of SMIA in real-world settings.

[AA] : Milad Nasr, Reza Shokri, and Amir Houmansadr. "Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning". In 2019 IEEE symposium on security and privacy (SP)

[BB] Suri, Anshuman, Xiao Zhang, and David Evans. "Do Parameters Reveal More than Loss for Membership Inference?." arXiv preprint arXiv:2406.11544 (2024).

(w2 & q2)- How will the performance of SMIA change if changing the embedding model or classification models?

We have added Section C.7 to present additional ablation experiments that demonstrate the robustness of SMIA under variations in both embedding models and classifier network sizes. Specifically, Table 9 includes results using the E5-mistral-7b-instruct embedding model alongside the original Cohere v3 model, showing consistent performance across different embedding models. Additionally, Table 10 explores the impact of varying classifier network sizes by comparing the original network with a smaller linear classifier and a larger network with additional fully connected layers. These results collectively highlight SMIA’s adaptability to different model configurations.

(w3 & q3)- What is the general behavior of loss from members? For example, will the loss from members change less than non-members considering neighbours?

In MIAs, it is commonly observed that target models assign lower loss values to members due to overfitting, resulting in a distinct loss trend compared to non-members. We provide these baseline results as LOSS attack in our experiments. However, relying solely on loss values can be problematic because certain non-member inputs—such as short or repetitive texts—may also receive low loss values, leading to false positives. Our proposed method, SMIA, addresses this issue by incorporating semantic embeddings of the text. By leveraging the semantic information, SMIA captures the contextual relationships between data points, allowing us to more effectively distinguish between members and non-members. Specifically, we analyze how the loss values change concerning the semantic neighbors of each input. Members tend to have loss patterns that are not only lower but also more consistent within their semantic neighborhoods. This approach enhances the attack's precision by considering both the loss and the semantic context.

(q1)- The scalability of SMIA deserves further exploration - how would the method perform with larger training datasets and more complex model architectures? The current evaluation uses 6,000 members and non-members each, which may not fully reflect real-world scenarios.

In Figure 3, we investigated the impact of varying training dataset sizes on the performance of SMIA by experimenting with 6,000/6,000, 4,000/4,000, 2,000/2,000, and 1,000/1,000 members and non-members. The results demonstrate that increasing the size of the training data enhances the effectiveness of SMIA, indicating its potential scalability. However, we acknowledge that scaling to much larger datasets and more complex models would entail significantly higher computational costs, particularly for neighbor generation and embedding calculations. Addressing these computational challenges and extending our evaluation to larger, real-world scenarios is an important direction for future work.

(q2)- The choice of neural network architecture for the SMIA model itself appears somewhat arbitrary. Have the authors explored alternative architectures that might improve performance or efficiency? Additionally, how sensitive is the method to the quality and dimensionality of the semantic embeddings?

We based our architecture on the work of Nasr et al. (2019), which has proven effective in white-box MIAs against deep neural networks in vision tasks. To explore alternative architectures and assess sensitivity to embedding quality and dimensionality, we have added Section C.7 with additional ablation experiments. Specifically, Table 9 presents results using the E5-mistral-7b-instruct embedding model (with 4096 dimensions) alongside the original Cohere v3 model (with 1024 dimensions), demonstrating consistent performance across different embeddings. Additionally, Table 10 investigates the impact of varying classifier network sizes by comparing our original network with a smaller linear classifier and a larger network with extra fully connected layers. These findings collectively highlight SMIA's robustness and adaptability to different model configurations, suggesting that its performance is not overly sensitive to the choice of neural network architecture or embedding model used.

We thank the reviewer for their feedback. We provide responses for the weaknesses and questions raised below:

(w1)- The choice of key hyperparameters, particularly the use of 25 neighbours ...

In Section B of our paper, we provide a detailed cost estimation that encompasses generating neighbors, computing embeddings, and evaluating loss values for the target model. Our experiments, as shown in Table 7, indicate that performance improves with an increasing number of neighbors up to 25. Beyond this point, we observed diminishing returns in performance gains while computational costs continue to rise significantly. Specifically, utilizing more than 25 neighbors would necessitate larger training datasets to capitalize on the additional information, which leads to increased computational overhead without proportional benefits. Therefore, we selected 25 neighbors as it strikes an optimal balance between effectiveness and efficiency, offering substantial performance improvements without incurring excessive computational costs.

(w2 & w4)- A concerning weakness emerges in the method's inconsistent performance across different types of text modifications .... The method shows notably better performance with word deletions compared to additions or duplications, as evidenced in Table 3 ...

This discrepancy stems from how these modifications affect the coherence and consistency of the input texts, which in turn influences the model's loss values. In the case of word duplication, we duplicate an exact word consecutively within the text. This repetition can introduce redundancy without significantly altering the semantic meaning, which may substantially impact the model's loss values. Consequently, the loss patterns between members and non-members become less distinguishable, affecting SMIA's ability to infer membership accurately.

For word additions, we currently insert only the first word generated by the T5 model into the original text. This approach can lead to incomplete or out-of-context additions, disrupting the text's coherence and causing inconsistent loss values when processed by the target model. Such inconsistencies make it challenging for SMIA to detect meaningful patterns for membership inference.

In contrast, word deletions often result in more coherent sentences, as the removal of a word may not drastically alter the overall meaning of the text. This maintains a consistent loss landscape, allowing SMIA to more effectively differentiate between members and non-members based on their loss and semantic relationships.

(w3)- A minor problem is the heavy reliance on the Cohere Embedding model, a third-party service, introduces both a potential point of failure and a privacy concern ...

In response, we have added Section C.7 to our paper, where we present additional ablation experiments demonstrating the robustness of SMIA under different embedding models and classifier network sizes. Specifically, Table 9 includes results using the E5-mistral-7b-instruct embedding model alongside the original Cohere v3 model, showing consistent performance across different embeddings. This indicates that SMIA does not rely heavily on any specific third-party service, thereby mitigating potential points of failure and privacy concerns associated with data sharing. Furthermore, we have provided a comprehensive cost analysis in Section B, detailing the computational expenses of generating neighbors, computing embeddings, and evaluating loss values for the target model. This addresses the cost implications of using such services at scale and demonstrates the practicality of our approach.

(w5 & q3)- How might defenders mitigate against SMIA?

While our paper primarily focuses on demonstrating the effectiveness of SMIA to highlight existing vulnerabilities in large language models, we acknowledge the importance of discussing potential defenses against such attacks; however, an in-depth exploration of countermeasures such as Differential Privacy (DP)—which is the main solution for defending against membership inference attacks—is beyond the scope of our current work; we consider this an important direction for future research and defer a comprehensive treatment of potential defenses to future studies.

We thank the reviewer for their feedback. We provide responses for the weaknesses and questions raised below:

(w1)- There are various works at this point in literature which have argued that "WC" based assessment of MIAs when data is split across a cut-off date, is not sound. This work bases many of their gains on that setting.

We agree that evaluating membership inference attacks using datasets split across a cut-off date (e.g., 'WC' dataset) can lead to unsound assessments due to temporal distribution shifts. To address this concern, our main experiments, as detailed in Sections 4.2.1 and 5.1, are conducted using the 'WT' dataset. In 'WT', both members and non-members are drawn from the same distribution: members come from the training split of the Pile dataset, and non-members are from the test split of Pile. This ensures that the data distributions for members and non-members are consistent, providing a sound basis for evaluating MIAs.

We included the 'WC' dataset to demonstrate how temporal shifts can lead to misleading conclusions about privacy leakage. As reported in Section 5.1, we observed significant differences in MIA performance between the 'WT' and 'WC' datasets. For instance, the SMIA method achieves an AUC-ROC of 67.39% on 'WT' but increases to 93.35% on 'WC' for the Pythia-12B model. Similarly, the true positive rates at low false positive rates improve markedly on 'WC' compared to 'WT' (e.g., TPR of 3.8% vs. 46.2% at 2% FPR) which is consistent with findings from cited papers [1,3].

(w2)- The method should be tested across all 20 train-test splits of Pile. Paper [2] on LLM Dataset Inference particularly shows how most MIAs perform well only on a few datasets.... Given that the goal of MI in LLMs is hard, I would love to see experiments on dataset inference.

We understand the importance of evaluating our method across diverse datasets. However, we'd like to clarify a key distinction: The paper you mentioned introduces a dataset inference attack aimed at detecting whether an entire dataset was used in a model's training. In contrast, our research focuses on membership inference attacks targeting individual training samples of much smaller size (between 130 to 150 words). These are fundamentally different problems: dataset inference operates at the dataset level, while our MIA operates at the individual sample level. Because of this difference, the authors in [2] evaluated their approach on all train-test splits of the Pile to suit their specific problem.

Testing our method across all train-test splits of the Pile would be computationally infeasible due to the sheer scale of the data. Additionally, some splits contain copyrighted material that we cannot use due to legal and ethical considerations. To the best of our knowledge, no prior work on MIAs has considered all splits of the Pile for individual sample inference, largely because of these constraints.

Most existing works in this area utilize the MIMIR dataset, which we also employ in our study. For our experiments, we selected four significant and commonly used subsets of the Pile: Wikipedia (en), GitHub, PubMed Central, and ArXiv. Notably, Figure 3 in the cited paper [2] shows that these four datasets are the hardest to analyze, with AUC-ROCs close to 50%, indicating performance near random chance.

(w3)- The blind baseline of n-gram is not considered in this work (paper [3]). This is an important comparison to understand if the gains in this work are meaningful.

To address this, we have added Section C.6 to our paper, where we implement two blind attacks from Das et al. (2024): (a) greedy rare word selection (n-gram approach) and (b) bag-of-words classification. These attacks operate without any signal from the target model, and serve to evaluate whether our gains are meaningful beyond exploiting distribution shifts. Our results, summarized in Table 8, show that these blind attacks achieve high AUC-ROC scores on the WC split (58.30% and 83.6%, respectively), indicating that WC is susceptible to exploitation due to distributional differences between members and non-members. As discussed in Section 5.1, WC contains distribution-shifted data, and we acknowledge that this can inflate the performance of MIAs that exploit such shifts. Our main evaluations focus on the WT dataset, where members and non-members are from closely aligned distributions. On the WT split, the same blind attacks perform no better than random guessing, with AUC-ROC scores around 52% and TPRs at low FPRs near zero. In contrast, our proposed SMIA method maintains strong performance on WT (as shown in Tables 7 and 8), effectively detecting membership where blind attacks fail. This demonstrates that SMIA is not merely capturing distribution shifts but is robustly inferring membership information based on the model's learned representations, thereby addressing true privacy leakage concerns.

Dear Authors,

Thank you for your detailed and thoughtful rebuttal. I appreciate the care you've taken to address each concern systematically. Let me share my thoughts on your responses:

Regarding w1 (WC-based assessment):.
I appreciate your acknowledgment of the potential issues with WC-based evaluation and your clarification about the primary focus on WT datasets. However, I remain concerned that the paper's current structure and results presentation might still lead readers to draw conclusions from the WC results. I suggest:

• Moving the WC results entirely to an appendix
• Adding explicit warnings about interpretation of WC results in all relevant figure/table captions
• Strengthening the emphasis on WT as the primary evaluation metric in the abstract and introduction
• Make the nomenclature for WT and WC absolutely clear. It should not be hard to find their meaning.

Regarding w2 (Testing across Pile splits):.
I acknowledge your explanation about the fundamental difference between dataset-level and sample-level inference. My comment was in regards to Fig 3 which you point to later. I am not sure why this should be computationally expensive. Could you take a 1000 example subset of train and test to do this?

Regarding w3 (Blind baseline comparison):
I commend you on adding Section C.6 with the blind baseline comparisons. However, I have some important recommendations:

• It is important to move these results to the main paper rather than the appendix. This is a critical baseline. The independent table in appendix offers no comparative discussion to the average reader.
• Add a brief discussion of why your method succeeds where blind attacks fail
• Include error analysis for cases where your method outperforms the baselines

Regarding the use of MIMIR dataset:
Please check the line below Table 2 in this paper. The authors explicitly mention: We clarify that this step is not a suggestion for researchers to alter their benchmarks

In conclusion: This method might very well be promising. But the current presentation, the focus on irrelevant and unsound benchmarks leads to the authors hurting their own paper and its promise. This work needs to be re-written with a focus on the right practices for membership inference research, with a focus on the right train-test splits. I would strongly encourage the authors to extend this to multiple PILE subsets following the recommendations in [2] ignore the use of MIMIR dataset as suggested by [1], and include blind baselines in the main paper as suggested in [3]. On top of this, WC should only see a place in Appendix, if at all.