Response to Reviewer J72D

Thank you for your thoughtful and constructive feedback on our manuscript. We appreciate the opportunity to address your concerns and provide additional clarification. Additionally, thank you for bringing to my attention the issue with the unfilled NeurIPS reproducibility checklist. I apologize for this mistake, and I am grateful to the AC and reviewers for allowing me the chance to submit a rebuttal despite this. I will make sure to fully and accurately complete the reproducibility checklist in the camera-ready version, adhering to all NeurIPS guidelines. Furthermore, I will carefully address all the weaknesses highlighted by the reviewers in the limitations section. I appreciate your attention to this crucial aspect of responsible research reporting. Below, we respond to the key points raised.

Response to Q1

Q1: How does the attack performance change with different surrogate model? In other words, a comprehensive studies on the impact of the surrogate model to proposed attacks would comprehend this study.

Answer: Thank you for raising this important point. We have conducted an ablation study to evaluate the impact of different surrogate models on the attack performance. The results of this comprehensive analysis are presented in the following tables.

Table: Complete results of various attacks on LOMIA/LAION when the surrogate model is LLaVA.

Table: Complete results of various attacks on LOMIA/LAION when the surrogate model is MiniGPT4.

Table: Complete results of various attacks on LOMIA/LAION when the surrogate model is LLaMA-Adapter.

Response to Q2

Q2: How is the attack performance with a more number of testing data point?

Answer: Thank you for raising this important point. To evaluate the impact of a larger test set, we expanded the LOMIA/LAION dataset to 1,000 samples (500 members and 500 non-members). The attack performance with this increased number of testing data points is reported in the following table （Due to character limitations, the complete comparative experiments with other methods will be presented in the revised version.）:

Table: Complete results of our attacks on larger dataset.

Table: Performance of larger dataset on GPT-4o

Response to Q3

Q3: In this paper, the author claims that this is the first study on MIAs on VLLMs. However, [17] seems to be a label-only MIA on VLLMs. Please clarify how your setting differs from [17], and if the claim of being the first still holds.

Answer: Thank you for your question. To the best of our knowledge, our work is the first to investigate label-only MIAs on pre-trained VLLMs, whereas [17] focuses on MIAs against fine-tuned VLLMs. Attacking pre-trained models is generally more challenging. This can be attributed to the fact that VLLMs are pre-trained on large-scale corpora, with each sample typically seen only once or a few times (often fewer than three). As a result, VLLMs exhibit superior generalization compared to fine-tuned VLLMs, which in turn reduces the gap in distance to the decision boundary between members and non-members. We will clarify this distinction in the revised manuscript.

Response to W2

W2: The absolute TPR@1%FPR is low, which might not be actionable in practice with small sample batches.

Answer: Thank you for raising this important point. We agree that the absolute TPR@1%FPR are relatively low, which may limit the direct applicability of the attack in scenarios with small sample batches. However, it is worth noting that this phenomenon is not unique to our proposed method. In fact, as shown in our results and in prior work, many logit-based membership inference attacks on VLLMs also achieve similarly low or even lower TPR@1%FPR values under the same evaluation protocol. We will discuss this point in the revised manuscript.

Response to Reviewer HBHB

Thank you for your thoughtful and constructive feedback on our manuscript. We appreciate the opportunity to address your concerns and provide additional clarification. Below, we respond to the key points raised.

Response to Q1

Q1: Lack of detail optimization on attacking. How to optimize the Equations 3,6 and 9?

Answer: Many thanks for your valuable suggestions. Equation 3 employs closed-form least squares optimization for text-to-text similarity mapping:

$$R : S_{t2t} \mapsto PPL, \quad \min_{k,b} \sum \left\| PPL^{(i)} - k S_{t2t}^{(i)} - b \right\|^2$$

where we minimize L2 loss between text-to-text similarity and PPL using the analytical solution:

$$k = \frac{n\sum xy - \sum x \sum y}{n\sum x^2 - (\sum x)^2}$$ $$b = \frac{\sum y - k\sum x}{n}$$

with $x$ representing text-to-text similarity scores and $y$ representing PPL.

Equation 6 follows the same optimization framework but leverages CLIP-based image-to-text features:

$$R : S_{i2t} \mapsto PPL, \quad \min_{k,b} \sum \left\| PPL^{(i)} - k S_{i2t}^{(i)} - b \right\|^2$$

utilizing identical closed-form solutions where $x$ now represents CLIP image-to-text similarity scores and $y$ representing PPL.

Equation 9 extends to multivariate optimization:

$$R : (S_{i2t}, S_{t2t}) \mapsto PPL, \quad \min_{k_1, k_2, b} \sum \left\| PPL - k_1 S_{t2t}^{(i)} - k_2 S_{i2t}^{(i)} - b \right\|^2$$

employing normal equations:

$$\boldsymbol{\beta} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$$

where $\boldsymbol{\beta} = [k_1, k_2, b]^T$, feature matrix $\mathbf{X}$ combines both text-to-text and image-to-text similarities, and $y$ representing PPL.

Response to Q2

Q2: Lack of the threshold of determining whether the sample is a member or non-member, which is very important to reproduce this paper.

Answer: Thank you for raising this important point. In our evaluation, we use the AUC score, balanced accuracy, and TPR@low FPR as our primary metrics. These metrics are standard in the previous work, such as [3], as both are threshold-independent. Specifically, the AUC represents the area under the ROC curve, which plots the true positive rate against the false positive rate across all possible thresholds. This means it is not necessary to select a specific threshold when comparing different MIA methods. However, if the goal is to deploy a particular MIA method in practice, a threshold can be determined by performing hyperparameter search on a small validation set. We will clarify this explanation in the revised version to enhance reproducibility.

Response to Q3

Q3: The checklist is unfinished and it may conflict with the conference policy. Furthermore, this paper doesn't discuss limitations or social negative impacts.

Answer: Thank you for pointing out the omission regarding the NeurIPS reproducibility checklist. I sincerely apologize for this oversight and appreciate the AC and reviewers’ understanding in still providing the opportunity for rebuttal. I will ensure that the checklist is properly completed in the camera-ready version, in full compliance with NeurIPS requirements. Additionally, I'll consider all the weaknesses provided to include in our limitations section. Thank you again for your attention to this important aspect of responsible reporting.

Response to Q4

Q4: Constructing the relationship between sample similarities and perplexities as an attack lacks a detailed explanation.

Answer: Thank you for your valuable question. The detailed explanation for the relationship between sample similarities and perplexities are as follows: First, it is well established that LLMs tend to assign higher probabilities to tokens and sequences that are more semantically or syntactically similar to their training data. This means that, for member samples, the model is more likely to generate outputs with higher confidence, resulting in lower perplexity scores. Conversely, for non-member samples, the model is less familiar with the content, leading to higher perplexity. Second, the process of text generation in LLMs inherently favors high-probability tokens, further amplifying the difference in perplexity between member and non-member samples. This property is inherited by VLLMs, as they are built upon LLM architectures and share similar probabilistic behaviors. To provide a deeper analysis, we also examine the relationship between relevant features and perplexity. Our results, summarized using standard regression metrics in the following table.

Table: Standard regression metrics for regression models

References

[3] Ko M, Jin M, Wang C, et al. "Practical membership inference attacks against large-scale multi-modal models: A pilot study." ICCV 2023.

Response to Reviewer gVAv

Thank you for your thoughtful and constructive feedback on our manuscript. We appreciate the opportunity to address your concerns and provide additional clarification. Below, we respond to the key points raised.

Response to Q1

Q1: How do attacks perform when real unseen images (e.g., held-out from COCO/CC or external datasets) are used as non-members instead of diffusion-generated ones?

Answer: Many thanks for your suggestions. We randomly sampled 500 image-caption pairs from open-images-captions-micro (2024 release) on Hugging Face, ensuring exclusion of any data overlapping with COCO or LAION datasets. We also enlarge our LOMIA/LAION member data samples to 500 (500 members + 500 non-members), the final results can be found in the following table. （Due to character limitations, the complete comparative experiments with other methods will be presented in the revised version.）

Table: Complete results of our attacks when non-member data are real images

Table: Performance of real non-member dataset on GPT-4o

Response to Q2

Q2: Have the authors explored the sensitivity of the attack to the choice of threshold τ?

Answer: Thank you for raising this important point. In our evaluation, we use the AUC score, balanced accuracy, and TPR@low FPR as our primary metrics. These metrics are standard in the previous work, such as [1][3], as both are threshold-independent. Specifically, the AUC represents the area under the ROC curve, which plots the true positive rate against the false positive rate across all possible thresholds. This means it is not necessary to select a specific threshold when comparing different MIA methods. However, if the goal is to deploy a particular MIA method in practice, a threshold can be determined by performing hyperparameter search on a small validation set. We will clarify this explanation in the revised version to enhance reproducibility.

Response to Q3

Q3: Can the authors report standard regression metrics (e.g., RMSE, R²) for the sim→PPL model?

Answer: Many thanks for your suggestions. The standard regression metrics for regression models are listed in the following Table.

Table: Standard regression metrics for regression models

Response to Q4

Q4: Could the authors clearly specify the vision encoder used in each target model (LLaVA, MiniGPT-4, LLaMA-Adapter)?

Answer: Many thanks for your suggestions. The versions and base models of target VLLMs are listed in the following Table.

Table: VLLM details

Response to Q5

Q5: More broadly, could the authors address the various concerns raised under the Weaknesses section of this review?

W1: The generality of the attack beyond CLIP-based models.
Answer: We have evaluated study when surrogate model is Mini-GPT4, the vision processor for Mini-GPT4 is BLIP2/EVA-ViT-G, the results are listed in the following table:

Table: Complete results of our attacks on LOMIA/LAION when the surrogate model is MiniGPT4

W2: How performance is affected by differences in model size (e.g., 7B vs. 13B)

Answer: We also test our LOMIA on larger VLLMs, such as LLaVA-1.5-13b and MiniGPT4-vicuna-13b, which can be found in appendix:

Table: Performance of LOMIA on LLaVA-1.5-13b

Table: Performance of LOMIA on MiniGPT4-vicuna-13b

W3: The potential bias introduced by the sentence embedding model

Answer: Thank you for raising this important point. We acknowledge that some sentence embedding models (e.g., MiniLM) may have been partially trained on datasets such as COCO or LAION, which are also used in VLLM training. However, the sentence embedding model is used solely for measuring semantic similarity, not for text generation. Since embedding models encode text into vector representations to capture semantic relationships rather than generate content. Even if the embedding model was trained on datasets that overlap with VLLM training data, this would not introduce bias in our similarity measurements because: (1) the embedding model's role is to map semantically similar texts to nearby points in vector space, which is a fundamentally different task from text generation; (2) our evaluation focuses on whether the VLLM can generate captions that are semantically similar to ground truth, which requires the embedding model to have good general semantic understanding rather than memorization of specific samples. We will clarify this point and discuss the potential impact in the revised manuscript.

Reference

[1] Shi, Weijia, et al. "Detecting Pretraining Data from Large Language Models." ICLR 2024.

[2] He Y, Li B, Liu L, et al. "Towards label-only membership inference attack against pre-trained large language models." USENIX Security 2025.

[3] Ko M, Jin M, Wang C, et al. "Practical membership inference attacks against large-scale multi-modal models: A pilot study." ICCV 2023.

Response to Reviewer WrZE

Thank you for your thoughtful and constructive feedback on our manuscript. We appreciate the opportunity to address your concerns and provide additional clarification. Below, we respond to the key points raised.

Response to Q1

Q1: How is the threshold value determined for deciding whether an input is a member or non-member?

Answer: Thank you for raising this important point. In our evaluation, we use the AUC score, balanced accuracy, and TPR@low FPR as our primary metrics. These metrics are standard in the previous work, such as [1][3], as both are threshold-independent. Specifically, the AUC represents the area under the ROC curve, which plots the true positive rate against the false positive rate across all possible thresholds. This means it is not necessary to select a specific threshold when comparing different MIA methods. However, if the goal is to deploy a particular MIA method in practice, a threshold can be determined by performing hyperparameter search on a small validation set. We will clarify this explanation in the revised version to enhance reproducibility.

Response to Q2

Q2: Can the authors provide more details about the surrogate models and clarify whether their training data differs from that of the target model?

Answer: Thank you for your question. In this paper, we use LLaVA-1.5-7B as the surrogate model, the based LLM for LLaVA-1.5-7B is Vicuna v1.5 7B and the vision processor is CLIP-ViT-L. To ensure the validity of the experiment, we randomly sampled a subset from the intersection of the datasets used by the pre-trained target model mentioned in our paper. The use of a surrogate model is solely aimed at identifying the relationship between relevant features and the perplexity of generated descriptions, rather than determining whether a specific sample is a member. Additionally, [2] similarly exhibits training data overlap across surrogate models and target models, with no observed effect on experimental conclusions.

Response to Q3

Q3: Can the authors provide results on more recent VLMs?

Answer: Thank you for your suggestion. As shown in Table 3, we have evaluated our methods on GPT-4o, which is one of the most recent and widely used VLLMs. This evaluation demonstrates that the privacy issues identified in our study are also present in state-of-the-art VLLMs. The detailed results can be found in the following table.

Table: Performance of LOMIA on GPT-4o

Response to Q4

Q4: Can the authors provide a clear justification or explanation for why the proposed method is effective? A deeper analysis or theoretical insight would help strengthen the contribution.

Answer: Thank you for your valuable question. The effectiveness of our proposed method can be attributed to the following reasons: First, it is well established that LLMs tend to assign higher probabilities to tokens and sequences that are more semantically or syntactically similar to their training data. This means that, for member samples, the model is more likely to generate outputs with higher confidence, resulting in lower perplexity scores. Conversely, for non-member samples, the model is less familiar with the content, leading to higher perplexity. Second, the process of text generation in LLMs inherently favors high-probability tokens, further amplifying the difference in perplexity between member and non-member samples. This property is inherited by VLLMs, as they are built upon LLM architectures and share similar probabilistic behaviors. To provide a deeper analysis, we also examine the relationship between relevant features and perplexity. Our results, summarized using standard regression metrics in the following table.

Table: Standard regression metrics for regression models

Response to W3

Many thanks for your suggestions. To evaluate our attacks on real unseen images, we randomly sampled 500 image-caption pairs from open-images-captions-micro (2024 release) on Hugging Face, ensuring exclusion of any data overlapping with COCO or LAION datasets. We also enlarged our LOMIA/LAION member data samples to 500 (500 members + 500 non-members), the final results can be found in the following table（Due to character limitations, the complete comparative experiments with other methods will be presented in the revised version.）:

Table: Complete results of various attacks when non-member data are real images

Table: Performance of real non-member dataset on GPT-4o

Reference

[1] Shi, Weijia, et al. "Detecting Pretraining Data from Large Language Models." ICLR 2024.

[2] He Y, Li B, Liu L, et al. "Towards label-only membership inference attack against pre-trained large language models." USENIX Security 2025.

[3] Ko M, Jin M, Wang C, et al. "Practical membership inference attacks against large-scale multi-modal models: A pilot study." ICCV 2023.

Thanks the authors for providing the rebuttal.

We also enlarged our LOMIA/LAION member data samples to 500 (500 members + 500 non-members),

The updated results are interesting. Compared to Table 1, the performance rankings appear reversed. In Table 1, TTFA appears to be the most effective attack method. However, in this updated setting with real images, DUFA generally performs best, while TTFA consistently ranks the lowest. Could the authors clarify/explain this?

Response to Q4

Thank you for the explanation. However, it does not fully address my concern. As I pointed out in my review, a key weakness of the method lies in its strong and potentially unrealistic, assumption that a meaningful and transferable relationship exists between the surrogate and target models. For instance, if the surrogate model is trained on natural image-text pairs, while the target model is trained on domain-specific data (e.g., medical images), there is no guarantee that the surrogate will generalize or provide useful the relationship between relevant features and the perplexity of generated descriptions.

Response to Q2

I think it is essential that the training data for surrogate models differ from that of the target model. This is important because the paper focuses on label-only MIA. If there is overlap between the training data of surrogate and target models, then the label-only setting becomes less meaningful, as access through the surrogate model undermines the assumption of label-only restriction.