We thank the reviewer for careful attention to the paper. Our response is as follows.

Comment 1: The proposed mask prediction task uses shape- or color-based probe to identify training samples. But I’m wondering whether it’s still possible to construct these two types of probes for certain important training samples. For example, in the case of a patient’s X-ray image or someone's ID card, it might be very difficult to find appropriate objects or color in the images.

Thank you for your thoughtful observation. We respond in three parts: (1) clarify the current scope, (2) explain why these domains are challenging, and (3) propose domain-specific extensions.

• Current Scope. We acknowledge that our current implementation is primarily designed for natural images and faces challenges when applied to specialized images with low semantic diversity or weak structure (e.g., X-rays, ID cards), where meaningful visual features (e.g., shape, color) are difficult to extract or segment. This is noted in the Limitation section of Supplementary Material.

• Potential Solution. Such images often exhibit subtle or uncommon patterns that are challenging to interpret, even for humans without domain expertise. To address this, it may be beneficial to incorporate domain-specific knowledge into the probe construction process of our KCMP method, enabling better adaptation to such specialized data. For instance, in medical images, auxiliary models such as saliency detectors [1] or organ/lesion segmenters [2] can help localize clinically relevant regions for generating semantically meaningful distractors. For ID cards, OCR models [3] can identify fields like names or ID numbers to guide probe generation. We plan to explore these extensions of KCMP in future work.

Comment 2: If we use weaker models for filtering, will MIA performance decrease significantly?

Thank you for the suggestion. We conducted new ablations regarding CLIP and filtering model to investigate this:

• CLIP Model Impact

  We replaced the default vit-large-p14-336 (used in the main paper) with weaker CLIP variants:

  Dataset	CLIP Model	Size	LLaVA	LLaMA Adapter	MiniGPT-4
  VL-MIA/Flickr	vit-large-p14-336	~428M	0.598	0.573	0.544
  	vit-base-p16	~151M	0.585	0.555	0.547
  	vit-base-p32	~151M	0.592	0.550	0.532
  VL-MIA/DALL-E	vit-large-p14-336	~428M	0.565	0.568	0.543
  	vit-base-p16	~151M	0.558	0.560	0.527
  	vit-base-p32	~151M	0.552	0.544	0.534

• o1 Filtering Mechanism Impact

  We replaced GPT-4o with a lighter open-source model (LLaMA-3-8B) in the prior knowledge calibration step. Performance on VL-MIA/Flickr and LLaVA model is shown below:

  Filtering Model	LLaVA
  GPT-4o	0.598
  LLaMA-3-8B	0.572

These results show that KCMP is robust to the choice of filtering models and does not rely heavily on specific high-capacity models. We will include these results in the revision.

Comment 3: Could the authors provide some cost estimation and comparison for their method?

We address this in three parts: (1) the optional nature of GPT-o1 filtering, (2) trade-offs between query repetition and performance, and (3) API cost estimation per image.

• 'GPT-o1 for filtering' is optional

  The GPT-o1 filtering module provides performance gains but is not required. Removal incurs only modest performance degradation:

  Method	LLaVA	LLaMA Adapter	MiniGPT4
  KCMP	0.598	0.573	0.544
  KCMP (no GPT-4o filtering)	0.572	0.553	0.536

• Query Count Reduction via Fewer Repeats

  By default, each probe is repeated R = 4 times. We evaluated the effect of using fewer repetitions on VL-MIA/Flickr:

  Method	LLaVA	LLaMA Adapter	MiniGPT4
  KCMP (R=4)	0.598	0.573	0.544
  KCMP (R=3)	0.579	0.558	0.536
  KCMP (R=2)	0.585	0.560	0.515
  KCMP (R=1)	0.548	0.531	0.519

  Reducing R from 4 to 2 causes a mild drop in AUC, indicating that fewer repetitions can be used to reduce cost.

• Cost Estimation

  The average number of probes per image is 4.12. Assuming R = 4, the total API calls per image is ~17. Each API query consists of a single image (~300×350 resolution) and a prompt of approximately 60 characters. Based on current public pricing, we estimate the per-image cost as follows:

  API Model	Image Price ($/image)	Text Price ($/1M tokens)	Estimated Cost($)
  GPT-4o-mini	0.001275	0.15	0.0218
  Gemini-2.5	0.001315	1.25	0.0014
  Claude-3.7	0.00042	3	0.0006

  These costs are manageable—especially in use cases like copyright enforcement or model misuse detection, where the value at stake per image far exceeds the query cost.

Comment 4: The authors claim that they "construct a benchmark dataset based on the DAM model" and evaluate different methods on three benchmarks. But I didn't find the new benchmark in the method or experiment sections.

Thank you for pointing this out. Our response is structured as follows: (1) we explain how the DAM-based benchmark was constructed, (2) how it was used in our main paper experiments, and (3) provide additional baseline results.

• Benchmark Construction

  We created a dedicated benchmark using the DAM model [4], whose training data and segmentation annotations are publicly available. The goal was to build a transparent and controlled testbed.

  • 200 member samples: drawn from DAM's released COCOStuff training set [5]
  • 200 non-member samples: sampled from COCOStuff, ensuring no overlap with DAM's training data

• Benchmark Usage in Experiments
  In Section 4.3 from main paper, we used DAM's segmentation annotations to study how region-level training annotations affect probe question effectiveness (Figure 4).

• Additional Baseline Evaluation We also benchmarked several MIA methods the dataset:

  Method	AUC
  Perplexity (inst)	0.702
  Perplexity (desp)	0.645
  Max-Prob-Gap (inst)	0.601
  Max-Prob-Gap (desp)	0.622
  Min-K% (inst)	0.743
  Min-K% (desp)	0.674
  Image Infer (Rouge)	0.502
  Image Infer (MPNet)	0.489
  KCMP	0.584

We will revise the manuscript to include benchmark details and add baseline comparisons.

Comment 5: What does the K mean in the experiments?

Thank you for pointing this out. To clarify the distinction:

• $K$ in Table 1 and Experimental Results:
  This denotes the dataset size used in set-level membership inference. Specifically, it indicates the number of suspected samples included in each candidate set.

• $K$ in "Min-K% Prob" and Similar Methods:
  This represents a hyperparameter that specifies the percentage of lowest-probability tokens (e.g., bottom 10%) used in computing the membership score for a single sample.

We will revise the manuscript to clarify this and avoid ambiguity in the notation.

Comment 6: The authors emphasize that "mainstream LVLMs" generally only expose generated outputs. But I feel "mainstream" is kind of subjective. Currently, some popular LVLMs such as LLaVA, Qwen, InternVL and so on are open-source, and some proprietary models like GPT-4 also provide part of logit outputs via API. Thus, the authors are recommended to slightly modify their arguments.

Thank you for the helpful feedback. We understand that the term "mainstream" may appear subjective and will revise our wording for clarity.

Our key point is that many widely used LVLMs—especially commercial APIs—restrict access to model internals, making existing gray/white-box MIA methods inapplicable due to a misalignment between what these methods require and what commercial APIs provide:

• What gray/white-box MIAs need:
  Traditional MIA methods estimate a model's confidence in generating a test sample by computing scores (e.g., generation loss [6]), which requires access to logits—typically needing gray/white-box access. For images and LVLMs (text output), recent gray-box methods [7] first generate a description from the target model, then construct an input sequence, i.e., image + prompt (instruction + generated description) and compute scores using logit slices from different parts of the input sequence:

  • Image logits: Logits slice extracted from the complete logit sequence corresponding to the image embeddings.
  • Prompt logits: Logits over prompt tokens.

• What commercial APIs expose:
  Most commercial LVLM APIs do not expose token logits. Even in cases where partial logits are exposed (e.g., GPT-4o provides top 5 token logits per position), they are limited to the generated output tokens only, which are incompatible with existing MIA methods.

We surveyed several commercial LVLM APIs and summarized their accessibility below (we omit GPT-4 which you mentioned as it does not support image input):

We will revise the manuscript to clarify this and explicitly highlight our focus on the black-box threat model, which better reflects real-world deployments.

Comment 7: Typos

Thank you for flagging typos and formatting issues. We have corrected them and carefully proofread the entire manuscript to ensure clarity and consistency in the revised version.

[1] Wavnet—visual saliency detection using discrete wavelet convolutional neural network."
[2] Medical sam adapter: Adapting segment anything model for medical image segmentation."
[3] A convolutional neural network-based chinese text detection algorithm via text structure modeling."
[4] Describe anything: Detailed localized image and video captioning."
[5] https://huggingface.co/datasets/nvidia/describe-anything-dataset
[6] Copyright traps for large language models."
[7] Membership inference attacks against large vision-language models.

We appreciate the thoughtful review and the noted strengths on presentation, problem motivation and results of our paper. Our response to the queries and comments is as follows.

Question 1 and Comment (5): While the paper is motivated using frontier models that we only have black box access to, there are no results using these models. Now of course, the advantage of using white box models as the authors do is that we know what data they were trained on and what they were not, which is not true for closed source models, thus we cannot have collect AUROC of the method on closed source models. With that being said, I wonder if it is possible to use image data released after a closed source models release as "not in training set", and some very common image data on the internet, e.g. imagenet, as "assumed to be in the training set". Reporting even just the average detection score on these two datasets for closed source models would be of value.

Thank you for your valuable suggestion. We have constructed an image-based MIA dataset targeting closed-source models to demonstrate the effectiveness of our method.

Dataset Construction: Following the comments, we constructed the image MIA dataset by leveraging the knowledge cut-off dates of the target models. The table below presents the release dates and knowledge cut-off dates of two representative API-based LVLMs. Based on this information, we identified Flickr—a widely used data source for LVLM training—and selected images uploaded after July 1, 2024 as candidate non-members (i.e., likely unseen during training), while images uploaded before this date were considered members. We curated a pilot benchmark consisting of 200 images (100 members and 100 non-members) for preliminary evaluation. This dataset will be publicly released following the rebuttal period.

Performance Evaluation: We conducted experiments on the curated benchmark under realistic API constraints. The AUC results are summarized in the table below. This evaluation highlights the practical effectiveness of our method in detecting membership for closed-source models. We will include both the newly constructed dataset and the corresponding results in the revised manuscript. Additionally, we plan to incorporate more closed-source models (e.g., Claude) into the evaluation to further strengthen our analysis.

Question 2: In table 1, performance increases with K. Why do you only test to k=30? I seems that you have not hit diminishing returns at K=30? If this is true, why not increase K further?

Thank you for the question. We clarify our choice of $K$ as follows:

$K$ defines an experimental setting, not a tunable hyperparameter: $K$ reflects the size of the candidate set in the evaluation and is not a parameter of our method. As expected, larger $K$ improves performance by aggregating more membership signals. We include additional results for other $K$ values below:

• Implication of $K$ in real-world scenarios
  In practice, users (e.g., creators) often hold large sample collections (hundreds or thousands), enabling larger $K$ and thus offering strong potential for high real-world performance.

Question 3 and Comment (6): The baselines at the bottom of page 6 could be explained in more detail (you could link to an appendix section to do this). In particular, it is not clear to me what prompts you use for the baselines.

Thank you for the helpful suggestion. We have added a section in Appendix E detailing the prompts used for KCMP and will include a dedicated section clarifying the baselines. Below is a brief explanation due to space constraints:

• Gray-box baselines For each image, these methods [1] generate a description using the prompt 'Describe this image', then construct an input sequence with the image, the prompt, and the description. Characteristic scores are computed from logit slices across different input regions during generation. We reproduce these baselines using the original open-source code to ensure fair comparison.

• Black-box baselines The only black-box baseline (Image Infer in Table 1) [2] measures the consistency of model outputs by computing the mean similarity among repeated descriptions of the same image. We use the same prompt (“Describe this image”) as in the original paper.

[1] Li, Zhan, et al. "Membership inference attacks against large vision-language models." Advances in Neural Information Processing Systems 37 (2024): 98645-98674.
[2] Hu, Yuke, et al. "Membership inference attacks against vision-language models." arXiv preprint arXiv:2501.18624 (2025).

Thank you for your thoughtful rebuttal!

The dataset you have constructed seems useful and the results on closed source models are good. The AUROC scores are lower than those seen for open source models in the paper, if I understand correctly? Could you elaborate as to why this is? Possibly I have missed some detail. Nonetheless the performance is above random, showing a promising technique.

Thank you for clarifying the point concerning the K value, I understand now.

Thank you for adding Appendix E. Adding reference to this in the main body will be helpful to some readers.

Overall I will stay with my current score of 5, and argue for the paper to be accepted. Thank you for your time and effort in creating this valuable piece of work.

Thank you again for your valuable comments and insightful suggestions. They have significantly contributed to improving the quality of our work.

We have now extended our evaluation to include additional closed-source model: Claude-3. The updated results are summarized below:

We will incorporate these results into the revised version of our paper.

We appreciate the positive feedback and the recognition of our work’s strengths. Our response to the comments is as follows.

Comment 1: Influence of Proxy Model Selection: The KCMP framework utilizes a proxy model for the prior knowledge calibration module, which plays a critical role in disentangling memorization from generalization. How sensitive is the proposed method to the choice of proxy model? Specifically, how does the inference performance vary when different language models are employed for knowledge calibration? A discussion or empirical analysis on this aspect would help assess the generalizability of the method.

Thank you for raising this important question. We conducted an ablation to evaluate the sensitivity of KCMP to the choice of proxy model used in the prior knowledge calibration module. Our findings are summarized below:

• Impact of Proxy Model Choice

  We replaced GPT-4o with a lighter open-source model (LLaMA-3-8B) for knowledge calibration and measured the impact on KCMP performance on the VL-MIA/Flickr dataset using LLaVA as the target model.

  Used Proxy Model	LLaVA
  GPT-4o	0.598
  LLaMA-3-8B	0.572

  The above results suggest that KCMP is robust to the choice of filtering models and does not rely heavily on specific high-capacity models. We will include additional ablation results with other target models and datasets in the revised manuscript to further support this claim.

Comment 2: Comparisons with Adapted Gray-box Variants: While the black-box baseline adopted in the study is understandably simple due to the limited literature on black-box membership inference for multimodal models, it would be valuable to explore whether existing gray-box methods can be adapted for black-box settings by removing internal access. Have the authors considered implementing such adapted variants as enhanced baselines? If so, how does KCMP perform in comparison? Including or discussing such enhanced baselines would better highlight the advantages of the proposed method.

We appreciate this valuable suggestion. Below, we (1) discuss the design of black-box variants adapted from gray-box methods, and (2) present experimental results comparing them with KCMP.

• Adapted Black-box Variants from Gray-box Methods

  Directly removing internal access (e.g., logits) from gray-box methods renders their scoring functions inapplicable, as such methods rely heavily on internal signals. To address this, we designed two black-box variants based on ideas from prior gray-box MIA literature:

  1. Neighbourhood (adapted from [1])

     The original method computes loss differences between original and perturbed inputs, assuming member data lie in local minima.
     In our adaptation, we generate three perturbed versions of each test image (e.g., via rotation or cropping), concatenate them with the original, and prompt the model to rank them by familiarity. The rank position of the original image is used as the MIA score.

  2. Entropy (adapted from [2])
     The original method uses loss or token-level confidence to distinguish members.

     We compute the entropy of the output token distribution as a proxy for confidence, based solely on the generated text. The idea is that member samples produce more confident and focused responses, resulting in lower entropy.

• Experimental Results

  We evaluate both variants on two datasets and three target models. KCMP consistently outperforms these adapted baselines:

  VL-MIA/Flickr:

  Method	LLaVA	LLaMA Adapter	MiniGPT4
  Neighbourhood	0.524	0.492	0.518
  Entropy	0.537	0.506	0.489
  KCMP	0.598	0.573	0.544

  VL-MIA/DALL-E:

  	LLaVA	LLaMA Adapter	MiniGPT4
  Neighbourhood	0.495	0.503	0.493
  Entropy	0.511	0.526	0.485
  KCMP	0.565	0.568	0.543

  These results demonstrate that while some intuition from gray-box designs can be translated into black-box settings, the resulting methods show limited performance. KCMP significantly outperforms these enhanced baselines, underscoring its strength in strictly black-box scenarios.

[1] Mattern, Justus, et al. "Membership inference attacks against language models via neighbourhood comparison." arXiv preprint arXiv:2305.18462 (2023).
[2] Yeom, Samuel, et al. "Privacy risk in machine learning: Analyzing the connection to overfitting." 2018 IEEE 31st computer security foundations symposium (CSF). IEEE, 2018.

Comment 3: Clarification of the Prior Knowledge Calibration Module: The mechanism by which the rationality of alternatives is evaluated in the prior knowledge calibration module remains somewhat unclear. It would be helpful if the authors could elaborate on the specific criteria or scoring functions used in this step. Additionally, including a concrete running example throughout the pipeline may aid in illustrating the logic and decisions made during calibration.

Thank you for your helpful feedback. We clarify this question by (1) explaining the design objective of the Prior Knowledge Calibration (PKC) module, (2) detailing its implementation, and (3) providing a concrete running example to illustrate the workflow.

• Objective: The PKC module enhances membership inference reliability by filtering out semantically implausible or irrelevant confusers. This ensures that only rational, contextually valid alternatives are retained, which is critical for generating meaningful contrast and amplifying membership signals.

• Evaluation of Alternative Rationality: The plausibility of each confuser is assessed through a two-stage LLM-based process:

  1. Contextual Textualization: Each masked image is converted into a natural language description, which captures the visible context of the image while disentangling model predictions from low-level visual patterns. It provides a textual prior that allows language models to reason about what could plausibly occupy the masked space.
  2. Plausibility Scoring: Confuser candidates (objects or colors) are then scored for contexual plausibility. A candidate is retained if judged plausible (“Yes”); otherwise, it is discarded and regenerated, up to a fixed number of attempts.

  This process ensures that only diverse, plausible, and semantically coherent alternatives are included in the final candidate set.

• Illustrative Workflow: Given a masked image with the hidden object labeled as “dog”:

  • The LLM first checks if “dog” is too obvious based on the masked image description (e.g., leash and walking woman); if so, this probe is skipped.
  • Valid distractors (e.g., “cat”, “scooter”, “child”) are then evaluated for plausibility.
  • Implausible candidates (e.g., “scooter”) are replaced via LLM-based filtering until three plausible alternatives are selected.

We will revise the paper to include this clarification and add a full example in the supplementary material.

We thank the reviewer for careful attention to the paper. Our response to the queries and comments is as follows.

Comment 1: Heavy auxiliary tool and compute cost – The attack depends on SAM (segmentation), CLIP (saliency ranking), and GPT-4o-style reasoning, which inflates GPU requirements and complicates large-scale scanning.

Thank you for the thoughtful comment. The auxiliary tools mentioned are primarily employed to further enhance detection performance. However, our method remains effective even in the absence of these tools or when they are replaced with lighter alternatives. The following analysis of both detection accuracy and efficiency substantiates this claim.

The performance of simplified KCMP. We construct a simplified variant of KCMP by applying the following three modifications: (1) removing the CLIP-based saliency ranking module, (2) removing the prior knowledge calibration component (i.e., GPT-4o-style reasoning), and (3) replacing the GPT-4o probe generator with a lightweight model (Qwen2.5-VL-7B). Experiments conducted on two datasets and across three models demonstrate that the simplified version achieves performance comparable to the original KCMP, while significantly outperforming intuitive black-box baselines. (The results for the baselines and the original KCMP method are reported in Table 1 of the paper.)

Efficiency Analysis: We next analyze the time and GPU resource requirements of our method. The simplified KCMP only relies on a segmentation model (e.g., SAM) and a lightweight LVLM (Qwen2.5-VL-7B) for generating probing questions. All computations can be executed on a single NVIDIA RTX 5000 GPU (32 GB VRAM), indicating that our method does not impose demanding hardware requirements.

Furthermore, the results presented in the following table show that the average time required to detect a single image is approximately 10.2 seconds. Given that membership inference attacks are typically performed offline and that the number of images involved in real-world detection tasks is often small- to moderate-scale (e.g., fewer than 10,000), we argue that our method is practically applicable.

Additionally, as a black-box approach, our method does not require local deployment of the target LLM. This feature significantly reduces the detection cost, particularly in terms of device-related expenses, and highlights the potential of our method for scalable application with large-scale target LVLMs.

We will incorporate these discussions and results into the revised manuscript. In addition, we will evaluate a further simplified variant of our method by replacing the SAM model with alternative lightweight segmentation models, in order to further demonstrate the efficiency potential of our approach.

Comment 2: High query overhead per image – Roughly 12 fine-grained probes, each repeated R times, are needed for a single membership decision; such dozens-of-queries overhead is expensive on metered APIs and increases the likelihood of detection or throttling.

Thank you for this valuable comment. We address it from the following four aspects: (1) Clarification of the actual number of probes used in our method. (2) Evaluation of detection performance under varying probe sizes or repetition times. (3) Estimation of the associated API costs.(4) Discussion of potential defense strategies.

Actual Number of Probes Used for Detection: In our paper, we analyze the impact of probe size on detection performance in Figure 4. We also mention "up to 12 probes" in the related discussion to refer to the range of probe sizes used for performance analysis, rather than the actual hyper-parameter setting—an expression that may have led to misunderstanding. In practice, we set the maximum number of probes for detection to 5. The table below further presents the distribution of probes generated per image, showing that the average number of probes actually used during detection is approximately 4.12.

Impact of probe sizes or repetition times: Figure 4 of our paper illustrates the impact of the number of probes on detection performance, demonstrating that using approximately five probes yields optimal results for our method. Furthermore, we evaluate the detection performance under a fixed probe size of five and varying query repetition times ($R = 1, 2, 3, 4$). The results indicate that reducing the repetition count from 4 to 2 leads to only a modest decline in detection performance, suggesting that fewer repetitions can be adopted to effectively reduce the API query overhead without significantly compromising accuracy.

API Costs:In our method, the average number of probes generated per image is approximately 4.12. Assuming a query repetition count of $R = 4$, this results in an estimated 17 API calls per image. Each API query involves a single image (with a resolution of approximately 300×350 pixels) and a prompt of around 60 characters, along with the generation of several response tokens (e.g., choice outputs). Based on current pricing for SOTA models, we provide a detailed cost estimation in the table below. The results indicate that the cost of detecting 1,000 images targeting Claude-3.7 is approximately 0.60, demonstrating the practicality and cost-efficiency of our method. Moreover, by reducing the repetition count $R$ to 2, which has minimal impact on detection performance as previously discussed, the total cost can be further halved, enhancing scalability in real-world applications.

Discussion of potential defense strategies: Although our method issues multiple queries to the target LLM for a single image, these queries are carefully designed to resemble genuine user inputs—such as requests for image descriptions or responses to natural questions—rather than malformed or adversarial content. Consequently, potential defense mechanisms targeting our approach may also incur substantial false positives on benign user queries, thereby degrading the overall user experience. Moreover, we acknowledge that no method can be entirely future-proof against all possible defenses. We will include a detailed discussion of this limitation in the revised manuscript.

Comment 3: Limited evaluation scope – Validation is confined to MiniGPT-4, LLaVA-1.5, and LLaMA-Adapter v2. No experiments demonstrate resilience under commercial endpoints (GPT-4o, Gemini, Claude) with real-world rate limits or safety filters; even a small-scale test on such services would strengthen the claim of practical applicability. There are some unlearning benchmark can be leveraged.

Thank you for your valuable suggestion. We have constructed an image-based MIA dataset targeting closed-source models to demonstrate the effectiveness of our method.

Dataset Construction: Following the comments from you and Reviewer 9Jf3, we constructed the image MIA dataset by leveraging the knowledge cut-off dates of the target models. The table below presents the release dates and knowledge cut-off dates of two representative API-based LVLMs. Based on this information, we identified Flickr—a widely used data source for LVLM training—and selected images uploaded after July 1, 2024 as candidate non-members (i.e., likely unseen during training), while images uploaded before this date were considered members. We curated a pilot benchmark consisting of 200 images (100 members and 100 non-members) for preliminary evaluation. This dataset will be publicly released following the rebuttal period.

Performance Evaluation: We conducted experiments on the curated benchmark under realistic API constraints. The AUC results are summarized in the table below. This evaluation highlights the practical effectiveness of our method in detecting membership for closed-source models. We will include both the newly constructed dataset and the corresponding results in the revised manuscript. Additionally, we plan to incorporate more closed-source models (e.g., Claude) into the evaluation to further strengthen our analysis.

We hope that our response adequately addresses your concerns and respectfully invite you to reconsider your initial evaluation. We remain fully committed to addressing any further comments or questions you may have.