ImageSentinel: Protecting Visual Datasets from Unauthorized Retrieval-Augmented Image Generation

We appreciate Reviewer yaqi's insightful feedback and specific suggestions.

[W1] Identifying duplicate images in the dataset, and finding representative images that are unique based on the type, description, characteristics or even with perceptual similarity using DINO and filtering the duplicates or over-represented content.

Thank you for raising this important concern about duplicate content filtering. We conducted additional experiments to evaluate this potential attack strategy.

1. Attack Implementation

We implemented a duplicate filtering attack using DINO similarity scores. For each image pair, we compute their DINO similarity and remove images exceeding a threshold $\delta$, prioritizing the removal of images containing random character-like patterns.

2. Performance

We evaluated various filtering thresholds $\delta$ (0.5-0.9) across two datasets with 10 queries for detection:

LLaVA Visual Instruct Dataset:

Product-10K Dataset:

3. Analysis:

• Lower $\delta$ thresholds (0.5-0.6) can filter out sentinel images and cause low detection accuracy.
• However, they lower $\delta$ thresholds also remove an unacceptable proportion of legitimate images (62.8-96.2%). This severely compromises the dataset's utility. Especially for Product-10K dataset, it suffers more severe filtering due to its inherent nature of containing visually similar product images.
• Setting a high $\delta$ (0.8-0.9) preserves more legitimate content but fails to effectively filter sentinel images, as evidenced by the high detection scores.
• Our method's resilience against similarity-based filtering is largely attributed to our pipeline shown in Figure 3. Instead of directly feeding reference images into GPT-4o for sentinel image generation, we first perform attribute extraction followed by language-guided generation. Direct reference-guided generation would result in substantial content overlap, while our language-guided approach maintains stylistic consistency while avoiding identical content duplication. This design ensures that sentinel images share semantic and stylistic elements with the reference images while maintaining sufficient visual diversity to survive similarity-based filtering.

We will include these quantitative results and visual examples in our revised paper to better illustrate our findings.

[W2] Another approach could be leveraging OCR models to detect the text from images and filtering out images with random character sequences or masking such image regions.

We appreciate this insightful comment about detection evading techniques. We have conducted experiments to evaluate potential attacks that attempt to detect and neutralize sentinel images.

1. Attack Implementation

We implemented two attack strategies using OCR-based random character detection:

• Filtering: removing detected sentinel images
• Masking: obscuring detected text regions

2. OCR Detection Performance

We use three widely-used OCR systems (EasyOCR, Tesseract OCR, and PaddleOCR) on two datasets, and apply the PyEnchant library [1] to determine whether the detected text contains random character sequences. The results are shown below.

While OCR-based attacks are possible to filter most sentinel images, the high false positive rates on clean images indicate that such attacks would significantly impair RAIG's normal functionality. This is particularly problematic for datasets like Product-10K, where a substantial portion of legitimate images containing product names, URLs, or watermarks would be incorrectly filtered out.

[1] Ryan, K. PyEnchant: a spellchecking library for Python.

3. Impact Analysis

Detection performance under EasyOCR-based attacks (LLaVA Visual Instruct dataset):

Generation quality impact (300 samples):

• These results demonstrate that while attacks can initially degrade detection effectiveness, this can be mitigated by increasing query numbers. For example, under filtering attacks, the AUC drops to 0.63 with 5 queries, but recovers to 0.97 with 100 queries.
• Such attacks significantly compromise RAIG's generation quality, making them impractical for real-world deployment. Under attacks, the CLIP scores drop from 0.772 to around 0.531 (Filtering attack) and 0.559 (Masking attack), while SigLIP scores decrease from 0.743 to around 0.53 (Filtering attack) and 0.55 (Masking attack), representing a degradation in generation quality.

We will discuss these potential detection evading techniques in an expanded limitations section, along with detailed experimental results. Future work could explore more sophisticated attack strategies and corresponding defense mechanisms to further validate the robustness of our protection framework.

Thank you for running the adaptive attacks. I have a few clarifying questions and suggestions:

1. Clean-image filtered: When authors report that X % of “clean” images are filtered out, are those images share the same attribute values as the retained images? It would be helpful to measure the attribute-space coverage of the surviving dataset—ideally using the same attribute taxonomy you employed when synthesizing sentinels. This would show whether the remaining images still form a representative sample of the original distribution. Reporting only the raw percentage removed makes it hard to judge the real utility loss. This coverage analysis applies to both the DINOv2-similarity filter and the OCR-based detection.

2. Text masking / rewriting: In practice, a RAIG provider could

• Offline phase: detect and in-paint text regions, or regenerate text-free variants with an image-to-image diffusion model before building the retrieval index.

• Online phase: If the retrieved image still contains text, perform a lightweight second-pass in-painting (or apply a targeted “rewrite-the-text” diffusion prompt), then repeat retrieval to ensure the result matches the query description before returning the final output.

These steps allow the RAIG system to preserve retrieval quality while neutralizing sentinel keys.

3. Masking evaluation: When evaluating the performance of masking, different queries were used. However, such masking could be performed by the RAIG provider to refine the dataset before it is indexed for retrieval.

Overall, I appreciate the thorough evaluation, but I believe an explicit attribute-coverage metric would clarify the true loss of dataset utility, and RAIG providers could automate text removal or rewriting both offline and online.

Please analyze these two images and answer with ONLY "Yes" or "No":
Are these two images either near-duplicates (very similar visual content) OR do they share highly similar attributes (same subject type, style, and composition)?

Rules:
1. Near-duplicates means: same scene/object from similar angle with minimal variations
2. Similar attributes means: same subject type + same visual style + similar composition
3. Only answer "Yes" if either condition is met
4. Only answer "No" if neither condition is met
5. Do not explain your reasoning, only answer Yes/No

Thank you for the prompt response. Based on the additional experiments and clarifications, I have few comments and suggestions:

Attribute-space coverage: I will accept the results with existing classification system in Products-10K dataset to analyze the impact on class distribution for the attribute coverage aspect. This result make it clear that the similarity based approaches fail to retain useful and unique characteristics images in the dataset, and this is not an effective attack strategy for this protection.

Detect-and-in-paint strategy: I appreciate the new experiment demonstrating that automatically detecting and in-painting text regions can neutralize sentinel images while preserving generation quality. Expanding this test to a larger scale in the next revision would strengthen the paper.

Selective text removal: Authors note that removing brand names can be undesirable. It may be worth acknowledging that an adversary could maintain a vocabulary of critical terms like brand names to avoid erasing them. It is an incremental yet realistic evasion tactic.

Online phase: I agree with your statement. I realized this aspect earlier today and edited my comment accordingly. Nevertheless, the same response from offline phase can also apply in this scenario.

Masked evaluation: The explanation you provided for the masked evaluation protocol is clear and addresses my question.

Overall, the “detect-and-in-paint” approach appears to be the most credible attack strategy in a grey-box scenario where the protection method is disclosed. My evaluation is not based on whether the protection is easily evaded but to openly discuss such limitations to enhance paper’s transparency and practical value. I believe that including this attack analysis in the revised version would strengthen the contribution.

With these points satisfactorily covered, I will revise my recommendation to borderline accept.

Thank you again for your thorough and thoughtful engagement.

2.1. Offline phase: detect and in-paint text regions, or regenerate text-free variants with an image-to-image diffusion model before building the retrieval index.

Following your suggestion, we implemented an attack using EasyOCR+Stable Diffusion 2.0 inpainting (Inpainting Attack). We evaluated the generation quality after applying this attack on a subset of the LLaVA dataset (1000 images). The results are as follows:

The results show minimal impact on generation quality, indicating that this attack can maintain generation performance while potentially compromising protection. Results from larger-scale dataset testing will be included in the next version of the paper.

We greatly appreciate this suggestion, as it highlights what we consider to be a more effective attack method. Yet it's worth noting that this attack approach carries a limitation: important text elements such as brand names might be mistakenly identified and removed, which would impair legitimate brand-based retrieval and generation. While simply increasing the number of sentinel images could help, as shown in our last response, more sophisticated protection mechanisms would be needed to defend against such attacks effectively.

2.2. Online phase: if a retrieved or generated image still contains text, run a lightweight second-pass in-painting step (or a targeted "rewrite-the-text" diffusion prompt) before returning the final output.

This proposed attack strategy would be ineffective against our protection mechanism. Our detection mechanism does not rely on rendering text in the generated images. Instead, we detect unauthorized use by measuring visual similarity between RAIG-generated images and our sentinel images. While text plays a role during the retrieval phase, post-retrieval text manipulation cannot evade detection because our generation phase only requires the semantic content of retrieved images to generate outputs containing sentinel visual elements, which we then use for detection.

3. Masking evaluation: When evaluating the performance of masking, different queries were used. However, such masking could be performed by the RAIG provider to refine the dataset before it is indexed for retrieval.

We would like to ensure we have correctly understood your concern. In our experiments, we did process the dataset before generation, including both the original images and embedded sentinel images. The different queries mentioned in our evaluation refer to the detection process, where each query represents one instance of inputting our designed key to the black-box RAIG system to obtain output images.

We deeply appreciate your suggestions and acknowledge the critical importance of designing effective attacks. These potential risks should be communicated to readers and users to ensure transparency. We also believe understanding vulnerabilities is essential for developing more robust protection mechanisms. We greatly value this discussion and welcome any further questions or concerns during this discussion phase.

We thank Reviewer qQTS for the thorough review and constructive suggestions.

[W1] Statistical Strength of Detection Evidence

We appreciate the suggestion to strengthen our statistical evidence and have conducted additional analyses.

1. Extended Evaluation Metrics

We follow practices in detection evaluation [1-4] to use TPR@1%FPR in our experiments. We acknowledge the need for more stringent statistical guarantees for legal applications. We extended our evaluation to include TPR@0.1%FPR, providing stronger statistical confidence aligned with recent works [5,6].

2. Results

We keep the same experimental setting as Table 3 in our paper.

Our method maintains high detection rates even at FPR=0.001.

We will expand Table 3 in the main paper to include these more stringent metrics.

[1] Wen, Y. et al. "Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust." NeurIPS 2023.

[2] Dathathri, S. et al. "Scalable watermarking for identifying large language model outputs." Nature 2024.

[3] Pan, L. et al. "MarkLLM: An Open-Source Toolkit for LLM Watermarking." EMNLP 2024.

[4] Liu, Y. et al. "Dataset Protection via Watermarked Canaries in Retrieval-Augmented LLMs." ICML 2025 Workshop R2-FM.

[5] Pang, Q. et al. "No free lunch in LLM watermarking: Trade-offs in watermarking design choices." NeurIPS 2024.

[6] Kirchenbauer, J. et al. "On the Reliability of Watermarks for Large Language Models." ICLR 2024.

[W2.(a), Q1] Exact number of sentinel images

In our experiments, as shown in Table 3, S3, and S4, the number of queries is equal to the number of sentinel images used, as each query is designed to trigger a specific sentinel image.

Specifically, our experiments used:

• Table 3: 3-20 sentinel images in 10,000 references (0.03%-0.2%)
• Table S3: 3-30 sentinel images in 10,000-100,000 references (0.003%-0.3%)
• Table S4: 1-20 sentinel images in 30,000 references (0.003%-0.07%)

These sentinel images were randomly selected from a pool of 2,800 private images. The small ratios help ensure the stealthiness of our protection mechanism. We will explicitly clarify these numbers in the revised paper.

[W2.(b)] Clarity of Performance Metrics Calculation

We explain the process of our trials and TPR/FPR calculation as follows.

Trial Process

For each trial with N queries:

1. Positive Sample Generation:

• Randomly select N images from our private dataset.
• Convert them into sentinel images.
• Embed them into the reference database.
• Generate N images using their corresponding N key queries.
• Calculate the mean DINO similarity between generated images and their corresponding sentinel images.

1. Negative Sample Generation:

• Use the same N queries but with the original reference database (without sentinel images).
• Generate N images and calculate mean DINO similarity similarly.

Evaluation Details

• We conduct 100 trials for positive samples.
• For robust threshold estimation, we conduct 2000 trials for negative samples.
• TPR@1%FPR is calculated by:
  1. Setting detection threshold at 1% percentile of negative samples' similarities.
  2. Computing the fraction of positive samples exceeding this threshold.

We will add these detailed evaluation settings to our revised paper.

[W3.(a)] Vulnerability to Database Filtering

We have conducted experiments to evaluate potential attacks that attempt to detect and filter sentinel images.

1. Attack Implementation

We implemented two attack strategies using OCR-based random character detection:

• Filtering: removing detected sentinel images.
• Masking: obscuring detected text regions.

2. OCR Detection Performance

We use three widely-used OCR (EasyOCR, Tesseract OCR, and PaddleOCR) on two datasets, and apply the PyEnchant library [1] to determine whether the detected text contains random character sequences. The results are shown below.

While OCR-based attacks are possible to filter most sentinel images, the high false positive rates on clean images indicate that such attacks would significantly impair RAIG's normal functionality. This is particularly problematic for datasets like Product-10K, where a substantial portion of legitimate images containing product names, URLs, or watermarks would be incorrectly filtered out.

[1] Ryan, K. PyEnchant: a spellchecking library for Python.

3. Impact Analysis

Detection performance under EasyOCR-based attacks (LLaVA dataset):

Generation quality impact (300 samples):

These results demonstrate that while attacks can initially degrade detection effectiveness, this can be mitigated by increasing query numbers. However, such attacks significantly compromise RAIG's generation quality, making them impractical for real-world deployment.

We will discuss these potential adaptive adversaries in an expanded limitations section, along with detailed experimental results. Future work could explore more sophisticated attack strategies and corresponding defense mechanisms to further validate the robustness of our protection framework.

[W3.(b)] Vulnerability to Prompt Filtering

We have conducted additional experiments to evaluate our approach's robustness against prompt preprocessing:

1. Prompt Stripping Attack

We implemented a prompt stripping attack that removes random character sequences in the queries.

Our experiments on the Product-10K dataset (5 queries) show:

The results show that prompt stripping can effectively degrade our detection performance.

2. Practical Challenges

While prompt stripping significantly impacts detection performance, implementing such filtering in RAIG systems faces serious practical challenges:

• Many legitimate use cases require precise alphanumeric inputs (product codes, brand names).
• Our analysis on Product-10K identified numerous legitimate brand names that would be incorrectly filtered, such as "Royalstar", "YASEE", "IPASON", "LIEYANZHANDUI", "kocotree", "Aocilenda", and "NUTRICIA".
• Such filtering would significantly degrade RAIG system utility in e-commerce and technical applications.

We will include a detailed discussion of these limitations in our revised paper.

[How the method might handle W3.(a) and W3.(b)]

We propose two potential strategies to mitigate the vulnerability to OCR-based filtering and prompt stripping:

1. Using Meaningful Characters as Keys. Instead of random character sequences, we could use meaningful words or phrases as sentinel keys. However, this approach faces two challenges:

• Finding the right balance between protection effectiveness and normal user experience, as legitimate users might naturally use these meaningful keys in their queries
• The meaningful keys might retrieve images related to their semantic meaning rather than images containing the actual text, potentially reducing detection accuracy

2. Embedding Keys in Natural Contexts. We could embed random character keys in natural-looking contexts, such as "VasWiW brand cup" or "VasWiW restaurant". This approach could help bypass prompt filtering as these queries appear more legitimate.

We plan to explore these mitigation strategies in our future work.

[Q2] Unintended Character Generation in Normal Usage

In Section 5.2 "Generation quality preservation", we investigated whether using protected images would affect normal generation capability. Among all queries, 8.3% retrieved sentinel images. We analyzed all generated images from these queries and found:

1. 36% of these images generated by SDXL-based RAIG after retrieving sentinel images did contain text characters.

2. These characters were not direct copies of the retrieval keys embedded in the retrieved sentinel images. Instead, they appeared with significant modifications and transformations, making the original characters unrecognizable.

3. The characters were naturally integrated into the generated scenes, appearing as store signs, ship decorations, or wall graffiti. This integration did not noticeably impact the visual quality or coherence of the generated images.

In this test, we intentionally increased the number of sentinel images to 300 to evaluate the generation capability. In practice, only a small number of sentinel images (1-20 images in a reference dataset of 10,000 images) are needed for protection, which would reduce the likelihood of retrieving sentinel images during normal generation.

We will include these visual examples and a discussion in our revised paper.

We thank Reviewer LHUd for the rigorous analysis and valuable feedback on security concerns.

[W1, Q1] Defense against adaptive adversary: No evaluation against attackers who could use OCR to filter out images with random text strings.

We have conducted experiments to evaluate potential attacks that attempt to detect and filter sentinel images.

1. Attack Implementation

We implemented two attack strategies using OCR-based random character detection:

• Filtering: removing detected sentinel images
• Masking: obscuring detected text regions

2. OCR Detection Performance

We use three widely-used OCR systems (EasyOCR, Tesseract OCR, and PaddleOCR) on two datasets, and apply the PyEnchant library [1] to determine whether the detected text contains random character sequences. The results are shown below.

While OCR-based attacks are possible to filter most sentinel images, the high false positive rates on clean images indicate that such attacks would significantly impair RAIG's normal functionality.

[1] Ryan, K. PyEnchant: a spellchecking library for Python.

3. Impact Analysis

Detection performance under EasyOCR-based attacks (LLaVA Visual Instruct dataset):

Generation quality impact (300 samples):

These results demonstrate that while attacks can initially degrade detection effectiveness, this can be mitigated by increasing query numbers. Such attacks also compromise RAIG's generation quality, making them impractical for real-world deployment.

We will discuss these potential adversaries in the limitations section, along with experimental results. Future work could explore more attack and defense strategies to further validate the robustness of our method.

[W2] Some baselines are not competitive. Comparison with more relevant and challenging baselines like membership attacks for RAG cited in related work.

Thank you for raising this important point about baselines.

1. Baseline Selection

Our work addresses dataset protection for Retrieval-Augmented Image Generation (RAIG) systems, where protection methods are still largely unexplored. Therefore, we explored adapting text RAG protection method (Ward [1]) to RAIG by creating baselines Ward-HiDDeN and Ward-FIN in our paper. We elaborate below on why other existing methods cannot be directly adapted to RAIG.

2. Challenges in Adapting RAG Methods to RAIG

We investigated the feasibility of converting existing RAG-related methods:

• MIA for RAG [2] ([25] in main paper): This method relies on text-based LLMs to answer "Yes" or "No" for membership inference. However, in RAIG, the image generator replaces the text-based LLM and can only generate images rather than provide binary answers.

• Ward [1] and [3]: These methods watermarks texts in the reference dataset and relies on the persistence of text watermarks in generated outputs for detection. Since our reference data consists of images, we adapted the text watermarking approach to image watermarking, creating baselines Ward-HiDDeN and Ward-FIN.

• S^2MIA [4] and MBA [5]: These methods rely on using masked target documents as input and observing accuracy of RAG-generated content for the masked portions to determine if the target document exists in the reference dataset. When adapting to RAIG, we would need to input masked target images. However, this is infeasible since RAIG accepts text rather than image inputs.

• Phantom [6]: This method depends on white-box attacks, which conflicts with our threat model's black-box assumptions.

• TrojanRAG [7]: This method requires attackers to have the ability to train the RAG system. However, since we directly use existing black-box models, this approach cannot be effectively adapted.

[1] Jovanović, N. et al. "Ward: Provable rag dataset inference via llm watermarks." ICLR 2025.

[2] Anderson, M. et al. "Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generation." ICISSP 2025.

[3] Liu, Y. et al. "Dataset protection via watermarked canaries in retrieval-augmented llms." arXiv 2025.

[4] Li, Y. et al. "Generating is believing: Membership inference attacks against retrieval-augmented generation." ICASSP 2025.

[5] Liu, M. et al. "Mask-based membership inference attacks for retrieval-augmented generation." WWW 2025.

[6] Chaudhari, H. et al. "Phantom: General trigger attacks on retrieval augmented language generation." arXiv 2024.

[7] Cheng, P. et al. "TrojanRAG: Retrieval-augmented generation can be backdoor driver in large language models." arXiv 2024.

3. Additional Baseline

To provide a more comprehensive evaluation, we implemented a semantic-based baseline that uses semantic descriptions as retrieval keys instead of character sequences. This baseline represents a more intuitive approach that leverages semantic descriptions to detect unauthorized dataset usage. Specifically, we:

1. Extract semantic descriptions from images.
2. Use these descriptions as retrieval keys to query the RAIG system.
3. Compare generated images with original images to detect unauthorized use.

As shown in Table 2 of main paper, most semantic descriptions (more than 60%) as retrieval keys bypass retrieval as their generators create satisfactory images directly, preventing detection of target images in the reference dataset.

We further evaluated this baseline on the LLaVA Visual Instruct Pretrain Dataset:

The results demonstrate that our approach outperforms the semantic-based baseline. We will include these results in the main paper for a more comprehensive comparison.

[W3, Q2] (W3) Detection mechanism validation: Need to verify whether high similarity comes from retrieval or direct key rendering. (Q2) Validating Retrieval vs Direct Rendering.

We appreciate these important questions about our evaluation methodology.

1. Analysis of Detection Mechanism

Direct key rendering would be insufficient for detection.

Our detection mechanism does not rely on rendering keys from prompts. Instead, we detect unauthorized use through measuring visual similarity between RAIG-generated images and our sentinel images. The design the key as random characters (e.g., "VasWiW") without any information about corresponding sentinel image's semantical attributes.

Therefore, it is theoretically impossible for generative models to recreate the specific visual characteristics of sentinel images solely from these random characters. The high similarity scores can only be achieved through successful retrieval of the sentinel images.

2. [Q2] Experiments for Direct Key Rendering

We conducted a controlled experiment comparing generation with retrieval enabled versus disabled (direct generation) using the same keys. We use 10 queries for LLaVA Visual Instruct Pretrain Dataset:

The results are expected since our keys are random character sequences with no semantic correlation to sentinel images' visual content, making it impossible for direct generation to reproduce specific visual elements without retrieval.

3. Definition of Triggering Rate

The 'triggering rate' is defined as the proportion of queries where a RAIG system initiates the retrieval process when given a key. This metric is important since some RAIG systems [1] may bypass retrieval if their generators believe they can directly generate satisfactory images. Our method ensures a high triggering rate by carefully designing the key.

Triggering rate is an intermediate metric used during our experimental analysis. In actual deployment (black-box setting), we only need to measure the semantic similarity between generated images and our sentinel images to detect unauthorized use. As shown in Table 3, this detection achieves high performance with black-box setting.

[1] Shalev-Arkushin, R. et al. "ImageRAG: Retrieval-Augmented Image Generation with Semantic Guidance." arXiv 2025.

[W4, Q3] Dependency of Defense Mechanism on Generator Capability.

We demonstrated that high similarity scores cannot be achieved through direct key rendering in response to [W3, Q2].

Our keys are random character sequences with no semantic correlation to visual content of sentinel images. Therefore, regardless of the generator's capability, it cannot directly render specific visual elements of sentinel images solely from these keys. The high similarity scores can only be achieved through the successful retrieval of sentinel images.

The requirement for RAIG systems is that the generator can incorporate semantic features from retrieved images into its output, which is a fundamental capability that functional RAIG system possesses.

prompt = (
    "Please determine if the following text (extracted by OCR) contains random/meaningless characters or gibberish. Consider it random if:\n"
    "1. It's not a common word, brand name, or recognizable abbreviation\n"
    "2. It has no clear pattern or structure\n"
    "3. It appears to be a random combination of letters\n\n"
    "Note: Ignore minor OCR mistakes (like '0' for 'O', '1' for 'I', etc.) and common typos. Only focus on whether the underlying text appears meaningful or random.\n\n"
    "Answer only with 'yes' if it's random, or 'no' if it has clear meaning or structure.\n\n"
    f"Text: {text}"
)

We sincerely thank Reviewer GDVj for the thorough review and constructive suggestions.

[W1] The evaluations are only performed on synthetic datasets; there is no assessment of threats from proprietary or fine-tuned RAIG systems in real-world scenarios.

For datasets:

Our evaluation covers both synthetic and real-world datasets:

1. Our main experiments used the LLaVA Visual Instruct Dataset [1], which contains real-world images from diverse sources and scenarios, including natural scenes, everyday objects, people, and more.
2. We also evaluated ImageSentinel on Product-10K [2] in the supplementary materials, a real-world commercial dataset with 30,000 product images including both in-shop photos and customer-generated images.

We will move more Product-10K experiments to the main paper to strengthen our evaluation by including more real-world e-commerce applications.

[1] Liu, Haotian, et al. "Visual instruction tuning." NeurIPS 2023.

[2] Bai, Yalong, et al. "Products-10k: A large-scale product recognition dataset." arXiv preprint 2020.

For RAIG systems:

1. We evaluated ImageSentinel on both proprietary and open-source RAIG systems: Proprietary system (GPT-4o) and Open-source systems (SDXL and OmniGen).

2. Following the common practice in retrieval-augmented generation research [3, 4] and widely-used frameworks like LangChain [5], we use pre-trained generators without fine-tuning. As RAIG is an emerging field, fine-tuned RAIG systems are not yet widely available in the community. We will extend our evaluation to fine-tuned variants when they become available.

[3] Shalev-Arkushin, Rotem, et al. "ImageRAG: Retrieval-Augmented Image Generation with Semantic Guidance." arXiv preprint 2025.

[4] Ru, Dongyu, et al. "RagChecker: A fine-grained framework for diagnosing retrieval-augmented generation." NeurIPS 2024.

[5] Mavroudis, Vasilios. "LangChain v0.3." preprint 2024.

[W2] The approach relies on black-box querying, requiring repeated queries to the suspected RAIG system, which may not be feasible in practical settings.

We appreciate this practical concern. We discuss the feasibility of our black-box query approach as follows:

1. Minimal Query Requirements

As shown in Table 3, S3 and S4 in the main paper and supplementary materials, our approach achieves efficient detection with very few queries. For example, only 5 queries are needed to reach 99.6% TPR at 1% FPR on Products-10k (total 30,000 reference images).

2. Established Practicality

Query-based approaches have been widely adopted in retrieval-augmented systems for backdoor attack detection [6], membership inference [7], and dataset protection [8]. Such querying overhead occurs only during investigation of suspected misuse, not as a continuous requirement.

[6] Chen, Zhaorun, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. "AgentPoison: Red-teaming LLM agents via poisoning memory or knowledge bases." NeurIPS 2024.

[7] Liu, Mingrui, Sixiao Zhang, and Cheng Long. "Mask-based membership inference attacks for retrieval-augmented generation." WWW 2025.

[8] Liu, Yepeng, Xuandong Zhao, Dawn Song, and Yuheng Bu. "Dataset Protection via Watermarked Canaries in Retrieval-Augmented LLMs." ICML 2025 Workshop R2-FM.

[W3, Q1] The authors should further extend their evaluation with adversarial analysis, specifically regarding the detectability of sentinel images; malicious users could potentially filter or recognize sentinel patterns.

We appreciate the reviewer's suggestion about adversarial analysis. We have conducted experiments to evaluate potential attacks that attempt to detect and filter sentinel images.

1. Attack Implementation

We implemented two attack strategies using OCR-based random character detection:

• Filtering: removing detected sentinel images
• Masking: obscuring detected text regions

2. OCR Detection Performance

We use three widely-used OCR systems (EasyOCR, Tesseract OCR, and PaddleOCR) on two datasets, and apply the PyEnchant library [1] to determine whether the detected text contains random character sequences. The results are shown below.

While OCR-based attacks are possible to filter most sentinel images, the high false positive rates on clean images indicate that such attacks would significantly impair RAIG's normal functionality. This is particularly problematic for datasets like Product-10K, where a substantial portion of legitimate images containing product names, URLs, or watermarks would be incorrectly filtered out.

[1] Ryan, K. PyEnchant: a spellchecking library for Python.

3. Impact Analysis

Detection performance under EasyOCR-based attacks (LLaVA Visual Instruct dataset):

Generation quality impact (300 samples):

• These results demonstrate that while attacks can initially degrade detection effectiveness, this can be mitigated by increasing query numbers. For example, under filtering attacks, the AUC drops to 0.63 with 5 queries, but recovers to 0.97 with 100 queries.
• Such attacks significantly compromise RAIG's generation quality, making them impractical for real-world deployment. Under attacks, the CLIP scores drop from 0.772 to around 0.531 (Filtering attack) and 0.559 (Masking attack), while SigLIP scores decrease from 0.743 to around 0.53 (Filtering attack) and 0.55 (Masking attack), representing a degradation in generation quality.

We will discuss these adversarial analyses in the limitations section, along with detailed experimental results. Future work could explore more sophisticated attack strategies and corresponding defense mechanisms to further validate the robustness of our protection framework.

[Q2] How would your approach perform if the RAIG system filtered or sanitized the random character prompts before retrieval (e.g., by stripping non-semantic input)? How much would the detection rate drop when such preprocessing is applied to the prompts?

We have conducted additional experiments to evaluate our approach's robustness against prompt preprocessing:

1. Prompt Stripping Attack

We implemented a prompt stripping attack that removes random character sequences in the queries.

Our experiments on the Product-10K dataset (5 queries) show:

The results show that prompt stripping can effectively degrade our detection performance.

2. Practical Challenges

While prompt stripping significantly impacts detection performance, implementing such filtering in RAIG systems faces serious practical challenges:

• Many legitimate use cases require precise alphanumeric inputs (product codes, brand names).
• Our analysis on Product-10K identified numerous legitimate brand names that would be incorrectly filtered, such as "Royalstar", "YASEE", "IPASON", "LIEYANZHANDUI", "kocotree", "Aocilenda", and "NUTRICIA".
• Such filtering would significantly degrade RAIG system utility in e-commerce and technical applications.

We will include a detailed discussion of these limitations and potential mitigation strategies in our revised paper.

[Q3] Can you clarify how many queries and how much compute/time is required for reliable detection? What are the limits of detection with fewer queries?

We thank the reviewer for this important question about the practical aspects of our detection method.

1. Query Requirements for Reliable Detection

As shown in Table 3, S3, and S4, ImageSentinel achieves reliable detection with very few queries. We list some results in Table S4 as:

To address the limits with fewer queries, we specifically tested the performance with just one query, which still achieves a reasonable AUC of 0.870 and TPR of 0.754 at 10% FPR, though with reduced reliability compared to using more queries.

2. Computational Cost

The main computational overhead comes from image generation time per query:

With parallel processing (common in real-world deployments), total detection time approximates one query duration. Without parallelization, total time equals per-query time multiplied by query number.

Given that reliable detection requires only 3-10 queries and supports parallel processing, the computational cost remains practical for practical applications.