Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Models

1. Difference between color attack classification vs. VLP

Thank you for the important question. Attacking VLP models differs significantly from attacking classifiers due to the following:

a）Cross-modal structure. VLPs generate captions or answers, requiring joint reasoning over vision and language. Attacks must target this multimodal alignment, not just label prediction.

b) semantic alignment instead of label prediction. VLP models are trained to align visual features with textual descriptions. Simply changing the image color may not disrupt this alignment, as the model can rely on robust, multimodal embeddings.

c) Increased Robustness. Pretrained VLPs like BLIP2 or CLIP are trained with large-scale vision-text pairs and are often more robust to simple perturbations. As we show in our experiments, existing color-based attacks (e.g., ColorFool, AdvCF) achieve limited effectiveness when directly applied to VLP tasks like captioning or VQA.

We address these challenges by:

a) Cross-modal attack objective. We design a cross-modal attack objective that specifically targets the semantic misalignment between relighted images and their textual outputs, something not addressed by classification-based color attacks.

b) GPT-based reference relighting. We adopt a reference relighting approach based on GPT accommodation, enabling more flexibility in generating semantically disruptive but visually coherent transformations.

c) Two-step collaborative optimization. It effectively expands the capabilities of our model by first optimizing lighting parameters and then refining the reference lighting image, achieving robust adversarial effects while preserving visual naturalness.

Generalization to classification tasks: We evaluate LightD on CLIP with ImageNet to test its applicability to the classification task. Compared with ALA, AdvCF, and ColorFool, LightD achieves significantly lower Top-1/Top-5 accuracy while maintaining better perceptual quality (NIQE):

2. Contribution

We respectfully disagree with the notion that our method is simply a "PGD attack with a novel initialization." While our optimization process shares the conceptual structure of gradient-based attacks, w would like to emphasize the contributions of our work from the following perspectives:

a) Novel problem setting. To the best of our knowledge, we are the first to explore non-suspicious adversarial attacks via natural relighting against VLPs. Previous works mainly target classification tasks using $L_p$-bounded noise.

b) Flexible framework. Our pipeline is modular and extensible; IC-Light and SGA can be swapped for other relighting or optimization techniques. The ChatGPT-guided initialization enhances semantic alignment, while the two-step optimization bridges visual quality and attack effectiveness.

c) Cross-modal integration. Though individual components may appear simple, combining them for cross-modal attacks (e.g., on captioning or VQA) is non-trivial. Our joint loss formulation (Eq. 7) enables this by balancing visual coherence with semantic misalignment.

3. Impact of ChatGPT

Thank you for your comment. While ChatGPT does not directly interact with the victim VLP models, we find that its role as a semantic prior generator is highly beneficial to the attack pipeline.

As shown in Table 3, lighting parameter initialization via ChatGPT consistently yields higher attack effectiveness across multiple VLP models compared to random initialization. In Figure 7, we further illustrate a qualitative example where the ChatGPT-based suggestion results in a more natural and semantically coherent relighting effect. These improvements stem from ChatGPT's strong understanding of scene structure, lighting conventions, and human visual perception.

1. NIQE result analysis on BLIP model

Thank you for pointing this out. NIQE is a widely adopted no-reference metric in the field and is also used by the baselines for fair comparison. However, it is not perceptually linear and may not be highly sensitive to subtle semantic-preserving perturbations. In some adversarial settings, small differences in NIQE scores can sometimes correspond to more noticeable perceptual variations or vice versa. Therefore, we also provide qualitative results to complete the quantitative results in the paper.

In certain VLP+dataset combinations (such as Flickr30K + BLIP), the captioning model’s semantic sensitivity may prompt our optimization strategy toward more aggressive relighting variations to ensure successful attacks. These changes, while still visually acceptable, may deviate from the "natural statistics" that NIQE favors, leading to relatively higher NIQE values.

2. Computation cost

Thanks for this comment. The following table shows the computation cost of all involved models. LightD relies on IC-Light, a pre-trained diffusion-based relighting model, to generate semantically consistent and physically plausible lighting variations. This naturally introduces additional inference time due to the sampling process inherent in diffusion models. The additional computation enables greater visual naturalness, semantic alignment, and stronger attack effectiveness, which are core goals of non-suspicious adversarial attacks.

1. Contribution

We respectfully disagree with the assessment that our work lacks significant contribution. While our approach does utilize IC-Light, we would like to clarify several important points. First, scientific advancement often comes from novel combinations and applications of existing technologies rather than entirely new tools. Our contribution lies not in creating a new relighting model, but in addressing the previously unexplored challenge of leveraging relighting techniques for natural adversarial attacks against VLP models. Second, our integration is far from straightforward. We encountered and solved several significant challenges:

a) Semantic preservation. Default IC-Light outputs (e.g., text-/background-conditioned) often alter semantic content. We propose a reference-based relighting approach that preserves semantics by introducing only lighting differences.

b) Cross-modal adversarial optimization. Achieving attacks that mislead VLPs while maintaining naturalness required a carefully designed optimization objection.

c) Contextual parameter selection. We are the first to use LLM for semantically aligned lighting parameter initialization for adversarial relighting, allowing coherent image-specific manipulations.

We also emphasize the contributions of our work from the following perspectives:

a) Novel problem setting. To the best of our knowledge, we are the first to explore non-suspicious adversarial attacks via natural relighting against VLPs. Previous works mainly target classification tasks using $L_p$-bounded noise.

b) Flexible framework. Our pipeline is modular and extensible; IC-Light and SGA can be swapped for other relighting or optimization techniques. The ChatGPT-guided initialization enhances semantic alignment, while the two-step optimization bridges visual quality and attack effectiveness.

c) Cross-modal integration. Though individual components may appear simple, combining them for cross-modal attacks (e.g., on captioning or VQA) is non-trivial. Our joint loss formulation (Eq. 7) enables this by balancing visual coherence with semantic misalignment.

2. Compared methods

Thank you for the suggestion. To the best of our knowledge, the color-based adversarial attack methods included in our evaluation are still among the most widely cited and representative non-suspicious color transformation attacks in the literature. We truly welcome suggestions for more recent or under-reviewed methods that we may have missed. We are happy to incorporate additional baselines in the final version or future iterations of this work to ensure a fair and up-to-date evaluation.

3. Transferability and robustness against defense

a) Cross-model transferability.
We evaluate the transferability of LightD-generated adversarial examples on MSCOCO for image captioning and VQA, where adversarial examples were generated on one VLP model and tested on a different model. Results (https://imgur.com/a/NG2l77b) show significant performance drops across unseen models, confirming that our attack generalizes across architectures.

b) Defense robustness. We tested LightD against common input transformation defenses (grayscale conversion to remove color information and median filtering for denoising). Results (https://imgur.com/a/LH1r8I2) show that while these reduce attack strength to some extent, LightD remains substantially more effective than baselines, highlighting its robustness under simple defenses.

We will provide these transferability and robustness against defense experiments in the final version.

4. Optimization spaces explanation

Existing non-suspicious attacks typically rely on low-dimensional global transformations (e.g., hue, brightness), which limit perturbation diversity and attack power. In contrast, LightD introduces a two-stage optimization: 1) Lighting parameters, 2) Pixel-level adjustment of the reference lighting image. This extends the optimization from a small parameter set to an entire image matrix, greatly expanding the perturbation space while preserving realism. As shown in our ablation study (Table 4), pixel-level refinement significantly improves performance without degrading visual quality—supporting the importance of a richer optimization space.

5. Limitations discussion

Thank you for the thoughtful suggestion. We will include a Limitations section in the final version. We acknowledge two key limitations: a) Reliance on IC-Light. Our results depend on IC-Light's relighting quality. However, the framework is modular and can adopt alternative models, which we plan to explore in future work. b) Dependence on ChatGPT. ChatGPT provides semantically meaningful initialization but introduces some response variability. We mitigate this using fixed templates and normalization, but future work may involve a dedicated learnable lighting predictor for more stable, task-specific initialization.

1. Impact of lighting model

Thank you for your comment. While existing tools help generate reference lighting, directly applying them in adversarial relighting for VLPs presents challenges:

1) Optimization vs. semantic integrity. Complex lighting models can introduce artifacts or distortions, compromising the "non-suspicious" quality needed for effective attacks.

2) Naturalness vs. attack success. Simpler lighting manipulations, when applied strategically, can yield strong adversarial effects while maintaining visual realism.

Our LightD framework addresses this via:

1) ChatGPT-guided parameter selection. We employ ChatGPT to recommend contextually appropriate lighting parameters that align with image content, enabling targeted manipulation despite the simpler lighting model.

2) Two-step collaborative optimization. It effectively expands the capabilities of the simple lighting model by optimizing lighting parameters and reference lighting images, achieving robust adversarial effects while preserving visual naturalness.

We agree that complex models may expand possibilities. Future work includes: 1) Exploring environment maps and richer lighting models. 2) Studying intensity and shape of light sources. 3) Improving optimization for high-dimensional parameter spaces.

2. NIQE result analysis

Thanks for the comment. NIQE is a non-reference metric, though widely used and adopted by our baselines, is not perceptually linear, and may miss subtle, semantic-preserving changes. Small NIQE differences may not reflect actual perceptual variation. Therefore, we provide qualitative results (e.g., Figs. 5 and 7) to complete the quantitative results.

In some VLP + dataset settings, the semantic sensitivity of captioning models may drive the optimization to adopt stronger relighting changes to ensure attack success. While these are visually acceptable, they may depart from the “natural statistics” that NIQE favors, resulting in higher NIQE scores.

We will include a detailed NIQE analysis in the final version. For future work, we plan to explore alternative perceptual metrics, including learned models and human-in-the-loop evaluations.

3. Performance on other relighting models

Thank you for the suggestion. Our use of IC-Light over other methods is based on:

a) Generalizability. Unlike many relighting models focused on faces or portraits, IC-Light supports diverse categories in our datasets (e.g., animals, scenes, vehicles).

b) Compatibility with optimization. IC-Light allows continuous control of lighting parameters (e.g., direction, color gradient), enabling both parameter-level and pixel-level optimization, which is crucial for balancing attack effectiveness and visual naturalness.

c) Future potential. We appreciate the mention of LightIt, which introduces promising advances in controllable illumination for diffusion models. Although we currently rely on IC-Light, we plan to explore LightIt once its code and models are available.

We will discuss the applicability of the other relighting methods you mentioned within our proposed framework in the final version.

4. Contribution

Thank you for recognizing our framework design. While some components (e.g., SGA-based optimization) build on prior work, our key contributions are:

a) Novel problem setting. To the best of our knowledge, we are the first to explore non-suspicious adversarial attacks via natural relighting against VLPs. Previous works mainly target classification tasks using $L_p$-bounded noise.

b) Flexible framework. Our pipeline is modular and extensible; IC-Light and SGA can be swapped for other relighting or optimization techniques. The ChatGPT-guided initialization enhances semantic alignment, while the two-step optimization bridges visual quality and attack effectiveness.

c) Cross-modal integration. Though individual components may appear simple, combining them for cross-modal attacks (e.g., on captioning or VQA) is non-trivial. Our joint loss formulation (Eq. 7) enables this by balancing visual coherence with semantic misalignment.

5. Lighting parameterization design

Appreciate the comment. While IC-Light supports per-pixel lighting control, we chose a simplified setup (start color, end color, light direction) for the following reasons:

a) Naturalness and interpretability. Gradient lighting mimics natural illumination and allows intuitive, controllable manipulation, ensuring coherence and effectiveness.

b) Avoiding artifacts. Per-pixel control expands the optimization space and risks introducing unnatural patterns or artifacts, violating the “non-suspicious” constraint. Our approach strikes a balance between expressiveness and perceptual quality.

6. Typos

We appreciate your attention to detail and will thoroughly proofread the final version to eliminate such issues.

1. Baseline performance on clean images

Thank you for this comment. The baseline performance on clean images for both image captioning and VQA tasks (shown in Table 1 and Table 2 of the original submission) can be found at: https://imgur.com/a/Kk9VYDG. These results confirm that our method induces a significant performance drop across both tasks, highlighting its effectiveness as an adversarial attack.

We will include the clean image results in the final version of the paper, either in the main paper or the appendix, to ensure a clear and direct comparison between clean and adversarial performance.

2. Performance on classification task

Thank you for your valuable suggestion. While classification is indeed important, we prioritize image captioning and VQA as they pose more complex cross-modal reasoning challenges and provide clearer semantic outputs, making them better suited for evaluating the subtle effects of our non-suspicious adversarial relighting. Additionally, these tasks help avoid label-space ambiguity that can arise in classification evaluations.

We have added a new classification experiment using CLIP on the ImageNet dataset. In this experiment, we compare LightD with three representative non-suspicious color-based attack baselines (ALA, AdvCF, and ColorFool) and report Top-1 and Top-5 classification accuracy as well as NIQE scores for visual quality.

As shown in the following table, LightD causes a significantly larger drop in classification accuracy while achieving better perceptual quality than the baselines. These results demonstrate that LightD remains effective and transferable beyond captioning and VQA and can generalize well to vision-only classification tasks. The full results and analysis will be included in the revised manuscript.

3. NIQE result analysis

Thank you for your insightful question. NIQE is a widely used no-reference metric, but it is known to be neither perceptually linear nor highly sensitive to subtle semantic variations. As such, small numerical differences in NIQE may correspond to perceptually noticeable differences, particularly in edge cases. Therefore, we also provide qualitative examples (e.g., Figure 7) to complement the quantitative evaluation.

In Table 3, the average NIQE difference between randomly selected lighting parameters and ChatGPT-recommended ones is relatively small, indicating that both strategies generally yield visually acceptable adversarial relighted images. However, the example in Figure 7 was selected to highlight a representative case where ChatGPT’s lighting suggestions lead to better semantic consistency and perceptual quality, especially in preserving scene structure and object boundaries. These refinements are often more apparent to human observers than reflected by NIQE scores.

4. Performance on natural (non-adversarial) relighting

Thank you for this comment. We conduct comparable experiments to test the robustness against natural relighting. Specifically, we use IC-Light to generate the natural lighting effects simulating day and night environments without any adversarial optimization. As shown in the following table, these natural relightings (Natural-Day and Natural-Night) induce only a minor degradation in performance compared to clean images. In contrast, LightD leads to a much more significant drop across all captioning metrics, demonstrating that the observed attack effectiveness is not merely due to lighting variation but rather to carefully optimized adversarial perturbations.

5. Is GPT-4 vulnerable to the attack images?

Thank you for the question. Although our method is designed as a white-box attack, we further evaluate its transferability to GPT-4. Specifically, we use BLIP-2 as the surrogate model on the MSCOCO dataset for the VQA task and test the performance of the generated adversarial images on GPT-4.

As shown in the following table, the adversarial examples cause a drop in both APA and WUPS0.9 scores on GPT-4, demonstrating that our method retains a certain degree of transferability and attack effectiveness even against large, black-box models like GPT-4.