Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images

We sincerely thank you for acknowledging the novelty of our method and for your positive feedback. We would like to address the questions individually below:

W'1

Thank you for pointing out the spelling error. We will correct it in the revised version.

Q'1 Results of other advanced LVLMs

Based on your suggestion, we conducted experiments using QWEN2-VL-7B and InternVL2-2B as the original models. Since the fine-tuning code for MiniGPT-4 is not open-sourced, we opted to use these two models within the limited rebuttal period. The results are shown in the table below.

The experimental results demonstrate that our method is effective in protecting the copyright of QWEN2VL-7B and InternVL2-2B, further showing the generalizability of PLA to other LVLMs.

Q'2 Performance of benchmarks after fine-tuning

We tested LLaVA’s performance on benchmarks related to several specific datasets before and after fine-tuning, and the results show that fine-tuning was conducted as expected, as shown in the table below.

Q'3 Tracking performance on unrelated LVLMs

PLA constructs triggers by simulating fine-tuning behavior through parameter reverse updates, which does not incorporate any design to enable the cross-model & cross-architecture transferability of triggers. As a result, the triggers do not cause unrelated models to produce the target output. The tracking results for models unrelated to LLaVA demonstrate that triggers generated by PLA do not falsely track these models, as shown in the table below.

It is worth noting that LLaVA 1.6 (LLaVA-NEXT) was not obtained by fine-tuning LLaVA 1.5. Instead, it utilized a two-stage training process and was not trained based on LLaVA 1.5. In our paper, related models refer to those obtained through fine-tuning the original model. Therefore, the trigger tracking success rate for LLaVA 1.6 is 0%, as it is an unrelated model.

We appreciate your feedback and hope our response addresses your points. We are open to further discussion if needed. Thank you once again for recognizing our work.

Dear Reviewer PhEj,

We greatly appreciate the time you've invested in reviewing our response. Having submitted our rebuttal, we are eager to know if our response has addressed your questions. The discussion phase is nearing its end, but we have not yet received your response. We look forward to hearing from you for any further clarification that you might require.

Best,

Submission 13438 Authors

Dear Reviewer PhEj,

We sincerely thank you for your time and effort in reviewing our paper and for your thoughtful feedback and appreciation of this work! We have carefully read and addressed your questions. As the discussion period deadline approaches (Dec.3, 2024), if you have any additional questions, we would be delighted to continue the conversation with you! We look forward to hearing from you for any further clarification that you might require.

Best,

Submission 13438 Authors

Dear Reviewer PhEj,

Thank you for your encouraging feedback and for taking the time to review our rebuttal. We are delighted to hear that you find the results added during the rebuttal period valuable and better demonstrating the effectiveness of our approach.

We have added the experimental results addressing Q1 and Q3 in Sections D.4 and D.2 of the revised paper's appendix before. Since the deadline for submitting the revised PDF has passed, we are unable to re-upload a new revision containing the results for Q2. Nonetheless, we are prepared to include the experimental results for Q2 in the final version.

Building on your initial positive assessment of this work, we hope these results and improvements will further reinforce your recommendation for acceptance. Thank you once again for your constructive review and support in improving our work!

Best,

13438 Authors

We thank the reviewer for the thorough review of our work. We have taken note of the concerns raised by the reviewer and below we will address them accordingly.

W'1 Fine-tuning modules of LVLMs

In the fine-tuning experiments, we set the trainable components to the MLP projector and the LLM by default, while keeping the vision encoder frozen, which is consistent with the instruction-tuning phase of LLaVA. We have now conducted ablation experiments on the trainable modules using the ChEBI dataset, and the results are shown below (full fine-tuning for LLM).

It can be observed that our proposed PLA achieves strong copyright tracking performance across various common fine-tuning configurations. Notably, PLA achieves the highest performance (62%) on projector-tuned models. This is likely because the smaller number of trainable parameters results in less significant changes to the original model. We have included these new experimental results in the revised version of our paper.

W'2 Perturbation budget

Enhancing the concealment of trigger images can help prevent stealers from detecting the image input. If the perturbation budget is too large, stealers may identify anomalies in the input images, causing the model to refuse to respond and thereby evade copyright tracking. We have conducted an ablation study on the perturbation budget using the ChEBI dataset, and the results are shown below.

Experimental results show that the tracking performance does not significantly improve when the perturbation budget exceeds 16/255. Therefore, considering the concealment of the triggers, we chose a perturbation budget of 16/255.

W'3 Attack steps

We conducted an ablation study on the attack steps using the ChEBI dataset. The change of TMRs with the number of attack steps is shown below.

Experimental results show that performance is poor when the number of attack steps is small; as the attack steps approach 1000, performance improves and begins to stabilize.

W'4 False positives and false negatives

PLA constructs triggers by simulating fine-tuning behavior through parameter reverse updates, which does not incorporate any design to enable the cross-model & cross-architecture transferability of triggers. As a result, the triggers do not cause unrelated models to produce the target output. The tracking results for models unrelated to LLaVA demonstrate that triggers generated by PLA do not falsely track these models, as shown in the table below.

Dear Reviewer BfWs,

We greatly appreciate the time you've invested in reviewing our response. Having submitted our rebuttal, we are eager to know if our response has addressed your concern. The discussion phase is nearing its end, but we have not yet received your response. We look forward to hearing from you for any further clarification that you might require.

Best,

Submission 13438 Authors

Dear Reviewer BfWs,

We greatly appreciate your time and effort in reviewing our work! In our previous responses, we have actively addressed your concerns by supplementing the relevant experiments you mentioned and providing detailed explanations and clarifications regarding your questions about the paper.

As the discussion period deadline approaches (Dec.3, 2024), if you have any additional questions, we would be delighted to continue the conversation with you! We look forward to hearing from you for any further clarification that you might require.

Best,

Submission 13438 Authors

Thank you for your response! After reading it, I still have a question regarding the motivation and influence of concealment. In principle the input image can be arbitrary, how would anomaly detection work in this scenario? This raises another question that whether the "rare question-answer pairs" used for tracking will be detected as anomaly in a similar manner?

From the results of varying the budget, the TMR will decrease after the budget exceeds 32, this is counter-intuitive since a larger budget should strictly outperform lower budgets if the optimization is correct.

Thank you for your thorough reading of our paper. In response to your concerns, we provide the following analysis.

W'1 Cross-model transferable adversarial attacks

We believe there is a misunderstanding regarding your perspective. We want to clarify that cross-model transferable adversarial attacks and adversarial example-based copyright tracking are fundamentally different. The former refers to crafting adversarial examples on a specific model that can also effectively attack models with different architectures performing the same task. In contrast, the latter involves crafting adversarial examples on an original model that can only effectively attack models fine-tuned from that original model, while not attacking other models (e.g., models with different architectures).

Therefore, adversarial examples with cross-model transferability are not designed for copyright tracking. Copyright tracking requires that only related models, fine-tuned from the original model (LLaVA), are triggered to produce specific outputs. Cross-model transferable adversarial examples might cause unrelated LVLMs (e.g., QWEN2-VL, InternVL2) to also produce specific outputs. This could result in false tracking and lead to overclaims. Therefore, we did not compare our method with transferable adversarial attack methods.

To validate the effectiveness of our proposed PLA, we compared it with some of the cross-model transferable attack methods you mentioned. The results are as follows. Note that the SGA and VQAttack you mentioned are designed for Vision-Language Pretraining (VLP) models rather than Large Vision-Language Models (LVLMs).

The results show that our PLA outperforms these transferable attack methods. We believe this is because PLA is specifically designed to trigger fine-tuned models to produce predetermined outputs, which can be understood as “fine-tuning transferability.” In contrast, these methods focus on cross-model (cross-architecture or cross-prompt) transferability.

W'2 Complex datasets

We would like to clarify that we used two domain-specific downstream datasets in our main experiments: a chemistry molecular description dataset ChEBI-20 and a mathematical image question-answering dataset MathV360K, which involve tasks in the fields of chemistry and mathematics. Sample examples can be found in Appendix Section B. We believe these datasets are not “simple” tasks for LLaVA. Firstly, the image data distribution in these datasets differs significantly from that in LLaVA’s pretraining. Secondly, the task-specific knowledge required for these datasets also deviates from LLaVA’s pretrained knowledge. Both datasets require the model to adapt to highly specialized knowledge. Nevertheless, PLA still demonstrates strong tracking performance on models fine-tuned from these tasks.

[1] Dong et al. Boosting Adversarial Attacks with Momentum.

[2] Xie et al. Improving Transferability of Adversarial Examples with Input Diversity.

[3] Luo et al. An Image Is Worth 1000 Lies: Transferability of Adversarial Images across Prompts on Vision-Language Models.

We hope our response addresses your concerns. If you have any further questions, please feel free to reach out.

Dear Reviewer Rr8h,

We greatly appreciate the time you've invested in reviewing our response. Having submitted our rebuttal, we are eager to know if our response has addressed your questions. The discussion phase is nearing its end, but we have not yet received your response. We look forward to hearing from you for any further clarification that you might require.

Best,

Submission 13438 Authors

We sincerely appreciate your acknowledgment of our work and your recognition of our method. Regarding your concerns, we hope that our clarification can be helpful.

W'1 Lack of explanation

Model fine-tuning adapts the model to downstream tasks through parameter updates, and these updates inherently alter the model’s behavior, making traditional adversarial examples ineffective in triggering fine-tuned models. The core idea of PLA is to make triggers adapt to the model’s behavioral changes by simulating the behavior of a fine-tuned model rejecting the target output. During copyright tracking, since the triggers have already converged under the resistance of parameter updates (model rejection), they are more likely to successfully trigger the actual fine-tuned model to produce the predetermined output.

W'2 Broader vision-language Tasks

Actually, the PaintingForm and ChEBI datasets we utilized belong to image captioning tasks, with relevant examples provided in Section B in the Appendix. We have now fine-tuned LLaVA on visual grounding and multimodal classification tasks. And then we evaluated the trigger’s performance to track the copyright of fine-tuned models. The results are shown in the table below.

Specifically, the RefCOCO dataset is used for the visual grounding task, while the Hateful Memes dataset is used for the multimodal classification task. The results indicate that our method remains effective in these tasks, achieving better performance compared to baseline method IP. Thanks for raising the point about broader vision-language tasks. We have included these new experimental results in the revised manuscript.

W'3 Scalability and computational efficiency

Computational efficiency

While PLA introduces reverse parameter updates, these updates are confined to the adversarial trigger generation phase and are computationally less demanding compared to full model fine-tuning. Specifically, we employed a simple gradient ascent algorithm to perform reverse updates on the model parameters, and the number of triggers was significantly smaller than the actual training data. Thus, PLA is more efficient compared to fine-tuning. Additionally, adversarial sample generation is a one-time process, so PLA is computationally feasible for large-scale applications.

Potential optimizations

To further improve efficiency, we believe that LoRA can be utilized for reverse updates of the model in PLA. Besides, the efficiency of PLA can be enhanced by reducing the number of attack steps. However, there is a trade-off between efficiency and performance, as decreasing the number of iterations may lead to performance decline.

W'4 & Q'3 Novelty of our work

We highlight the novelty of our work from the following perspectives:

• Novel Direction: We are the first to address copyright protection for LVLMs, which is an unexplored area.
• Novel Motivation: Unlike prior watermarking methods that commonly use backdoor attacks for fingerprint embedding, we employ adversarial attacks to track copyrights, offering a non-intrusive approach. Furthermore, existing adversarial attack studies related to LVLMs have not focused on the copyright protection task.
• Novel Approach: Leveraging the characteristics of LVLMs, we innovatively design adversarial triggers capable of tracking the original copyright of fine-tuned models.

Dear Reviewer gTwA,

We greatly appreciate the time you've invested in reviewing our response. Having submitted our rebuttal, we are eager to know if our response has addressed your concern. The discussion phase is nearing its end, but we have not yet received your response. We look forward to hearing from you for any further clarification that you might require.

Best,

Submission 13438 Authors