WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models

Thanks for the valuable comments and recognition of the novelty and meaningfulness of our research. The reviewer acknowledged the clarity and thoroughness of our experimental design, as well as the strong evidence backing our claims. They highlighted WMarkGPT’s superior performance in watermark description and visibility prediction. The reviewer also thought that our datasets and training pipeline provided a new benchmark for watermark security. In addition, the reviewer commended our comprehensive ablation studies and found the manuscript well-structured and easy to follow.

Q1: Effective of the learnable query

Answer: The learnable queries are employed to extract high-level semantic features from images and filter out redundant visual noise. We conducted an experiment in which we remove the learnable queries and applied multi-stage SFT directly to the model, resulting in considerable performance drops. This shows that multi-stage SFT alone is not sufficient, as it lacks the targeted feature abstraction provided by the learnable queries.

Q2: Evaluating watermark position prediction

Answer: We agree that evaluating watermark position prediction is crucial. As suggested, we conducted additional evaluations to verify our performance. We employed the following prompt and GPT to assess the position prediction of our WMarkGPT:

Task: Determine whether the following two sentences describe approximately the same watermark position, allowing for minor variations in phrasing or slight positional bias.

Sentence 1: {predicted description} Sentence 2: {ground truth}

Judgment Criteria: 1. Do they describe the same general location, even if there are small differences? 2. Are the differences within an acceptable range (e.g., slight shifts in coordinates or wording variations like "top-left" vs. "upper-left")? 3. Is there any ambiguity that affects the interpretation? 4. Does the predicted description avoid mentioning any position information, as the watermark is inherently invisible?

Expected Output: Just Final Consistency Judgment: (Yes/No)

The precision, recall, and accuracy of the classification are computed referring to the visibility/invisibility of the watermark:

Q3: Structure of Visual Abstractor

Answer: The visual abstractor comprises six transformer layers, each employing cross-attention between extracted visual features and learnable queries. Through cross-attention with learnable queries, the model progressively refines visual feature extraction, ensuring these queries capture high-level semantic information while filtering out irrelevant visual clues. This design also strengthens the subsequent alignment between image features and textual descriptions. We will release the code after acceptance to clarify it.

Q4: The experimental comparisons

Answer: We appreciate the reviewer’s concern regarding the experimental setup of baseline models. To clarify, all baseline models were fine-tuned on our proposed datasets rather than directly evaluated in a zero-shot manner.

Thanks for your valuable comments and recognition of our innovation, comprehensive experiments, superior performance and practical significance. We appreciate that you highlighted WMarkGPT’s ability to predict watermark visibility without the original images, the newly constructed datasets, our progressive training strategy, and the benefits for subsequent research.

Q1: The WQ-Real dataset and our application

Answer: We adopt a three-stage training pipeline to progressively enable WMarkGPT to comprehend watermarked images. In the first stage, the model is trained on a large-scale object location-aware QA dataset based on natural images. In the second stage, it is further optimized on a synthetic watermarking QA dataset, bridging the gap from natural to watermarked images. Finally, the model is fine-tuned on a real watermarking QA dataset. Because the training sets in the first two stages are large, only a relatively small and high-quality real dataset is needed to align the model with practical data distribution. The significance of dataset quality over quantity has been well-established in MLLM research. To ensure that the watermarked images closely mirror real scenarios, we conducted a thorough investigation and selected typical image watermarking algorithms—including Hidden, BalujaNet, WengNet, HiNet, and Safe-SD—to build our real dataset. Experimentally, we expanded the WQA-Real dataset step by step and found that performance stabilized at around 2.5K samples.

Originally, we designed WMarkGPT to evaluate watermarked images more precisely and comprehensively, particularly when original images are unavailable for generative watermarking. The success can facilitate the development of MLLMs for other tasks involving mixed visual patterns, such as deepfake detection and edited image comprehension. In addition, our dataset collection paradigm and training strategy can also be consulted.

Q2: Effect of optimizing visual encoder and abstractor

Answer: For brief, the staged optimization of the vision encoder and visual abstractor enables the model to progressively refine its focus on watermarked regions. The vision encoder capture low- to mid-level semantics, providing the basis for further abstraction and alignment with the language model. The visual abstractor refines the visual features into high-level representations through a set of learnable queries. By aggregating crucial image features and filtering out noise, it delivers more compact and meaningful visual embeddings for subsequent processing by the language model. We will add more explanations in the final paper.

Q3: Efficiency of WMarkGPT

Answer: As detailed in Section 3.1 Model Architecture, WMarkGPT is built upon the LLaMA-2-7B framework (7 billion parameters). The complete model, including the vision encoder, visual abstractor and learnable queries, has a total of 8.198 billion parameters. We have made a detailed summary of training and testing costs in the response to Reviewer oNvm Q7.

Q4: Effect of previous watermark related description

Answer: Our experiments show that predicting watermark visibility without first extracting a watermark-related description results in lower accuracy on both the WQA-Synthetic and WQA-Real datasets. This indicates that capturing key features—such as spatial structure and texture—beforehand is crucial for accurately estimating watermark visibility.

Q5: Limitations and future works

Answer: The current WMarkGPT model is trained exclusively on image datasets, whereas video watermarking and its more precise evaluations remain crucial yet underexplored aspects of copyright protection. Future research will focus on collecting video watermarking datasets and developing video-based MLLMs to understand watermarked videos, further advancing digital watermarking technology.

Q6: More training details

Answer: We provide the complete configurations for the three training phases in the table below to ensure full reproducibility. After acceptance, we will release all the training code.

Thanks for your insightful feedback and recognition on our clear presentation, comprehensive comparisons and better results.

Q1: The explanation of the "real" dataset

Answer: Compared to our WQA-Synthetic dataset, which employs pseudo watermarking processes to generate watermarked images, WQA-Real uses genuine watermarking algorithms (e.g., Hidden, BalujaNet) to produce watermarked images that more closely align with evaluating data distributions. Additionally, all QA pairs and watermark visibility assessments in WQA-Real were manually annotated by trained human evaluators (Sec. 2.2 & Appendix E), ensuring that both descriptions and scores reflect genuine human perception rather than synthetic labels. WMarkGPT is proposed to comprehensively understand these watermarked images without accessing original images and further advance the development of watermarking algorithms. We use the term "real" to emphasize the distinctness of WQA-Synthetic and WQA-Real datasets.

Q2: Invisible watermark description

Answer: Ideally, watermarking algorithms generate watermarked images with fully invisible watermarks. However, in practice, existing methods often produce images with varying degrees of watermark visibility. To advance the research, we propose WMarkGPT for precise and comprehensive evaluations of these watermarked images. Our WQA-Real dataset thus includes both visible and invisible watermarks (see Fig. 4 for distribution). For genuinely invisible ones, we label them simply as “invisible” without adding spatial descriptions (Appendix E).

Q3: Details of WMarkGPT

Answer: As detailed in Section 3.1 Model Architecture, WMarkGPT is built upon the LLaMA-2-7B framework (7 billion parameters). The complete model, including the vision encoder, visual abstractor and learnable queries, has a total of 8.198 billion parameters. We will make it clear in the final paper.

Q4: The reference texts used in LLM-Score

Answer: As detailed in Appendix G, our evaluation metric LLM-Score [1][2] uses the reference template: ``Evaluate the relevance between the following description and ground truth on a scale from 0 to 4. Higher scores indicate better relevance. Return only the numeric score. - Description: candidates - Ground Truth: references."

[1]Lu Y, Yang X, Li X, et al. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Neurips 2023.

[2]Huang K, Sun K, Xie E, et al. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Neurips 2023.

Q5: The size of the WQ-Synthetic dataset

Answer: We conducted additional experiments by increasing the WQ-Synthetic dataset size. As shown in the table below, adding more training data only resulted in slight performance improvements. This might be due to the complexity of watermarked image understanding, where the model needs to predict watermark visibility and produce detailed textual descriptions of its location, content, and effect on image semantics. Our 7B-parameter LLM backbone may have reached its limit at this dataset scale. In our final paper, we will include experiments and analysis with a more powerful LLM backbone and larger datasets.

Q6: Model comparison

Answer: We clarify that all baseline models were fine-tuned on our proposed datasets rather than directly evaluated in a zero-shot manner.

Q7: Efficiency of WMarkGPT

Answer: The training and inference costs of our model are as follows (2 Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz).

• Training: 12 hours (Training Stage 1) and 6 hours each (Training Stages 2-3) on 8×NVIDIA RTX 6000 Ada GPUs.
• Inference: 3.395 seconds for a single image (avg.) on a single NVIDIA RTX 6000 Ada GPU.

Q8: Download links of the datasets

Answer: As stated in the abstract, we will publicly release both the code and dataset upon acceptance. To comply with the double-blind review policy, we provide an anonymous download link here: https://drive.google.com/drive/folders/1JoRq91b0UAbTU4SEblCycRVwCpUgkeg2?usp=drive_link.

Q9: Typo in Fig. 5

Answer: We apologize for the typo and will correct it in the final paper.

Thanks for your valuable comments and recognition on our clear presentation, contributions of extending the MLLMs ability to watermarked image understanding.

Q1: Experimental setup of baseline models

Answer: We clarify that all baseline models were fine-tuned on our proposed datasets rather than directly evaluated in a zero-shot manner.

Q2: The effectiveness of progressive training strategy

Answer: We conducted an experiment where the model was trained on all three datasets jointly in a unified manner using the same training configurations as in the progressive setting. The experimental results below indicate that the progressive training approach consistently yields much better performance compared to the unified training strategy. For example, on the WQA-Synthetic dataset, progressive training achieves an improvement of +0.064 in BLEU-1 and +5.367 in LLM-Score relative to unified training. Similar trends are observed on the WQA-Real dataset. These results suggest that the progressive training strategy effectively mitigates the domain adaptation challenges inherent in joint training, leading to improved generalization across diverse data domains.

Q3: Architectural enhancement for watermarked image understanding

Answer: We thank the reviewer for this valuable suggestion. In the MLLM research literature [1, 2, 3], the focus has primarily been on rigorous multimodal data collection and innovative training strategies. As the first work on watermarked image understanding using MLLMs, we also place our main emphasis on the novel WQA-Synthetic/WQA-Real datasets and a three-stage training pipeline.

That said, we recognize that specialized architectural enhancements can further improve watermark understanding. To address the challenges arising from mixed image and watermark patterns, we have integrated Mixture-of-Experts (MoE) into the LLM backbone. This addition enables more effective processing of watermark-specific features alongside natural image semantics. Experiments have shown that this modification boosts performance compared to our baseline model, and we will include a detailed analysis in the final paper.

[1] Li, Chunyuan, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao, Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 2023.

[2] Lin, Ji, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024.

[3] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, andSteven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. Advances in Neural Information Processing Systems, 2023.