Fusing Visual and Textual Cues for Sequential Image Difference Captioning

We thank the reviewer for their feedback and the recognition of METS as a significant contribution to the field. We address the reviewer's questions as follows:

Questions

1. Choice of 1000 sequences for the METS dataset The number of sequences in the METS dataset is limited by the labelling cost of the human annotations. We are considering releasing a larger sythetic-only version without human annotations in the future.

2. Why weren't other models included in the MagicBrush comparison Since the inference of GPT models incurs a cost, we focused on evaluating the proposed dataset first and foremost.

We thank the reviewer for recognizing the potential value of both the proposed approach and the METS dataset, while also providing valuable critique and suggestions for improvement.

We address the reviewer's concerns below:

Weaknesses

Unrealistic Motivation

The scenario we are considering is motivated by the real-world task of summarizing provenance metadata, to summarize the change history of an image. Such metadata (C2PA) is adopted across much of the tech industry and making tools to better comprehend this metadata is a real-world challenge. Such metadata includes the original and final editted image with intermediate image and text data. Please refer to e.g. C2PA Manifests.

The other scenario in which both the intermediate editing steps and their auxiliary metadata could be available is in the context of creative workflows, where the editing process is recorded and the metadata is generated automatically by the editing software, such as history states in Photoshop.

Evaluation

We have included an LLM based evaluation of the methods in Table 1. Please refer to the comment above for details: https://openreview.net/forum?id=rZxwa8JkJW&noteId=ZDpzdoct63

Questions

All of the models are prompted in the save few-shot learning fashion with two examples, below is the full prompt:

Given a list of manipulations, summarize the manipulations applied to make the last image from the first. Keep your answer to a few short sentences.Intermediate steps should not be added to the answer if they are not visible in the last frame \n
            Example1: List: Object: Roller skates, replacement: background
            Object: Roller skates, replacement: skateboards with LED lights
            Object: Skateboard, replacement: background
            Object: Skateboard, manipulapion: encoding_quality, intensity: decreased slightly
            Object: Skateboard, replacement: rollerblades
            Object: rollerblades, manipulapion: overlay_stripes, intensity: line width: 0.37617131348743227, line color: (170, 233, 137), line angle: 178, line density: 0.6794338640003309, line type: dashed, line opacity: 0.62
            Object: rollerblades, manipulapion: brightness, intensity: increased moderately
            Object: rollerblades, manipulapion: encoding_quality, intensity: decreased moderately
            Object: Roller skates, replacement: background
            Object: Skateboard, replacement: rollerblades
            Object: rollerblades, manipulapion: saturation, intensity: decreased moderately\n
            Summary: Shoes are now noisy; roller skates are replaced with a hover board and a patch.\n
            Example2: List: Object: Cheese, manipulapion: brightness, intensity: decreased severely
            Object: Cheese, manipulapion: contrast, intensity: decreased slightly
            Object: Cheese, replacement: pickle slice
            Object: pickle slice, manipulapion: sharpness, intensity: increased moderately
            Object: pickle slice, manipulapion: encoding_quality, intensity: decreased moderately
            Object: pickle slice, manipulapion: saturation, intensity: decreased severely
            Object: pickle slice, manipulapion: overlay_stripes, intensity: line width: 0.4923714860638555, line color: (194, 157, 0), line angle: -85, line density: 0.08991256536478587, line type: solid, line opacity: 0.73
            Object: Cheese, manipulapion: contrast, intensity: increased moderately
            Object: Cheese, replacement: spicy cheese
            Object: Cheese, replacement: sushi
            Object: Cheese, replacement: crackers
            Object: Cheese, replacement: background
            Object: Cheese, replacement: crackers
            Object: crackers, replacement: crackers
            Object: global, manipulapion: saturation, intensity: increased severely\n
            Summary: The cheese slices in the risotto bowls are replaced with a sunglass, chocolate pastry, an orange slice, a partially noisy pine cone, and a tennis ball.
            Your prompt:
            List: {lst}
            Summary: 

I appreciate the authors' efforts in conducting additional experiments and providing the few-shot prompts. I highly encourage the inclusion of all prompts used, as well as additional qualitative examples, in the manuscript or appendix. However, I still have a few major concerns that I outline below:

Motivation

The task of image difference captioning with multiple images (and text) (IDC-MI) is a sub-problem of difference image captioning (IDC). As such, it is narrower in scope and applies to very specific scenarios. While I agree with the authors that this task may arise in real-life contexts, this does not inherently make it particularly compelling from a research perspective—IDC-MI appears to be a straightforward extension of IDC. That said, the task may be of interest to some researchers and could inspire further work based on the proposed METS dataset.

To my understanding, the main contribution of this paper lies in the introduction of the METS dataset, while the FVTC model represents a standard large multimodal model fine-tuned on this dataset. A more impactful contribution, in my opinion, would involve an extensive analysis of typical failure cases that models encounter for particular editing operations or entity types. For example, testing multiple models on the METS dataset after fine-tuning them could create an interesting benchmark for evaluating model capabilities. In the current version of the paper, the authors fine-tune only their own model on the dataset, which limits comparability since other models are evaluated in a zero-shot setting.

Additional Concerns

Prompt Design. Examining the provided prompts, I now understand why text-only models may struggle. When multiple instances of the same object are present, the auxiliary text does not specify which instances are being edited. For models like GPT-4V (or similar), it would be much more effective to include the images in the few-shot prompts. Otherwise, the text alone might lead to confusion. This might also explain why adding the auxiliary text, intended to provide additional context about the editing steps and simplify the task, results in performance degradation for GPT-4V.

LLM-as-a-Judge. Thank you for incorporating the LLM-as-a-Judge experiment. While it partially addressed my concerns, my original suggestion (and I apologize if it was not clear) was to use the LLM to assess whether the generated caption accurately identifies the editing steps based on the ground-truth caption, not merely to determine if the generated caption is similar to the ground truth. This distinction is critical, as a generated caption could differ significantly in style, word choice, or length (causing the LLM to judge it as different from the original caption) while still correctly capturing all the edits made. Incorporating such a comparison would greatly enhance the clarity and interpretability of the results.

Due to all these reasons, I maintain my original score. However, I encourage the authors to carefully consider the feedback provided here and in other reviews, as I believe it offers insights that could significantly improve their paper.

Thanks for the reply. We are happy to know that you found the additional experiments and the prompt examples useful. We confirm are happy to include all of this requested data.

Regarding the motivation. It may be helpful to further clarify the use context for this paper, which helps motivate the dual contributions of both the algorithm and the dataset and why this 'IDC' problem variant is interesting.

There is an emerging standard for fighting fake news called C2PA (see paper for citation). It describes a data structure that lives inside an image metadata, that describes how an image was made (e.g. what was done – text information, and previous version of the image – visual information). In many cases this edit history comprises a sequence of visual versions of an image, interleaved with textual information describing the image (also noting that either of visual and textual information may be absent at each step).

The standard is widely adopted by industry and there are many millions if not billions of such images in circulation. However it is hard for users to understand this complex provenance information. A textual summary of it would be very helpful .

Thus a technique is required to intuitively summarize the multimodal edit history, which uniquely presents as sequence of visual and textual change descriptions. Thus we present FVTC and the METS dataset to train/evaluate it.

We thank the reviewer for their feedback, the recognition of the technical soundness of our work, and the suggestions for improving the evaluation and analysis of our method. We address the reviewer's concerns as follows:

Weaknesses

1. Evaluation Metrics We agree that n-gram-based metrics are not an ideal way to capture the semantics between two sentences. However, this is a common practice in the field of image captioning, and we follow the standard evaluation protocols to be able to compare our method with the existing methods, all of which report scores using these metrics.

We have included an LLM based evaluation of the methods in Table 1. Please refer to the comment above for details: https://openreview.net/forum?id=rZxwa8JkJW&noteId=ZDpzdoct63

2. Lack of Ablations and Analyses We agree that there are many interesting avenues that perhaps could be explored in the future works. Unfortunately, data generation (and labelling) and model training are both computationally and monetarily expensive, and we were unable to explore all the possible ablations and analyses. We will include a discussion on the potential impact of the number of generated image sequences on the final performance in the revised manuscript.

3. Missing Baselines We agree that adding more baselines would provide a better context, however it is not trivial to make LLAVA/QwenVL work with multiple images, interleaved with text. Unlike VQA, image difference captioning is not a part of the standard LLM benchmarks, which was one of the motivations for creating the METS dataset. We will include an overview of more recent MLLMs in the revised manuscript.

Questions

1. Running inference on GPT-4V costs money and we do not think it adds much value to the paper, since we are already comparing to SOTA methods for other tasks.

2. Both CLEVR Change and Spot-the-Diff datasets differ greatly from METS in terms of visual content, manipulation types and captioning style. Therefore, including METS in the training would not be beneficial. On the other hand, since METS is constructed to closely resemble real-world image editing scenarios, including it in the PSBattles training makes sense as it pushes the domain of the model from purely synthetic IP2P data to more realistic images of METS. Both CLEVR Change and Spot-the-Diff are already trained on in-distribution data, so adding METS would only add noise to the training process.

We thank the reviewer for their feedback and acknowledgment of the novelty and potential of our work as well as the importance of the METS dataset. We appreciate the reviewer's concerns and suggestions for improving the generalizability of our model and dataset. We address the reviewer's concerns as follows:

1. We would like to emphasize that whilst the proposed method benefits from auxiliary textual data, it does not require it. In Table 1, we evaluate FVTC both with and without auxiliary textual data. Model versions that make use of the textual data are denoted with a "T" in the model name (e.g., FVTC-4T). Please refer to the "text" column in Table 1, which is marked with yes/no to indicate whether the model uses auxiliary textual data. Additionally, all of the results in Table 2 are from models that do not use auxiliary textual data.

2. Task definition:

(1) Although the task of image difference captioning (IDC) could be extended to more diverse scenarios, our work focuses on the important and timely scenario of manipulation and/or tampering with an image e.g. for misinformation, rather than viewing angle changes. One use case is to combine visual and textual data in open (C2PA) metadata for image provenance to help describe such edits to consumers. We will further emphasise this focus vs. more general IDC in the paper.

(2) We agree that there are cases where semantic maps could provide more explicit and interpretable description of the changes. For example using SAM or similar to parse and compare images. In this work we explicitly focus on the task of image difference captioning, which is admittedly more challenging due to the linguistic nuances, but allows for a more natural description, especially in the context of a long list of edits, where summarization/omission is necessary.

3. We acknowledge there is a risk to overfit on any well represented domain in training data. However please refer to Table 2 for the results on Spot-the-diff and PSBattles, both of which are real-world datasets.

We are not using MSCOCO or LAION-2B for pair construction but do use the OpenImages dataset, containing real world data, for constructing the METS dataset.