Visual Transformation Telling

1. The difference between VTT and the existing Procedure Planning task seem not significant. Typically, Procedure Planning tasks are done on CrossTask and COIN datasets, and there is already a lot of related work on this. Moreover, both tasks are based on pre-collected temporal segment annotations from CrossTask and COIN, so the textual content should be the same. It seems like this paper simply transforms a classification problem into an open-ended text generation problem.
2. If the focus of VTT is on explanation, why not use two images instead of a reasoning chain? How does the previous reasoning process affect subsequent reasoning? When the model makes errors, are they due to the accumulation of prior mistakes or the inability to infer the current two states?
3. Specific details regarding testing models like GPT-4o need to be provided. For example, GPT-4 is a purely text-based model—how did you test it, or did you test GPT-4V? Additionally, the specific prompts and the details such as model version should be thoroughly explained.

It is recommended to add a figure that clearly describes the differences between the task proposed in this paper and all previously related tasks.

1. Motivation. The motivation to create this task is vague. It is unclear why describing the transformation between two images is more important than a single image. From the qualitative examples in the paper, it seems that only using one image is enough to generate the caption. The evidence for real-world or downstream applications is not clearly justified in the paper. Here are some questions that authors can think about when evaluating the motivation of this work: Will this new task benefit models on other tasks? What are the real-world use cases of such a task/model? Do I need a robot to understand the state transformation in natural language format to perform daily tasks? If the answer is only "current models cannot perform well on our task", then you need to really think about whether to do such research.

2. Distinction with other video-language tasks. Basically, this paper converts the existing video action recognition tasks like COIN into a video captioning task. The concept of state transformation is ambiguous, and it is hard to justify this in a better format. Furthermore, compared to the original format of COIN, the generation format of VTT poses additional difficulties in evaluation, as indicated by the bad performance of humans when using automatic metrics (Table 2).

3. Limited Contribution. Overall, this paper creates a task without strong motivation and claims the existing model cannot perform well on this new task. There is not much technical content in the paper, and even the data is reused from previous work. The overall contribution of this work is below the standard of ICLR.

The paper has a poor presentation.

• Fig. 1 could be improved by showing explicitly what are the inputs and the outputs in this task.
• Sec 3.1 should be linked to Fig. 1.
• The most importantly, the main text lacks details about TTNet, which makes following the paper very difficult.
• Fig. 2(d) completely could be put into the appendix (that could save some space for the TTNet).

The evaluation procedure.

• Some models achieve very high CIDEr scores. Could the authors please double check?
• While I appreciate the number of different evaulation metrics, I am skeptical about a couple of them.
• Why do we have two task-related human evaluation criterias (relevance/logical soundness) where they could be combined? If some generation is irrelevant, could the logical soundness matter for that generation? (could you please provide a counter-example if there is one?)
• Why do the authors evaluate fluency? The most of the transformation descriptions are relatively short (Fig 2.(c)) and the literature already evaluated the fluency enough previously.
• Is there a reason to avoid some evaluation procedure similar to LLM-as-judge in this work?
• It is not clear to me, how they evaluate the models. Is the evaluation per-transformation or per-scene?
• How are the images being processed by the models? For instance, as far as I know some models could process input images subsequently, are all of the tested models capable of doing the same? If not, how do the authors process input images? Do they concatenate input images?

The models.

• The number of open-source models could be increased (e.g. PaliGemma, OpenFlamingo, Idefics, InstructBLIP there are some suggestions, a newer suggestion could be Pixtral).
• Some video models could be added. Actually some of these models are pretrained using both video and image data, so they could be adapted easily (e.g. Frozen-in-Time, Video-LLaMA and many others).

Missing some relevant literature.

• Video-Text models should be included.
• The ViLMA benchmark's change-of-state task is quite related to the proposed benchmark. It should be mentioned and discussed.