Implicit Semi-auto-regressive Image-to-Video Diffusion

• I find the setting of this paper somewhat strange. From my perspective, the results presented in the paper looks like video prediction, which should be compared with many video prediction baselines.

• The choice of baselines for comparison is confusing. Firstly, I don't understand why video prediction is not compared. Secondly, even if we exclude commercial software like Runway, there are other options for image-to-video, such as the results showcased on the official website of Make-a-Video. Additionally, tools like VideoCrafter and references [1] and [2] can also perform similar tasks.

• Does the proposed framework generate different results? For the same input image, the paper only shows one result.

• The algorithm's performance is not good. For example, in Figure 4, the third row shows a completely blurred face. Would this be considered a failure case? Furthermore, in Figure 5, it is evident that the predicted images have different brightness compared to the original images.

[1] MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation

[2] Diffusion Models for Video Prediction and Infilling

Lack of discussion and comparison with most relevant works on the I2V setting (e.g. [1], [2]). Although I2V is relatively less explored than T2V, there have been works that were shown effective for the I2V tasks. While [1] and [2] focused mainly on T2V, their method can be naturally applied to perform I2V. They can generate a video that is highly consistent with the single input image. More discussions and comparisons with those existing works are needed to make it clearer for which aspects the proposed method offers more advantages compared to the current stage of knowledge on this problem.

It is not entirely clear from the current writing what the main advantages of the proposed temporal layers (based on 2DConv recurrent look-back) are compared to existing approaches to temporal modelling (such as 3DConv, ConvLSTM, or temporal attention). The authors had some limited discussion on this in the appendix, but no insight was provided, and very few visual examples. I believe this point needs more in-depth discussion and analysis because the proposed 2DConv look-back module is one of the key claimed contributions and novelties of this paper. The paper in its current form left me with the impression that the experiments were not carried out to the extent that they should be, which makes the current results somewhat premature.

• The authors mentioned that the model was trained with only 5000 clips and trained with only 72 hours on a single GPU. They mentioned that further training can lead to better performance but it is not clear how much and in which way it will improve.
• I observed that the quality of the generated videos is generally worse than the existing general video generation works [2], [3]. Is this an indication of a limitation of using the 2DConv look-back module compared to more powerful temporal layers? Or is it the effect of the model being under-trained?
• There are still noticeable jumps in appearance between the first and second frames, will training more make this better or worse?

Overall, while I appreciate the important problem that this paper proposes to solve, I found the current results unimpressive compared to existing methods that can solve the same problem. More importantly, such unsatisfactory performance could come from pre-mature experiments. I believe this paper can benefit from more thorough experiments and in-depth study.

[1] Make-A-Video: Text-to-Video Generation without Text-Video Data. Singer et al., 2022

[2] Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models. Blattmann et al., 2023

[3] Magic Video: Efficient Video Generation with Latent Diffusion Models. Zhou et al., 2022

• The idea of concatenating conditions with input noise volume is not novel in diffusion models. It has been used in many papers such as VideoComposer. The baselines (add noise or inversion) are weak, they are widely used in training-free diffusion models, but the proposed method is not a training-free model.
• In Figure 5, the results of I2VGen-XL and VideoComposer (especially VideoComposer) are much worse than the results shown in their paper. It will be good to explain the difference in the paper, i.e. what implementation of VideoComposer is being used in the comparison.
• In Figure-4, the temporal consistency / structure preservation is not very good.
• In terms of quantitative evaluation, only frame similarity and text-image alignment are measured. It will be helpful to provide other important metrics such as temporal consistency and image quality.

• What does the “implicit” in the title mean? The paper does not clearly explain its meaning in the method section.

• The results of the proposed method are more consistent than those of other methods, however, some cases appear to lack dynamics, e.g., the second and third rows of Fig. 5.

• Some details of generated results are still inconsistent although the overall appearance looks similar, e.g., for the first row of Fig. 5, the colors of dog legs in the last two frames are different from the colors in the given frames.

• How do the number of previous layer hidden states L and the look-back depth d affect the performance? Some related parameters analysis should be conducted in the experimental section.