Clearer Frames, Anytime: Resolving Velocity Ambiguity in Video Frame Interpolation

Thank you for recognizing our work's motivation, performance, and interpretability.

Before addressing specific questions, if there are any uncertainties or if a refresher on the core concepts of our paper's methodology is needed, we kindly invite you to review our comprehensive response to Reviewer aknz, where we offer a clearer and more detailed explanation:

First and foremost, it's essential to clarify that ... similar to that mentioned TTS paper (Wang, Yuxuan, et al.).

The training and inference time complexity will sightly increasing due to introducing the proposed tech.

We will incorporate the following details into our final paper:

Distance indexing: Transitioning from time indexing ($[T]$) to distance indexing ($[D]$) does not introduce extra computational costs during the inference phase, yet significantly enhancing image quality. In the training phase, the primary requirement is a one-time computation (offline) of distance maps for each image triplet.

Iterative reference-based estimation ($[D,R]$): Given that the computational overhead of merely expanding the input channel, while keeping the rest of the structure unchanged, is negligible, the computational burden during the training phase remains equivalent to that of the $[D]$ model. While during inference, the total consumption is equal to the number of iterations $\times$ the consumption of the $[D]$ model. We would like to highlight that this iterative strategy is optional: Users can adopt this strategy at will when optimal interpolation results are demanded and the computational budget allows. Our experiments (Table 2) show that 2 iterations, i.e., doubling the computational cost, yield cost-effective improvements.

Should you have any further questions about our research, including those raised by other reviewers, we are more than happy to engage in further discussion and provide explanations!

Thanks for recognizing the effectiveness of our approach and for the constructive suggestions. We will first address the concerns raised, except those requiring experimental validation.

Before addressing specific questions, if there are any uncertainties or if a refresher on the core concepts of our paper's methodology is needed, we kindly invite you to review our comprehensive response to Reviewer aknz, where we offer a clearer and more detailed explanation:

First and foremost, it's essential to clarify that ... similar to that mentioned TTS paper (Wang, Yuxuan, et al.).

Additionally, please specify which three out of the seven frames were used for testing purposes.

Notice that, within each septuplet, the i-th frame corresponds to $I_{i/6}$ in Fig. 5 ($i=0,1,\cdots,6$). We utilized the first ($I_0$) and last ($I_1$) frames as inputs to predict the rest five frames. We only show $I_{2/6},I_{3/6},I_{4/6}$ in Fig. 5 for visualization, but we report the results for all five intermediate frames in quantitative results. This detail will be explicitly clarified in the final version of our paper.

Provide a comparison of the inference cost between different methods, including additional optical flow estimation and iterative reference-based interpolation. It is important to note that the iterative reference-based strategy may involve running the interpolation multiple times, resulting in a significant increase in the overall inference time.

Distance indexing: Transitioning from time indexing ($[T]$) to distance indexing ($[D]$) does not introduce extra computational costs during the inference phase, yet significantly enhancing image quality. In the training phase, the primary requirement is a one-time computation (offline) of distance maps for each image triplet.

Iterative reference-based estimation ($[D,R]$): Given that the computational overhead of merely expanding the input channel, while keeping the rest of the structure unchanged, is negligible, the computational burden during the training phase remains equivalent to that of the $[D]$ model. While during inference, the total consumption is equal to the number of iterations $\times$ the consumption of the $[D]$ model. We would like to highlight that this iterative strategy is optional: Users can adopt this strategy at will when optimal interpolation results are demanded and the computational budget allows. Our experiments (Table 2) show that 2 iterations, i.e., doubling the computational cost, yield cost-effective improvements.

We will include these details in our final paper for clarity.

It would be even more beneficial to evaluate the proposed method on other high-resolution benchmarks for testing, particularly to demonstrate its superiority in handling larger motion scenarios.

Thank you for your constructive suggestions! Using RIFE as an example, we present experimental comparisons for 8x interpolation on the Adobe240 [1] and X4K1000FPS [2] benchmarks with uniform maps. The results are provided below for reference:

Distance indexing [D] and iterative reference-based estimation [R] strategies can consistently help improve the perceptual quality. In addition, it is noteworthy that [D]$_u$ is better than [T]$_u$ in terms of the pixel-centric metrics like PSNR, showing that the constant speed assumption (uniform distance maps) holds well on these two easier benchmarks.

We also show 16x interpolation on X4K1000FPS with larger temporal distance as follows:

The results highlight that the benefits of our strategies are more pronounced with increased temporal distances.

I am also interested in exploring the robustness of the proposed method when dealing with inaccurate optical flow estimation. Additionally, I am curious to see how well the model performs when incorporating it with another optical flow estimator, such as FlowNet, to provide the distance ratio map.

Since the inference speed of FlowNet is too slow, we use GMFlow [3], a newer optical flow estimator, to precompute distance maps for comparison. The results are as follows:

Our strategies still lead to consistent improvement on perceptual metrics. However, this more recent and performant optical flow estimator does not introduce improvement compared to RAFT. A likely explanation is that since we quantify the optical flow to $[0,1]$ scalar values for better generalization, our training strategies are less sensitive to the precision of the optical flow estimator.

P.S., the same experimental conclusions can be derived from other SOTA models like AMT as follows:

We will incorporate the above discussions into the final paper.

References:

[1] Su, Shuochen, Mauricio Delbracio, Jue Wang, Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang. "Deep video deblurring for hand-held cameras." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1279-1288. 2017.

[2] Sim, Hyeonjun, Jihyong Oh, and Munchurl Kim. "Xvfi: extreme video frame interpolation." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 14489-14498. 2021.

[3] Xu, Haofei, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. "Gmflow: Learning optical flow via global matching." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8121-8130. 2022.

Thank you for acknowledging our contributions and for constructive feedback. We will first address the concerns raised, except those requiring experimental validation.

The proposed distance indexing resolves the scalar velocity ambiguity in training, but nor in inference. Although the authors show in the experiments (Fig. 9) that the results achieved by using only uniform map in inference can already be satisfactory for humans, I think it would be better to explain this further in the Methods section.

Thank you for raising this concern. We kindly invite Reviewer 1WoD to refer to our detailed response to Reviewer aknz, where we present a more intuitive and detailed explanation:

First and foremost, it's essential to clarify that ... similar to that mentioned TTS paper (Wang, Yuxuan, et al.).

Does the iterative estimation mechanism require any changes to the structure of the existing model, especially the input and output parts? If so, is this consistent with the plug-and-play claim? The details need to be clarified.

The adaptation required is merely an expansion of the input channel for each model to include additional reference inputs, while the output configuration remains the same. More precisely, the iterative reference-based estimation mechanism increases the original 1-dimensional $t$ map channel to a 5-dimensional format. This includes a 1-dimensional $D_t$ map, a 1-dimensional $D_{ref}$ map, and a 3-dimensional RGB reference $I_{ref}$. Since this modification does not affect the core architecture of the model, we maintain that it aligns with the plug-and-play principle.

The training details of iterative estimation need to be clarified as well. Is D_{t/2} obtained in training using the optical flow computed from the ground-truth I_{t/2} in the dataset? Is it a uniform map of D=t/2 in inference?

Thank you for your suggestion. Indeed, during training, $D_{ref}$ is derived using optical flow computed from the ground-truth at a time point corresponding to a randomly chosen reference frame, such as $t/2$. In the inference phase, we utilize a uniform map with $D=t/2$. We will ensure to clarify these details in the final version of our paper.

The additional consumptions that the proposed new mechanisms bring to model training and inference need to be discussed.

Distance indexing: Transitioning from time indexing ($[T]$) to distance indexing ($[D]$) does not introduce extra computational costs during the inference phase, yet significantly enhancing image quality. In the training phase, the primary requirement is a one-time computation (offline) of distance maps for each image triplet.

Iterative reference-based estimation ($[D,R]$): Given that the computational overhead of merely expanding the input channel, while keeping the rest of the structure unchanged, is negligible, the computational burden during the training phase remains equivalent to that of the $[D]$ model. While during inference, the total consumption is equal to the number of iterations $\times$ the consumption of the $[D]$ model. We would like to highlight that this iterative strategy is optional: Users can adopt this strategy at will when optimal interpolation results are demanded and the computational budget allows. Our experiments (Table 2) show that 2 iterations, i.e., doubling the computational cost, yield cost-effective improvements. We will include these details in our final paper for clarity.

Does iterative estimation cause problems with error accumulation, e.g. the first time the direction is wrong, making it harder to get back to the right one the second time? Failure cases may be needed.

To mitigate error accumulation, we always retain the original start and end frames as appearance references in each iteration, which prevents devastatingly diverging directions. The notion of "wrong" directions is somewhat subjective due to the multiple feasible directions. Our goal is to predict one plausible trajectory with all interpolated objects remaining clear. Exploring methods to recover any potential direction, or perhaps to learn a distribution of directions, is a direction for our future research.

Is manipulable interpolation of optical flow based VFI methods also possible?

We assume that Reviewer 1WoD is referring to the traditional method of directly warping images based on optical flow. While manipulating certain regions by scaling optical flow through time interpolation is feasible, this approach is limited to strictly linear warping or motion. In addition, this approach might result in the interpolated image having unfilled gaps or “holes”.

Would changing to a newer optical flow estimation method result in more improvements? Especially for the exception AMT-S.

Thank you for raising this interesting question!

To begin, we kindly remind that AMT-S's exception occurs when using only the iterative reference-based estimation. The distance indexing can stabilize the benefits gained from iterative reference-based estimation.

We utilize GMFlow [1] to precompute distance maps for comparison. The results are as follows:

Our strategies still lead to consistent improvement on perceptual metrics. However, this more recent and performant optical flow estimator does not introduce improvement compared to RAFT. A likely explanation is that since we quantify the optical flow to $[0,1]$ scalar values for better generalization, our training strategies are less sensitive to the precision of the optical flow estimator.

P.S., the same experimental conclusions can be derived from other SOTA models like AMT as follows:

We will incorporate this discussion into the final paper.

Reference:

[1] Xu, Haofei, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. "Gmflow: Learning optical flow via global matching." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8121-8130. 2022.

Thanks to the author's detailed responses, which I think completely addressed my concerns. I maintain my initial rating and look forward to read the updated version.

Thanks for raising your concerns about the technical formulation, claim justification, experimental settings and analysis. Before going into the experiments, we would like to clarify some potential misunderstanding of our approach.

However, instead of learning to approximate or predict the distance map, they use the GT target frame to compute the distance map during training and use a uniform map (i.e. uniform speed) during inference. What is the point of attempting to address speed ambiguity during training, if the authors are assuming uniform speed at inference? How is the ambiguity being addressed at inference?

We would like to provide a more intuitive and detailed explanation on the ambiguity issue in video frame interpolation (VFI) and how we resolve it, which will be integrated into the final paper.

First and foremost, it's essential to clarify that velocity ambiguity can solely exist and be resolved in the TRAINING phase, not in the INFERENCE phase. The key idea behind our approach can be summarized as follows: While conventional VFI methods with time indexing rely on a one-to-many mapping, our distance indexing learns an approximate one-to-one mapping, which resolves the ambiguity during training. When the input-output relationship is one-to-many during training, the training process fluctuates among conflicting objectives, ultimately preventing convergence towards any specific optimization goal. In VFI, the evidence is the generation of blurry images in the inference phase. Once the ambiguity has been resolved using the new indexing method in the training phase, the model can produce significantly clearer results regardless of the inference strategy used.

Indeed, this one-to-many ambiguity in training is not unique to VFI, but for a wide range of machine learning problems. In some areas, researchers have come up with similar methods.

A specific instantiation of this problem: Let us look at an example in text-to-speech (TTS). The same text can be paired with a variety of speeches, and direct training without addressing ambiguities can result in a "blurred" voice (a statistical average voice). To mitigate this, a common approach is to incorporate a speaker embedding vector or a style embedding vector (representing different gender, accents, speaking styles, etc.) during training, which helps reduce ambiguity. During the inference phase, utilizing an average user embedding vector can yield high-quality speech output. Furthermore, by manipulating the speaker embedding vector, effects such as altering the accent and pitch can also be achieved. Here is a snippet from a high-impact paper which came up with the style embedding in TTS: “Many TTS models, including recent end-to-end systems, only learn an averaged prosodic distribution over their input data, generating less expressive speech – especially for long-form phrases. Furthermore, they often lack the ability to control the expression with which speech is synthesized.” from Wang, Yuxuan, et al. "Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis." International conference on machine learning. PMLR, 2018, cited by 816

Understanding this example can significantly help understand our paper, as there are many similarities between the two (e.g., motivation, solution, and manipulation).

(to be continued ...)

A minimal symbolic example to help readers understand better: Assuming we want to train a mapping function $\mathcal{F}$ from numbers to characters.

Training input-output pairs with ambiguity ($\mathcal{F}$ is optimized):

$1 \stackrel{\mathcal{F}}{\longrightarrow} a$, $1 \stackrel{\mathcal{F}}{\longrightarrow} b$, $2 \stackrel{\mathcal{F}}{\longrightarrow} a$, $2 \stackrel{\mathcal{F}}{\longrightarrow} b$

$\mathcal{\mathcal{F}}$ is optimized with some losses involving the input-output pairs above.

$\underset{\mathcal{F}}{\min} \quad L(\mathcal{F}(1), a) + L(\mathcal{F}(1),b) + L(\mathcal{F}(2),a) +L(\mathcal{F}(2),b),$

where $L$ can be L1, L2 or any other kind of losses (details on the losses are explained later). Because the same input is paired with multiple different outputs, the model $\mathcal{F}$ is optimized to learn an average (or, generally, a mixture) of the conflicting outputs, which results in blur at inference:

Inference phase ($\mathcal{F}$ is fixed):

$1 \stackrel{\mathcal{F}}{\longrightarrow} \{a,b\}?$, $2 \stackrel{\mathcal{F}}{\longrightarrow} \{a,b\}?$

Training without ambiguity ($\mathcal{F}$ is optimized):

$1 \stackrel{\mathcal{F}}{\longrightarrow} a$, $1 \stackrel{\mathcal{F}}{\longrightarrow} a$, $2 \stackrel{\mathcal{F}}{\longrightarrow} b$, $2 \stackrel{\mathcal{F}}{\longrightarrow} b$

In the input-output pairs above, each input value is paired with exactly one output value. Therefore, $\mathcal{F}$ is trained to learn a unique and deterministic mapping.

Inference phase ($\mathcal{F}$ is fixed):

$1 \stackrel{\mathcal{F}}{\longrightarrow} a$, $2 \stackrel{\mathcal{F}}{\longrightarrow} b$

Coming back to VFI: When time indexing is used, the same $t$ value is paired to images where the objects are located at various locations due to the speed and directional ambiguities. When distance mapping is used, a single $d$ value is paired to images where the objects are always at the same distance ratio, which allows the model to learn a more deterministic mapping for resolving the speed ambiguity.

It is important to note that fixing the ambiguity does not solve all the problems: At inference time, the "correct" (close to ground-truth) distance map is not available. In this work, we show that it is possible to provide uniform distance maps as inputs to generate a clear output video, which is not perfectly pixelwise aligned with the ground truth. This is the reason why the proposed method does not achieve state-of-the-art in terms of PSNR and SSIM (Table 1). However, it achieves sharper frames with higher perceptual quality, which is shown by the better LPIPS and NIQE.

We claim the "correct" distance map is hard to estimate accurately from merely two frames since there are a wide range of possible velocities. If considering more neighboring frames (more observation information), it is possible to estimate an accurate distance map for pixelwise aligned interpolation, which we leave for future work.

Furthermore, manipulating distance maps corresponds to sampling other possible unseen velocities, i.e., 2D manipulation of frame interpolation, similar to that mentioned TTS paper (Wang, Yuxuan, et al.).

P.S., the same experimental conclusions can be derived from other state-of-the-art models like AMT as follows:

The technical novelty of the work is quite limited.

The authors repeatedly use the term, "Our observation ...", yet fail to provide convincing empirical evidence

It may appear at first glance that this work lacks technical novelty since we do not propose any novel network architecture. However, we claim this is indeed the strength of our approach: By replacing time indexing with distance indexing as network inputs, the proposed approach can be applied to a wide range of VFI methods, driving consistent improvement in perceptual quality. We hope that through our additional explanation about our approach, Reviewer aknz can recognize our motivation and novelty as other reviewers did (Reviewer 1WoD, PtYf, and BPE4).

Please cite and discuss some relevant works [1,2].

[1] Exploring Motion Ambiguity and Alignment for High-Quality Video Frame Interpolation, CVPR 2023

[2] Deep Iterative Frame Interpolation for Full-frame Video Stabilization, SIGGRAPH Asia 2019

We appreciate the reminder and would like to point out that our paper does include a discussion about [1] in the second paragraph of the "Learning Paradigms" section. Additionally, we will incorporate a discussion of [2] in our final paper, which highlights the use of VFI techniques in an iterative manner for video stabilization.

What would be the added inference time by plugging the proposed method into existing methods?

Distance indexing: Transitioning from time indexing ($[T]$) to distance indexing ($[D]$) does not introduce extra computational costs during the inference phase, yet significantly enhancing image quality. So, the inference time is the same.

Iterative reference-based estimation ($[D,R]$): Given that the computational overhead of merely expanding the input channel, while keeping the rest of the structure unchanged, is negligible, the total inference time is equal to the number of iterations $\times$ the consumption of the $[D]$ model. We would like to highlight that this iterative strategy is optional: Users can adopt this strategy at will when optimal interpolation results are demanded and the computational budget allows. Our experiments (Table 2) show that 2 iterations, i.e., doubling the computational cost, yield cost-effective improvements. We will include these details in our final paper for clarity.

References:

[1] Su, Shuochen, Mauricio Delbracio, Jue Wang, Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang. "Deep video deblurring for hand-held cameras." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1279-1288. 2017.

[2] Sim, Hyeonjun, Jihyong Oh, and Munchurl Kim. "Xvfi: extreme video frame interpolation." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 14489-14498. 2021.

[3] Xu, Haofei, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. "Gmflow: Learning optical flow via global matching." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8121-8130. 2022.

Thank you for your suggestions about experiments!

Were all baseline trained using the same experimental setting? For instance, was RAFT used in all methods?

Yes, the distance maps used by each model were precomputed offline using RAFT.

Why did the authors only perform experimental comparisons on the Vimeo-90 K dataset? There are several VFI benchmarks that are commonly used to compare VFI methods. Why did not the authors use those benchmarks as well?

The septuplet set of Vimeo-90K is large enough to train a practical video frame interpolation model, and it represents the situations where the temporal distance between input frames is large (Please refer to the supplementary video for a better perception). Notably, state-of-the-art methods such as AMT (CVPR'2023) have struggled to train a clear arbitrary time video frame interpolator on this extremely challenging dataset, as claimed on their GitHub:

Zhen Li et al.: We previously attempted to train on Septuplet but did not achieve good results.

Thus, Vimeo90K (septuplet) can best demonstrate the velocity ambiguity problem that our work wants to highlight. With our disambiguation strategies, state-of-the-art models can be successfully trained on this dataset.

Nonetheless, we report more results on other benchmarks in the following. The following table shows the results of RIFE on Adobe240 [1] and X4K1000FPS [2] for 8x interpolation, using uniform maps.

Distance indexing [D] and iterative reference-based estimation [R] strategies can consistently help improve the perceptual quality. In addition, it is noteworthy that [D]$_u$ is better than [T]$_u$ in terms of the pixel-centric metrics like PSNR, showing that the constant speed assumption (uniform distance maps) holds well on these two easier benchmarks.

The problem the paper is trying to solve is more relevant in VFI situations where the temporal distance between input frames is relatively large. However, the authors do not show any experimental analysis in these scenarios.

As explained above, Vimeo90K (septuplet) dataset is the preferable dataset in terms of large temporal distance. We further show 16x interpolation on X4K1000FPS with larger temporal distance as follows:

The results highlight that the benefits of our strategies are more pronounced with increased temporal distances.

As the proposed distance map computation is heavily dependent on optical flow, did the authors experiment with using optical flow estimation methods other than RAFT?

We use GMFlow [3] to precompute distance maps for comparison. Our strategies still yield consistent improvement on perceptual metrics, as detailed below:

We will incorporate all the above content into the final paper.

(to be continued ...)

Thank you, Zhewei, for your generous recognition of our work and for providing an excellent baseline (RIFE) for our research.

In Appendix B, it seems more reasonable to use L1 loss?

Thank you for your suggestion. The proof was merely meant to show that the Equation 4 can be rigorously proven when L2 loss is used as a special case. For other losses, this specific equation no longer holds, but we empirically observe that the model still learns an aggregated mixture of training frames which results in blur (RIFE and EMA-VFI: Laplacian loss - L1 loss between the Laplacian pyramids of image pairs; IFRNet and AMT: Charbonnier loss). We will clarify this in the paper.

If the author focuses on visual effects, it would be meaningful to properly combine and compare the methods proposed in this paper with vgg loss.

We present the results of employing the more recent LPIPS loss [1] with a VGG backbone [2], as shown in the table below. Since [T] is directly trained to optimize the LPIPS loss at timestep $t$, it achieves slightly better LPIPS than [D]$_u$ and [D,R]$_u$, which use the uniform distance map as an approximation. Nevertheless, [D]$_u$ and [D,R]$_u$ yield significant improvement on the non-reference perceptual quality metric NIQE, consistently demonstrating the effectiveness of our strategies in resolving velocity ambiguity.

[1] Zhang, Richard, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. "The unreasonable effectiveness of deep features as a perceptual metric." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595. 2018.

[2] Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).