Aligning Anything: Hierarchical Motion Estimation for Video Frame Interpolation

Writing quality: The paper was difficult to read. I noticed more language errors than in a typical submission, and the core technical contributions (e.g. how the aggregation works) should be illustrated more clearly.

Experiment suggestion: Use ground truth semantic segmentation to demonstrate the upper bound of model accuracy and how much the quality of the segmentation matters.

1.I believe that introducing SAM to the VFI task is not valuable. Because the number of moving objects in the video cannot be determined, introducing SAM to distinguish each object will limit the generalization ability of the model and consume unstable resources. Specifically, it is recommended that the authors add experiments to demonstrate that SAM achieves the desired segmentation on VFI datasets, and that the algorithm is robust enough for different fundamental segmentation models. In addition, it is recommended that the authors analyze the frame interpolation time and resource consumption for different videos, especially for videos with only one object and those with many objects.

2.The theoretical analyses in section 3.1 are more like an introduction to preliminary work. This is because the formulas are commonly acknowledged in VFI and are not specifically linked to the authors' design. It is suggested that the authors connect video frame interpolation to the segmentation prior through theoretical analyses, in order to justify the introduction of SAM.

3. The rigid and non-rigid objects mentioned in the paper are not essentially resolved, and for non-rigid objects, SAM cannot obtain masks for intermediate frame moments. Can the authors give a detailed explanation of SAM handling non-rigid objects.

4. The mask generation process in this paper utilizes three open vocabulary models, RAM, DINO, and HQ-SAM, along with some post-processing procedures. The performance of the model is affected by them and the paper needs to be supplemented with quantitative experiments for each of them. For example, can the authors list in detail the quantitative metrics of object detection and segmentation such as mAP, mIOU, etc. to prove that DINO and SAM can cover all the objects in the video accurately enough.

5. Expanding the range of datasets used for testing could further validate the method’s robustness across different video types and scenarios. Including more diverse or challenging datasets would demonstrate broader applicability, such as XVFI, GOPRO, etc.

6. The paper could discuss potential real-world limitations or constraints, e.g., performance at frame rates X4, X8, non-uniform motion, rotation, etc.

• Although the performance improvements are somewhat consistent throughout various baseline algorithms and datasets, the performane gains obtained by the proposed HCE modules seem very marginal considering the hefty additional computational load on top of the baseline algorithm.

• Since the goal of a video frame interpolation model is to synthesize the intermediate frames given consecutive set of input frames, the input frames often include fast-moving objects with varying degree of blurry artifacts. Given this nature of the VFI problem, I have two questions.

  (1) How successful do baseline detection and segmentation algorithms HQ-SAM, RAM, and DIDO perform on blurry frames? It seems that the semantic information obtained from these algorithms play a crucial role for the proposed framework, and missed detections and segments can be critical for the performance.

  (2) Considering the aforementioned aspects, using binary segmentation masks seems inappropriate given that the object boundary can be often blurry due to varying degree of motion blur. It seems that using soft masks rather than hard binary masks can alleviate this problem. Are there any experimental validations for this?

• Although using open-set models such as RAM and DIDO seems general enough for many cases and scenarios, explicitly tagging object categories, finding bounding boxes, and obtaining segmentation masks seems to limit the capability of the proposed framework to pretrained concepts. Can the proposed method utiilze semantic information from objects of unseen category?

• Minor points

  There are many typos and grammatical errors throughout the entire manuscript, and even in the abstract section. I suggest the authors to perform a full top-to-bottom proofreading of the manuscript.

1. The proposed method in the article lacks novelty, primarily focusing on enhancing the video frame interpolation task through the utilization of foundational models such as HQ-SAM, Grounding DINO, and RAM. While it is foreseeable that the introduction of these methods could improve the performance of video frame interpolation, such enhancements are insufficient to support a highly impactful publication. I hope to see more insightful contributions from the authors in this regard.

2. The experimental section of the article does not demonstrate a clear performance gain from integrating this method into existing approaches. It is possible that achieving score improvements in the field of video frame interpolation (VFI) is inherently challenging; nonetheless, I would appreciate it if the authors could provide a comparison of other plug-and-play enhancement techniques alongside their proposed method to illustrate its relative superiority.

3. The qualitative comparison presented in Figure 6 of the article lacks necessary guidance and labeling, making it difficult for readers to discern the intended information. If the authors aim to highlight specific differences, I suggest improving the image's readability through localized zoom-ins and the addition of explanatory text.

4. The authors' experiments appear rather inconclusive, and I would like to see the distinctive advantages of leveraging semantic information for motion estimation. Specifically, what unique features emerge within the network when it is equipped with comprehensive semantic input? I hope the authors can derive more meaningful conclusions based on this foundation.