Generalizable Implicit Motion Modeling for Video Frame Interpolation

Thank you for the constructive comments. Please find the following for our response.

Q1:The authors use the test set ... I wonder the frame interpolation performance on Vimeo90k,...

A1: Due to the word limit, please refer to the global response and A5-2 in our response to Reviewer rbyn.

Q2: ... other options for arbitrary-timestep frame interpolation benchmarks, e.g., Adobe240fps [52].

Thanks for your suggestion. We calculate PSNR on the Adobe240fps following the evaluation settings of IFRNet [24] and use the test split divided by VideoINR [7]. The results are listed below:

Our method achieves stronger performance. This further illustrates the effectiveness of our method for the arbitrary-timestep VFI task.

Q3: What is the input / output of coordinate-based MLP (INR)

A3: We concatenate of the motion latent of $D \times H \times W$ and spatiotemporal coordinates of $3 \times H \times W$ as the input to the INR. The INR outputs the corresponding normalized flow.

Q4: ... why an INR form is necessary ... can't it be simply be in a form of a conditional U-Net, with target time-steps given as in diffusion literature?

A4: As described in Section 2, the INRs have effective modeling ability for complex, high-dimensional data and can learn continuous mapping from a set of coordinates to a specific signal. Thus, we are motivated to leverage the INR for continuous motion modeling at arbitrary timesteps. We experiment by replacing the INR with a timestep-conditioned U-Net as in diffusion literature. The results on VTF, VSF and model parameters are listed below:

Replacing INR with U-Net results in worse performance, especially on the 6X motion modeling benchmark VSF. This demonstrates the strong continuous modeling ability of INR. Besides, GIMM with INR has a much lighter architecture which improves the efficiency. Therefore, it is necessary and proper to use INR for continuous motion modeling.

Q5: ... the use of spatial coordinates do affect the performance, but I failed to understand the reason for this ...

A5: As explained in A4, we are motivated to use INR for its effective modeling ability, learning a continuous mapping from a set of coordinates to a specific signal. The coordinates provide positional information for INRs and help to learn continuous mapping since the coordinates are inherently continuous. In our motion modeling, we aim to model continuous motion between timesteps. The motion refers to dense optical flows, which consist of spatiotemporal changes. Therefore, it is necessary to use spatial coordinates for effective motion modeling. The experiment results in Table 2 demonstrate the necessity in practice.

Q6: why the spatial coordinates are necessary, as it will always predict the flow of its own spatial position.

A6: In addition to A5, we would like to further clarify that motion latent code serves as a function space for the implicit representations of different instances. The latent code is used for making INRs generalizable which means no need for test-time optimization. The spatiotemporal coordinates are still required for continuous modeling of spatiotemporal changes, such as dense optical flow in our case. Similar insights are shared in the related literature [5].

Q7: Generalizability:...

A7: As described in Section 2 (page 3) and Section 3.2 (page 4), “generalizability” means the generalizable modeling ability of the generalizable INRs (GINRs) across different instances. Unlike per-instance modeling INRs, GINRs do not require test-time learning. Our GIMM takes the motion latent code as an additional input to the INR and achieves generalizable motion modeling without the need for test-time optimization. A similar definition for generalizability can be found in the referenced paper [23].

Q8: How is the forward-warping method evaluated for motion modeling in Table 1?

A8: The forward-warping method produces 'backward' flows $F_{t→0}$ and $F_{t→1}$ from estimated flows $F_{0→1}$ and $F_{1→0}$ without flow reversal techniques. For example, $F_{t→0}$ can be easily obtained from $F_{t→0}=Fwarp(F_{1→0}\cdot t,F_{1→0}\cdot(1-t))$, where $Fwarp(a,b)$ indicates forward warping the objective $a$ with the referenced motion vector $b$.

Q9: I wonder the computational costs of the framework.

A9: Due to the word limit, please kindly refer to the global response.

Q10: How is the normalization of flows done?

A10: The normalization process follows IFE[13]. We agree with the reviewer and will delete this part from our key designs in our revised manuscript.

Q11: Motion encoding: … the experimental studies on the effectiveness of motion encoding cannot be found in the paper…

A11: Thanks for your suggestion. Due to the word limit, please refer to the global response.

Q12: What is the input / output of the Latent Refiner?

A12: The Latent Refiner refiner takes both the motion features and the coarse motion latent (warped motion features) as input and outputs the residual for the coarse motion latent. We will update the figure of Latnet Refiner’s architecture in the supplementary of our revised manuscript.

Q13: Presentation. I feel that some important details, ..., What is the input / output of the INR model, how are the input / output tensors shaped, etc….

A13: Thanks for your suggestion. We will specify more details about the INR in the revised manuscript.

Thank you for the detailed response.

Q1. Vimeo90k

My concerns on Vimeo90k has been partially addressed. To make myself clear, my initial concerns on Vimeo90k was raised since I could not understand the main reason for using VTF and VSF as a main benchmark, as it is on flows, which is not the ultimate goal of VFI.

Although It could be a benchmark to show that the proposed framework models motion fields properly, I believe it is a limited benchmark, which could only serve as an assistance to frame reconstruction, for deeper analysis on the method, and cannot be a main benchmark for comparing the validity as a VFI method.

Yet, the authors seem to use the VTF and VSF as one of the main benchmarks, considering the responses in the rebuttal. I wonder the reason for this, as the performance on flow reconstruction does not necessarily mean better frame interpolation results, although it has a correlation.

If the reconstruction performance between the flows and frames perfectly correlate, I think it is reasonable to report the frame reconstruction performance on original benchmarks, rather than reporting the flow reconstruction performance with a modified benchmark.

Could the authors provide further explanations on this matter?

Q3 - Q6. Necessity of spatial coordinates

Thank you for the response.

However, I still struggle to understand and agree with the use of spatial coordinates. The authors cite [5] as a work that shares the same insight, but to my understanding, there is a crucial difference to [5]. In [5], the model takes relative spatial coordinates between the query coordinate and the reference latent code as the coordinate input.

Let $(x,y)$ be spatial coordinates of an image. The work of [5] is on image super-resolution, and they use coordinates of $(0,0), (0,1), (1,0), (1,1)$ to predict the RGB of $(a,b)$, where $0<a,b<1$. Their INR model uses latent codes at $(0,0), (0,1), (1,0), (1,1)$ along with their corresponding relative coordinates to the query coordinate, $(a,b), (a, 1-b), (1-a, b), (1-a, 1-b)$, respectively, as inputs. Formally, it would be as follows: $z_{(a,b)}^1 = f(z_{(0,0)}, (a,b)), z_{(a,b)}^2=f(z_{(0,1)}, (a, 1-b)), z_{(a,b)}^3 = f(z_{(1,0)}, (1-a, b)), z_{(a,b)}^4 = f(z_{(1,1)}, (1-a, 1-b))$, where $f$ is the INR function.

I believe this is meaningful, as the goal of the INR model output is to predict the RGB of a different coordinate to the input latent code.

However, according to GIMM’s formula, the INR model predicts the feature of the same position to the motion latent code, of which I think is not necessary. Putting the GIMM formula in the form of equation above, it would be something like this: $Z_{(a,b)} = f(z_{(a,b)}, (a,b))$. The latent code of $(a,b)$ predicts the feature of the same location, different to the formula of LIIF [5]. Here, I do not think the coordinate information of $(a,b)$ is necessary to the function $f$.

For further explanation, Eq.2 of [5] takes $x_q - v^*$ as the coordinate input of the INR model, whereas I believe the GIMM formula takes $x_q$ itself as the coordinate input. $x_q$ denotes query coordinate and $v^*$ denotes the coordinate of reference (key) latent code. This is very different.

The authors express continuous mapping using the spatial coordinate inputs, but it still fails to convince me, as this task does not require continuous representation in terms of the spatial dimension, with the input and output sharing the same spatial size.

Q7. Generalizability

Thank you for the clear response. It has been well addressed that the term ‘generalizability’ has been used different with my understanding, and in that case, the use of such term is understandable. However, the main part that caused the confusion and still concerns me is, “Our GIMM can be smoothly integrated with existing flow-based VFI works without further modifications”, which is claimed in the abstract and conclusion. This part has not been shown adequately, without experiments, and I still feel that this part is an overclaimed part.

Q8. Forward warping

Thank you for the clarification. I think it would be better if described in the paper.

Q2, Q9-Q13.

Thank you for the clarification.

Although some of my questions have been clarified, the clarified parts are mostly on further description of the method. Yet, I still believe that the current version of the paper contains several overclaims (e.g., necessity of spatial coordinates, generalizability to existing VFI works).

Thanks for your feedback on our response. We would like to clarify your current concerns further. Please find the following for our response.

Necessity of spatial coordinates

First of all, we would like to highlight our explanations of using spatial coordinates, as described in our previous responses.

1. Our GIMM aims to perform continuous motion modeling. The modeled motion is dense optical flows, which are of spatiotemporal changes. Therefore, GIMM aims to continuously model spatiotemporal changes.

2. We use INR to achieve continuous modeling since INR learns continuous mapping. The key fact for INR to learn continuous mapping is that the input coordinates are continuous.

3. Latent code serves as a function space for the implicit representations of different instances [5]. The latent code is not the key to learning continuous mapping.

According to the above three points, we clarify that using spatiotemporal coordinates is necessary for our GIMM to continuously model the motion of spatiotemporal changes. We would also further make some specific clarifications according to the current feedback.

Q: The explanations they provided with an example of a moving ball, to my understanding, could make sense in a forward-warping based approach, but not in a backward-warping based approach.

A: We use the moving ball example as a piece of evidence for clarification in our above explanation 1), dense optical flows are of spatiotemporal changes. The spatiotemporal changes exist in all optical flows and all flows are faced with the problem of occlusion. This point is irrelevant to the warping approaches.

Q: According to their method, the latent codes of each spatial position would contain the history of pixels going through the corresponding locations with time, and would already contain the information necessary for their own location, which would only necessitate temporal timestep conditioning.

A: As described in the above explanation 2) and 3), the latent code is used to make INR generalizable. The key to achieving continuous modeling is the coordinates rather than the latent code. The latent code does not assure continuous modeling .

Notably, the goal of our method is to continuously model the motion of spatiotemporal changes. Therefore, it is necessary to include spatiotemporal coordinates as the inputs to our INR in GIMM.

Q: If not predicting a flow of position different to the latent code, the spatial coordinate input for INR still seems redundant.

A: The goal of INR is to perform continuous modeling for its target signal, i.e., dense optical flows in our case. Simply using the latent code as input will not make continuous modeling. Since the flows are of spatiotemporal changes, the spatiotemporal coordinates are necessary for continuous modeling.

In fact, the usage of spatiotemporal coordinates in INR can also be found in related literature [41].

Plug-in ability

Q: In response A5-1 to the reviewer rbyn, the authors said “Notably, plugging in a better continuous modeling module doesn’t guarantee better model fitting since model fitting requires more on the model’s learning strategies and its overall design.” I think this statement contradicts the claim of their paper, where they say that their method “can be smoothly integrated with existing flow-based VFI works without further modifications”, although they somehow succeeded in integration with existing VFI works.

A: Please consider the full context of the quoted sentence from our response in A5-1. Here is the complete description with the context:

“We would like to clarify that our method GIMM focuses on continuous motion modeling, which further enables frame interpolation at arbitrary timesteps. Arbitrary-timestep interpolation relies more on continuous modeling while fixed-timestep interpolation relies more on model fitting at the specific timestep of 0.5. Notably, plugging in a better continuous modeling module doesn’t guarantee better model fitting since model fitting requires more on the model’s learning strategies and its overall design.”

We would like to summarize and further clarify the following:

1. Our method aims to perform continuous motion modeling. The continuous modeling ability further enables frame interpolation at arbitrary timesteps. Since the task we focus on is interpolation at arbitrary timesteps, the plug-in ability we referred to is for enhancing the arbitrary-timestep interpolation ability of the existing flow-based VFI works.
2. Arbitrary-timestep interpolation relies more on continuous modeling while fixed-timestep interpolation relies more on model fitting at the specific timestep of 0.5. The quoted sentence is used to explain that a module designed for the arbitrary-timestep interpolation task doesn’t guarantee its performance on the fixed-timestep interpolation task.

Therefore, the quoted sentence neither contradicts our claim of plug-in ability nor holds relevance to it.

Q: The integration, for instance, to IFRNet, could have been possible, but not smoothly, as several components of IFRNet are not theoretically aligned well with the proposed method.

A: By integrating GIMM into existing flow-based VFI methods, we mean to use the flows from GIMM to replace the original flows in the VFI methods. For instance, regarding the IFRNet, we simply replace all the flows of IFRNet with the flows predicted by GIMM. We keep the architecture of the IFRNet exactly the same. We believe our operation of simply replacing the flows with ours (from GIMM) can be described as ‘smooth’.

The same plug-in operation can also be found in Section 3.3 of the reference [13].

Due to the limited discussion period, I leave my response on ones that could be quickly replied first, and may possibly fail to reply to all matters.

Continuous modeling

The authors emphasize the continuous modeling ability. Although their proposed design of using INRs for continuous modeling is novel, there are numerous methods capable of continuous modeling of flows. Many recent works either scale the bidirectional flows using the target timestep, i.e., in forms such as $F_{0 \rightarrow t} = t \times F_{0 \rightarrow 1}$, or use a conditioning method with the target timestep, and this way, continuous modeling of flows is in fact possible [1, 16, 18, 20, 21, 24, 32, 33, 42].

It is acceptable if the authors claim that their continuous modeling is more precise, but the claim that it is impossible for these existing methods to provide continuous motions in a plug-in form is not true. All of the methods cited above are capable of continuous estimation of flow maps at arbitrary timesteps, and their intermediate flows could also be plugged into other frameworks, in the sense of the authors use. Yet, none of these works do not claim their work to have a plug-in ability.

Thus it is hard to say that their method has a special plug-in ability.

IFE [13]

The same plug-in operation can also be found in Section 3.3 of the reference [13].

I belive this statement of the authors implies that [13] uses an plug-in operation, and I think the authors tried to use [13] to support that their plug-in strategy is valid. To my understanding, I think the authors' intention is that [13] do not have a plug-in ability, as their flow modeling method is "not generalizable", and is trained per-instance. However, I rather do not find this part important in the 'plug-in' perspective. If the authors claim 'generalizability' is the important part for plug-in ability, as mentioned above, numerous methods existing could also be said to have a 'plug-in' ability.

IFRNet

The experiments with IFRNet still does not sound to be be a 'smooth integration'. For a method to be easy to be 'plug-and-play' form, a framework needs to be capable of disentanglement. For instance, in flow-based VFI, a method should have explicitly separated parts of flow estimation and frame synthesis, so that the estimated flows can be smoothly replaced. However, IFRNet's contribution is 'joint' prediction of flows and features, which means that the prediction of flows and features could be entangled, and thus I find it awkward to replace the flows by predictions from GIMM.

If simple replacement of estimated flows on other flow-based VFI methods can be considered ‘smooth integration’, then majority of flow-based VFI methods can also claim the same thing, by replacing estimated flows by each other. In that case, it is no longer a novelty / contribution of the method.

We would like to clarify that our core module GIMM aims to perform continuous motion modeling.

GIMM can take flows between the input frames and predict intermediate flow at any given timesteps. Furthermore, GIMM can also achieve good performance with different flow estimators, which makes it quite flexible. Notably, in Table 3 (page 8, GIMM-VFI-F vs. GIMM-VFI-R), we show that integrating a better flow estimator (FlowFormer [17]) can enhance model performance.

To the best of my knowledge, no other existing VFI methods contain a similar module that i) is designed for continuous motion modeling with flows as input and ii) can be augmented flexibly with different flow estimators. Therefore, it is quite impossible or much harder for these existing VFI methods to provide proper continuous motion to achieve the plug-in ability claimed by us.

By plugging in GIMM, existing VFI methods can achieve better arbitrary-timestep interpolation performance as proven in A5-1 of our response to the reviewer rbyn.

Therefore, we believe that our GIMM is novel and has an effective plug-in ability.

The authors cite [13], but the work of [13] do not mention / claim their method to have a great plug-in ability. As part of their framework, they simply use an existing flow-based VFI work for blending / frame synthesis.

As described in our previous response, the specific operation of plugging in is to use the flows from GIMM to replace the original flows in the VFI methods. We cite IFE[13] because the same operation has been used in it.

As described in the “Implicit neural representations.” paragraph in Section 2 (page 3), IFE [13] also considers implicit flow modeling but it focuses on per-instance modeling and it is NOT generalizable. Therefore, a plug-in ability to existing VFI methods across various instances is not claimed in their paper.

We believe that IFE[13] does not contradict our claim.

Thank you for the constructive comments. Please find the following for our response.

Q1: The K, F, and Z symbols in Equations 6-7 should be labeled in Figure 2. In addition, the description of L150-151 for K is confusing to read, especially F is expressed as the difference between two coordinates.

A1: Thanks for your suggestion. We will add the K, F, and Z symbols to Figure 2 and reformulate the description of L150-151 in our revised manuscript.

Q2: How does Equation 11 get the optical flow of GT? As far as I know, none of the existing VFI datasets have a corresponding optical flow.

A2: As described in Section 4 (page 6), we utilize Flowformer [17] to produce pseudo-ground truth optical flow for training and evaluation usages.

Q3: Table 1 allows for a comparison of some of the methods that model more complex motion than just modeling linear motion, such as QVI, ABME, BMBC.

A3: Thanks for your suggestion. We would like to clarify that our GIMM focus on continuous motion modeling for arbitrary-timestep interpolation with 2 frames as the input. While QVI takes 4 frames as input and ABME can only predict at the timestep of 0.5, BMBC is the only method that allows for continuous motion modeling and interpolation. Therefore, we extend Table 1 with experiments on BMBC. The results of BMBC’s motion modeling and interpolation compared with GIMM are listed below.

Our GIMM outperforms BMBC concerning both motion modeling and interpolation. This further demonstrates the effectiveness of our method. We will add this to Table 1 (page 7), in our revised manuscript.

Q4: Table 3 should include parameter and runtime comparisons to demonstrate the effectiveness of the GIMM.

A4: Please kindly refer to the "Parameters and Runtime" section in the global response.

Q5: As a plug-and-play motion modeling module, Table 3, the authors should add GIMM to other existing flow-based VFI methods for a fairer comparison, e.g., ABME, TTVFI, RIFE, and IFRNet. Because the design of the optical flow estimation network and synthesis network used in the different methods are not the same, and the use of a higher performance optical flow estimation network also has a significant performance improvement. In addition, validation on vimeo and UCF101 is necessary.

A5-1: We would like to clarify that our method GIMM focuses on continuous motion modeling, which further enables frame interpolation at arbitrary timesteps. Arbitrary-timestep interpolation relies more on continuous modeling while fixed-timestep interpolation relies more on model fitting at the specific timestep of 0.5. Notably, plugging in a better continuous modeling module doesn’t guarantee better model fitting since model fitting requires more on the model’s learning strategies and its overall design. Therefore, in terms of continuous modeling, we evaluate the plug-in ability of our proposed GIMM on arbitrary-timestep interpolation benchmark SNU-FILM-arb rather than fixed-timestep interpolation benchmarks, such as Vimeo90K and UCF101. Since there is a time limit for the rebuttal, we plug in the GIMM module to two of the representative existing flow-based VFI methods, TTVFI and IFRNet, for the experiment. Particularly, we plug in GIMM with a pretrained flow estimator RAFT to IFRNet, since there is not an existing flow estimator in the model. The results of PSNRs are listed below.

Plugging in the GIMM module results in significant improvements for arbitrary-timestep interpolation. This demonstrates the effectiveness of GIMM for continuous modeling when integrated with existing flow-based VFI works. We will add this experiment to the Supplementary of our revised manuscript.

A5-2:In order to prove GIMM-VFI's effectiveness of the overall design for interpolation, we further provide evaluations on Vimeo90K and UCF101. Following EMA-VFI [50], we train GIMM-VFI for fixed-timestep interpolation. For evaluation, we calculate PSNRs and compare our method GIMM-VFI with the aforementioned and other state-of-the-art methods. The results are listed below:

Two variants of GIMM-VF with different flow estimators, both achieve competitive performance. This further demonstrates our method's strong interpolation ability.

Dear Reviewer rbyn,

We sincerely thank you for the review and comments. We have posted our response to your initial comments, which we believe has covered your concerns. We are looking forward to your feedback on whether our answers have addressed your concerns or if you have further questions.

Thank you!

Authors

Thank you for the constructive comments. Please find the following for our response.

Q1: The GIMM starts from pretrained flows, which may limit the network performance. In fact, recent VFI networks still tend to estimate blurry results according to inaccurate flow estimating while facing hard cases.

A1: Please kindly refer to the "Pretrained flow estimator" section in the global response.

Q2: More perceptual quantitative indicators need to be shown in the paper, eg. FID or LPIPS.

A2: Thank you for the suggestion. We add the perceptual metrics FID and LPIPS to Table 3 (page 8). We report the best results in boldface and the second best with Italic boldface. The results are listed below:

Our proposed method GIMM-VFI achieves competitive performance according to these perceptual quantitative indicators.

Q3: Is the flow normalization model necessary?

A3: As described in the “Flow normalization” paragraph in Section 3.2 (page 4), the normalization process aligns temporal direction and scales the values of input flows. First, without temporal direction alignment of the inputs, it would be hard to define the direction of the output flow $V_t$. The direction alignment is thus necessary. Second, for the scale operation, we experiment by skipping it and present results on VTF and VSF as below:

Skipping the scale operation performs much worse than the full GIMM settings across both benchmarks. It is therefore necessary to perform the complete flow normalization operation.

Q4: What are potential temporal inconsistencies mentioned in line 146?

A4: The potential temporal inconsistencies refer to the possible bias and noises in the estimated flows between input frames. In order to mitigate their negative impacts on motion modeling, we employ a Motion Encoder (ME) to extract motion features. The ablation experiment on Motion Encoder can be found in the global response.

Dear Reviewer d4Ah,

We sincerely thank you for the review and comments. We have posted our response to your initial comments, which we believe has covered your concerns. We are looking forward to your feedback on whether our answers have addressed your concerns or if you have further questions.

Thank you!

Authors

Thank you for the constructive comments. Please find the following for our response.

Q1: Novelty: The idea of using time-dependent generalizable INR is not new and has been explored in human motion synthesis [1]. [1] also uses the concatenation of spatial coordinates, time step and motion latent as the input for an INR MLP.

A1: We respectfully disagree with the reviewer that the idea of using time-dependent generalizable INR for motion modeling in Video Frame Interpolation lacks novelty due to the presence of NeRM. Although NeRM and our proposed method GIMM both leverage INR conditioned on coordinates and latent, they are quite different concerning the following main points. 1) GIMM performs the general type of motion modeling in the context of video frame interpolation. Unlike NeRM which focuses on human motion, GIMM models any types of motion that may exist within the input frames. 2) While NeRM synthesizes sparse pose motion at each timestep, GIMM predicts dense motion in the form of optical flow for the usage of interpolation. Consequently, GIMM requires spatial coordinates as the additional input to its INR for better modeling, akin to image-based INRs [5]. In contrast, NeRM only takes temporal coordinates. We plan to include NeRM in our references to enhance the comprehensiveness of our related work section. We will add this discussion in the revised manuscript.

Q2: Moreover, the way of using INR is very similar to CURE[41]... performance improvement over CURE might be attributed to the pre-trained flow estimator.

A2: As described in the “Generalizable INRs.” paragraph in Section 2 (page 3), CURE directly learns generalizable INRs from video, while our method leverages generalizable INRs for motion modeling to improve intermediate frame synthesis for flow-based VFI. Besides, we would like to clarify that despite both CURE and our method have leveraged pre-trained flow estimators, i.e., RAFT, CURE still performs linear motion modeling with a strong assumption of motion overlapping that should lead to suboptimal results. As observed from Table 3, our GIMM-VFI-R consistently outperforms the CURE algorithm across all benchmarks, achieving PSNR improvements exceeding 1 dB. Additionally, GIMM-VFI-R effectively handles interpolation at 4K resolution where CURE encounters out-of-memory issues. Furthermore, CURE contains a heavier architecture of 51.28M parameters (31.49M larger than our GIMM-VFI-R) and requires a longer time in inference.

Q3. Performance: In Table 1, GIMM (-VFI-R) only slightly outperforms a simple "Linear" approach on both Vimeo-Septuplet-Flow (VSF) and SNU-FILM-arb-Hard.

A3: As observed in Table 1 (page 7), the PSNR of our method is 0.36 dB and 0.20 dB higher than the "Linear" approach on the referred benchmarks, respectively. This improvement is significant, particularly considering that both methods utilize the pretrained flow estimator RAFT.

Q4: Also, in Table 2, without spatial coordinates, the performance slightly drops.

A4: As shown in Table 2 (page 8), removing spatial coordinates causes a 0.16 dB drop of PSNR on VTF and a 0.06 increase of End-Point-Error on VSF. This demonstrates the crucial role of spatial coordinates.

Q5: Do you even conduct ablation on the time variable? Maybe with only motion code as the input, the performance still remains similar?

A5: As presented in Table 2 (page 8) and analyzed in the “Implicit modeling.” paragraph in Section 4.2 (page 7), direct input motion latent code without using any coordinates results in a 0.52 dB decrease of PSNR on VTF and a 0.13 increase of End-Point-Error on VSF. This highlights the importance of implicit modeling in GIMM.

Q6: The method relies on the pre-trained flow estimators, which may lead to a suboptimal solution.

A6: Due to the word limit, please refer to the "Pretrained flow estimator" section in the global response.

Q7: The paper concatenates the motion latent code and the coordinates as the input to achieve generalizable INR. It seems the latent code dominates the input as it has a higher dimension. Will it make the INR less spatial-sensitive?

A7: Thank you for the suggestion. We reduce the dimension of motion latent from 32 to 8 and the results on VTF and VSF are listed below:

Reducing latent dimension leads to worse performance, with reductions of 0.40dB and 0.30dB in the PSNRs on VTF and VSF respectively. This indicates that a proper choice of a higher dimension, e.g., 32, will help generalizable INR achieve better performance rather than impose negative effects on the implicit modeling.

Q8: A follow-up question is why not use meta-learning approaches or INR modulation [6]?

A8: As suggested, we implement the Meta-learning approach [23] for modulating the weights of INR, specifically in modeling motion. We compare its performance on VTF and VSF with our method GIMM, and calculate their parameters. The results are listed below:

The Meta-learning approach performs much worse than GIMM while adding more than 170x larger model parameters.

Q9: Typo: "Latnet Refiner" in Figure 2 caption.

A9: We will correct the typo in our revised manuscript.

Dear Reviewer LUi9,

We sincerely thank you for the review and comments. We have posted our response to your initial comments, which we believe has covered your concerns. We are looking forward to your feedback on whether our answers have addressed your concerns or if you have further questions.

Thank you!

Authors