Controllable Human-centric Keyframe Interpolation with Generative Prior

We really appreciate your constructive comments. Please find the following for our response.

Q1: [Weakness 1] It is not clear if the groundtruth intermediate frames are used to extract line features for FCVG when evaluating against FCVG in Table 1 and 2. If not, this is a weakness in the evaluation as motion ambiguity should be removed for FCVG as well for a fair comparison to this method.

A1: As described in the `Evaluation Protocol' paragraph (Sec. 5) of our main paper, we would like to clarify that intermediate ground-truth control signals are leveraged for evaluating all the controllable interpolation methods, including FCVG. We will revise the paper to clarify this point more explicitly in the updated version.

Q2: [Weakness 2] In line with the previous point, I think the approach suggested in section 1.4 of the supplementary material is a critical evaluation criterion for this method. Current method relies on extracting the control signals from the groundtruth intermediate frames. However keyframe interpolation as a problem relies on just the start and end frames and doesn’t not have access to the intermediate frames. So, to support the claim of a keyframe interpolation method, this work should show performance on a typical keyframe interpolation task.

A2: We fully agree that it is important to evaluate interpolation methods in scenarios where no ground-truth intermediate control signals are available. We define this setting as `in-the-wild interpolation' in our supplementary material. In response to your suggestion, we conducted additional evaluations on the CHKI-Video test set for both FCVG and our proposed method, using only the start and end frames as inputs, without any intermediate information. The results are summarized in the table below:

We observe that under this setting, PoseFuse3D-KI consistently outperforms FCVG across all metrics, demonstrating the robustness and effectiveness of our method for in-the-wild interpolation. We will add this table in the updated version of our supplementary material.

Q3: [Question 1] Can you report numbers on an approach similar to what you describe in section 1.4 of the supplementary material? This will help address point 2 of the weaknesses.

A3: Thanks for your suggestion, we have conducted quantitative evaluations for the in-the-wild interpolation setting (described in section 1.4 of the supplementary material). The resulting table is presented in A2.

Q4: [Question 2] Can you clarify if lines extracted from groundtruth intermediate frames are used for evaluating FCVG (instead of the interpolations from the start and end frames). If not, can FCVG be evaluated in such a set-up? This will help address point 1 of the weakness.

A4: All evaluations of FCVG presented in both the main paper and the supplementary material were conducted with intermediate ground-truth control signals. We will revise the paper to clarify this point more explicitly in the updated version.

Thank you for the constructive comments. Please find the following for our response.

Q1: [Weakness 1] Limited Discussion on Scalability: … how well the method scales with different video lengths or more complex human actions, particularly in large-scale applications. [Question 2] What is the maximum temporal gap (in terms of frames or seconds) that the proposed method can effectively interpolate?

A1: Thank you for your suggestion to evaluate the scalability of our method across different video lengths and temporal gaps. We have conducted additional evaluations of PoseFuse3D-KI on varying temporal gaps, and the results are summarized in the table below:

As with most interpolation methods, performance decreases as the temporal gap increases. Nevertheless, PoseFuse3D-KI maintains competitive performance across a wide range of gaps, especially for 24 and 48 frames. At 96 frames, while performance degradation is observed (e.g., PSNR drops below 20 dB and LPIPS exceeds 0.2000), the method still provides reasonable results. These results demonstrate the robustness of PoseFuse3D-KI in long-range controllable interpolation settings.

Q2: [Weakness 2] Reproducibility Concerns: The code and data are not currently available, … limits the reproducibility of the results. [Limitation 3] Future Work on Data and Code Availability: …, the current lack of access limits the paper's ability to be fully reproducible and verifiable by the research community.

A2: We fully agree that open-sourcing code and data is important for the research community. We will try to make our database and codebase publicly available as soon as possible.

Q3: [Weakness 3] Computational Efficiency: … discussion about the computational efficiency of the framework or how it scales with increasing complexity or dataset size.

A3: Thank you for your valuable feedback regarding computational efficiency. We have conducted a runtime comparison between our method and existing baselines. Specifically, we measured the time required to interpolate 25 frames at a resolution of $1024 \times 576$ on an NVIDIA A800 GPU. The results are shown below:

Our method achieves the fastest runtime among the evaluated approaches, indicating comparatively superior computational efficiency.

Q4: [Weakness 4] Potential Overfitting: …, it would be useful to see how the model generalizes to other datasets or human motion tasks to avoid potential overfitting. [Limitation 1] Dependency on Specific Datasets: …The method's performance on other datasets or real-world video sources remains untested.

A4: We appreciate the reviewer for this valuable suggestion. To further evaluate generalizability, we conducted both qualitative and quantitative experiments on the FCVG test set, as described in Section 1.5 of the supplementary material. The results, shown in Table 1 and Figure 2, demonstrate that PoseFuse3D-KI consistently outperforms all baseline methods in terms of quantitative metrics and produces robust human-centric interpolation in challenging cases, such as the `Breaking Dance' case in Figure 2. These results support the generalization capability of our method beyond the CHKI-Video dataset.

Q5: [Weakness 5] Limited Practical Expandability: The reliance on 3D motion information (e.g., SMPL-X parameters) may hinder the method’s applicability in real-world scenarios … The lack of fallback strategies for 2D-only input scenarios limits the applicability of this framework. [Question 5] In some cases where human SMPLX parameters are not available … In other words, did the authors consider scenarios where 3D information is not available? [Limitation 2] Dependency on SMPLX Parameters: The method proposed by the authors, although capable of generating higher quality videos, is overly reliant on 3D information and lacks extensibility.

A5: Thank you for the insightful suggestion. We address scenarios where only start and end frames are available, without ground-truth 3D annotations, as in-the-wild interpolation, described in Section 1.4 of the supplementary material. In this setting, we estimate SMPL-X parameters from the input frames and apply linear joint interpolation to generate intermediate SMPL-X controls. Figure 1 in the supplementary material illustrates the qualitative results, showing the effectiveness of this strategy. Additionally, we conducted quantitative comparisons with FCVG under this setting. The results are shown below:

Our method consistently outperforms FCVG across all metrics, demonstrating its effectiveness for the in-the-wild interpolation setting. Furthermore, our strategy has the potential to incorporate advanced techniques such as text-to-motion models for generating intermediate human model controls, enabling more flexible and effective interpolation for complex motion.

Q6: [Question 4] Regarding the 3D auxiliary information aspect, does the proposed method require the SMPLX parameters for all frames, or only for the first and last frames? If the former is the case, can it be argued that the method actually draws on some information from ground truth rather than being truly effective?

A6: Our PoseFuse3D-KI framework requires intermediate SMPL-X controls to guide the interpolation process. However, these controls do not need to be ground-truth. As described in Section 1.4 of the supplementary material and elaborated in A5, the intermediate SMPL-X parameters can be obtained via simple interpolation or possibly generated using text-to-motion models. We emphasize that the method remains effective even in the absence of ground-truth 3D annotations. This is further supported by the quantitative results we present in A5 under the in-the-wild interpolation setting. This demonstrates the strong performance of our method even when only the first and last frames are available.

Q7: [Question 1] How does the framework perform on a wider range of human actions … Could the method be generalized to other domains, such as robotics or virtual character animation?

A7: While PoseFuse3D-KI is designed to perform plausible interpolation for real-world human motions with complex dynamics, it can also be applied to other video domains such as robotics and virtual character animation. In our tests, it has demonstrated promising results in these settings. However, due to limitations of the discussion format, we are currently unable to include the qualitative results here. These examples will be added to the supplementary materials in the revised version of the paper.

Q8: [Question 3] … how does the model address the change in background after a large change in character position?

A8: We would like to clarify that PoseFuse3D-KI is designed to address controllable human-centric interpolation, with a focus on plausibly interpolating foreground humans with control signals. For the background, our framework relies on the underlying adapted video diffusion model to synthesize the in-between frames, without explicit control signals. While this approach works well in many cases, we acknowledge that it may lead to artifacts under challenging conditions such as large background changes. Addressing this limitation is an interesting direction for future research and could extend the applicability of our method to more general video interpolation tasks. We will include this limitation in the revised version of our paper.

Q9: [Question 6] Has the performance been evaluated using SSIM or other structural similarity metrics? Including such results could offer a more complete view of visual quality.

A9: Thank you for the valuable suggestion. We have computed SSIM scores as additional evaluation metrics to complement Table 2 in the main paper. The results are presented below:

We observe that PoseFuse3D-KI achieves the highest SSIM scores across all image regions, further demonstrating its strong controllable interpolation performance and visual quality. We will include this table in the revised version of our paper for a more complete view of the interpolation quality.

Q10: [Question 7] How much GPU resources are required in the training phase of the proposed method? What about the inference time?

A10: PoseFuse3D-KI is trained using 4 NVIDIA A100 GPUs, each with 80 GB of memory. During inference, it takes 212 seconds to interpolate 25 frames at a resolution of $1024 \times 576$ on an NVIDIA A800 GPU. A more detailed discussion on runtime performance is provided in A3.

Thank you for the constructive comments. Please find the following for our response.

Q1: [Weakness 1] Although the claimed novelty mainly focus on the effectiveness of directly utilizing 3D SMPL information, the experiments in Table 1 fails to support this claim: the performance gain of VE+SE over VE/VE+DN is marginal. … Although the authors want to emphasize the contribution of direct learning from SMPL parameters instead of visualizations in Champ, the experiments does not support such claim.

A1: We respectfully disagree with the reviewer. We argue that our results support the effectiveness of our method. Specifically, in Table 1 of the main paper, our 'VE+SE' consistently outperforms the 'VE+DN' baseline across all quantitative metrics, with a notable 0.21 dB improvement on PSNR$_{mask}$, which focuses on the human foreground. More importantly, our SMPL-X encoder is explicitly designed to model occluded body parts, information typically discarded in the rendering process used in visualization. This advantage is evident in the qualitative results. For example, as shown in Figure 3 of the main paper, only our strategy `VE+SE' correctly handles the challenging occlusions, such as the occluded arm interpolation in Skateboarding ($1^{st}$ row) and Jumping ($2^{nd}$ row). Therefore, with both quantitative and qualitative results, we claim that our results demonstrate the effectiveness of our proposed SMPL-X encoder.

Q2: [Weakness 1] Besides, there is an important ablation study missing: only using SE. If only using SE is way worse than only using VE+DN, it will indicates that the proposed modules is not effective at all compared to Champ (which could be regarded as VE+DN).

A2: Thank you for your suggestion to include this ablation study. We would like to clarify that we have already included an experiment using only SE in Section 1.6 of the supplementary material. This variant, denoted as 'Non-Vis', is used to ablate the importance of visual encoding in PoseFuse3D-KI. The corresponding results are provided in Table 2 of the supplementary material.

Furthermore, we would like to emphasize that it is inappropriate to directly compare 'Non-Vis' with 'VE+DN', as our proposed method is designed as an integrated framework. While the SMPL-X encoder is responsible for preserving occluded information, our visual encoder incorporates 3D human rendering to provide direct shape and appearance cues. Therefore, a more meaningful comparison is between our method 'VE+SE' and 'VE+DN', whose results are shown in Table 1 and Figure 3 of the main paper. As discussed in A1, these comparisons demonstrate the effectiveness of our proposed SMPL-X encoder in modeling occluded human motion.

Q3: [Weakness 2] As we know 2D pose detector is generally more accurate than SMPL pose estimator. The paper relies on the SMPLer-X tool for automated 3D model fitting, supplemented by "manual verification". However, it lacks a detailed description of this verification process, such as the scope of manual checks or the final error rate. The quality and accuracy of the underlying SMPL-X annotations are critical for training a robust 3D-aware model, and any systematic errors from the automated tool could compromise the model's performance and the validity of the evaluation.

A3: As described in Section 4 of the main paper and Section 1.4 of the supplementary material, our annotation pipeline involves both 2D human keypoint annotation using Sapiens and 3D SMPL-X model fitting using SMPLer-X. To ensure annotation quality, we conduct manual verification at two stages: 1) Pre-annotation verification is applied to the outputs of the human detection stage to filter out low-quality detections before proceeding with annotation. 2) Post-annotation verification is performed on the annotated data to correct or discard erroneous fits. Thanks to the high re-projection accuracy of SMPLer-X, most fitted SMPL-X models closely align with the image human body motion evidence upon projection, which supports their use as reliable supervision in our human-centric keyframe interpolation task. Furthermore, the effectiveness of the resulting dataset is validated through both quantitative and qualitative results presented in our paper. We believe this demonstrates that the SMPL-X annotations are of sufficiently high quality for training and evaluation.

Q4: [Weakness 3] The paper creates a new dataset with 2,614 videos by sourcing videos from Pexels, but it does not discuss or compare its contribution to other recent datasets like HumanVid [a] or MiraData [b], which also use Pexels videos for video-related tasks. Both mentioned datasets have a scale larger than this submission. A discussion of why a new dataset was necessary instead of augmenting existing ones would strengthen the justification for this contribution. For example, HumanVid [a] already curated human-centric 20K videos from Pexels, will this dataset shares a large portion of videos with this submission?

A4: Thank you for the suggestion to clarify our motivation for creating a new dataset. Our dataset is specifically designed for the Controllable Human-centric Keyframe Interpolation (CHKI) task, with a focus on complex human motions, human-centric annotations, and distant keyframe inputs. These features are not directly addressed by existing datasets. As described in Section 4 of the main paper and Section 1.4 of the supplementary material, our dataset is sourced from both the SportsSlomo dataset and the Pexels website. Specifically, to ensure motion complexity and diversity, we begin with videos from the SportsSlomo dataset, which contains challenging scenarios. Then, we further expand the dataset in diversity by retrieving additional videos from Pexels using a curated list of motion-related keywords. This targeted video collection strategy provides more precise control over the dataset content, which is crucial for training and evaluation in the CHKI setting. While there may be some overlap in Pexels-sourced content with datasets such as HumanVid and MiraData, our dataset is independently curated with a distinct task focus. Unlike those datasets, our design prioritizes motion complexity, keyframe temporal sparsity, and annotation consistency specific to CHKI. We appreciate the reviewer bringing up these related datasets and will cite both HumanVid and MiraData in the revised version, along with a discussion highlighting the distinctions and motivation behind our dataset construction.

Dear Authors,

As reminded by the emails sent from NeurIPS organizers, I will update some information about my final justification here to confirm that I have participated this discussion and updated my final rating about this submission.

There are still some concerns about the small performance gain of VE+SE (the proposed method) over VE+DN (baseline method). The author also not fully justify the necessity of the proposed dataset compared to existing ones (although they are not originally designed for interpolation, it is very simple to do so) and the quality control of SMPL annotation process. Yet, this paper has provided extensive experiments and ablation studies to analyze the proposed method, which could be useful for the community to further explore in this direction. So I think it is acceptable to recommend the exposure of this paper to the community. I have increased the rating from borderline reject to be borderline accept. But I am not willing to defend acceptance for this paper against other reviewers. Instead, I hope the final recommendation of this paper could be achieved by aggregation of rating from all reviewers.

Q1: [Weakness 1] Introducing human studies would better evaluate the performance of generative interpolation models.

A1: Thank you for the valuable suggestion to include a human user study. To further evaluate the perceptual quality of our method, we conducted a user study involving 40 participants across 20 interpolation scenarios. Participants were asked to choose their preferred output among the results generated by the baseline methods and our proposed PoseFuse3D-KI. The summarized results are as follows:

Our method received a strong majority of user preferences (86.5%), consistently outperforming all baselines. These results further demonstrate the perceptual effectiveness of PoseFuse3D-KI.

Q2: [Weakness 2] I found that most cases provided by the authors target a single person, but when extending to many humans, such as a group of people (more than 5), can PoseFuse3D-KI still work? Can SMPL still estimate accurately?

A2: Thank you for the valuable question. As our method focuses on the plausible interpolation of human images with complex motion, we limit the number of humans in our training dataset. However, we have qualitatively validated the effectiveness of PoseFuse3D-KI in test cases involving more than five people. Unfortunately, due to limitations of the discussion format, we are currently unable to include the qualitative results here. These examples will be included in the supplementary materials of the revised version of the paper.

Q3: [Weakness 3] How does the performance fare when humans are not positioned at the center of the frame?

A3: We appreciate the reviewer for raising this question. Our method generalizes well to scenarios where humans are not centered in the frame, maintaining consistent and plausible interpolation performance. Such cases are included in the test set of our proposed dataset and are part of the quantitative evaluations presented in the paper. We will include visualizations of these examples in the revised version of the paper.

Q4: [Weakness 4] Is there any insight into the design of the SMPL-X Encoder?

A4: The SMPL-X encoder is introduced to supply additional spatial information, particularly in occluded regions where rendering operations result in information loss. While the rendering process transforms the 3D model into 2D images, it only preserves visible (non-occluded) information during projection. However, occluded information is crucial for modeling motion and generating plausible interpolations. Discarding it often leads to implausible results, as illustrated in Figure 3 of the main paper. To address this, the SMPL-X encoder directly processes 3D information and uses attention mechanisms to aggregate it into the image space, effectively preserving important cues that are lost during rendering.

Q5: [Weakness 5] Do you need to consider copyright when using Pexels data?

A5: Thank you for raising this important question. All Pexels content is provided under the Pexels license, which allows free use. The videos we collected are modified and used purely for research purposes, not distributed on other platforms or sold. Therefore, our usage complies fully with Pexels’ licensing terms.

Thank you for your response. Most of my concerns have been addressed.

Additionally, I have the following questions:

1. Will the CHKI-Video dataset be open-source?
2. Is the interpolation effective when there is camera motion? I believe it's important to discuss broader application scenarios and limitations in the revision.

Thank the reviewer for the valuable feedback. Please find the following for our response.

Q1: Will the CHKI-Video dataset be open-source?

A1: Thanks for the question. We believe that open-sourcing code and data is important for the research community, and we will make both our dataset and codebase open-source for future research.

Q2: Is the interpolation effective when there is camera motion? I believe it's important to discuss broader application scenarios and limitations in the revision.

A2: Thanks for pointing out the camera motion cases.

• Focusing on human-centric interpolation, our proposed method is effective for the foreground human under camera motion and can still perform plausible human interpolation. Such cases of camera motion are included in the test set of our CHKI-Video. We will add related qualitative results in the revised version of paper.
• While our proposed method focuses on plausibly interpolating foreground humans, our approach works well in many cases for the background interpolation under camera motion, a qualitative example (surfing, 3rd case in Part 1) can be observed in the supplementary video. We acknowledge that there might be artifacts under challenging conditions of extreme camera motion. Addressing this limitation is an interesting direction for future research and could extend the applicability of our method to more general video interpolation tasks. We will also include this limitation in the revised version of our paper.