ReMatching Dynamic Reconstruction Flow

We thank the reviewer for a detailed review. Below we address the main comments and questions expressed in this review.

W1: “I am quite confused with the definition used to derive Eq. (9)”.

A: Thank you for taking the time to spot this. Indeed it should have been $V=\mathbb{R}^{n\times d}$. We have corrected this typo in the latest revision.

W2: “how Eq. (9) is derived based on the above definition. Can authors provide details?”.

A: Thank you for this question. We agree with the reviewer on the value of clarifying this in the text. As a result, we added these details to section 8.1.2 in the appendix, included in the latest revision. The formal way to utilize discrete samples of trajectories ${\gamma_t^i}$ within the continuity equation (4) is by considering $\psi_t(x) = \sum_{i=1}^n \delta(x - \gamma_t^i)$, where $\delta(\cdot - a)$ is the dirac-delta generalized function concentrated around $a$. An alternative approach is to consider $\gamma_t^i$ as samples of integral curves of the (unknown) reconstruction flow, and utilize equation (3) to derive equation (9). However, due to the generality of the continuity equation (covering the discrete settings as a specific case), in the paper we followed the former approach suggesting utilizing equation (4).

W3: Inadequate visualization.”I am quite curious to know 1) how the reconstructed flow as well as velocity fields look like; 2) how the prior set look like; and 3) the difference between the prior and the trained flow.”

A: Thank you for this insightful question. We agree with the reviewer on the importance of this visualization. We will include it in the next revision, to be uploaded before the end of the discussion period.

W4: Can the authors clarify the efficiency of the flow-matching approach? “If I understand correctly, in Algorithm 1, the step in L189 will be solved analytically based on those priors specified Sec. 4, which should be quick. Further, the computation of in L190 should also be quick due to those priors”.

A: The reviewer’s description is accurate. To support this analysis empirically, we have included timing comparisons of the ReMatching loss in the appendix, included in the latest revision.

W5: Missed reference

A: Thank you for spotting this. We have added this reference to the latest revision.

Q1: Can authors clarify how to make HyperNeRF a multi-view dynamic view synthesis benchmark? If the authors indeed carry out monocular dynamic view synthesis, please correct L100 as the "multi-view images" statement is not correct.

A: Thank you for spotting this. We have evaluated HyperNeRF under monocular dynamic view synthesis settings. We clarified the settings of which our method operates in section 3, now included in the latest revision.

Q2: The authors state that there is a reference function to be pushed forward. Concretely, does $\psi_0$ refer to the initialized Gaussian mean $\mu^i$ in Eq. (29)?

A: Yes, in this instantiation, the $\mu^i$ serve as the references to be pushed forward. As a side remark, we note that one advantage of the matching formulation is its independence from the reference function $\psi_0$. Instead, it relies solely on the derivatives at intermediate times, allowing control over the flow deformation without the necessity of pushing forward the reference function.

Q3: The authors state that they use forward-mode auto differentiation. Can authors clarify the reason of not using the traditional backpropagation, i.e., reverse-mode?

A: To clarify, we have utilized forward-mode autodiff only to compute the partial derivatives of $\psi_t$, which are necessary for calculating the ReMatching loss. Once the ReMatching loss is evaluated and incorporated into the total loss, the gradient of the total loss with respect to the model parameters is computed using standard backward-mode autodiff. This clarification has been added to the text in the latest revision (see lines 905-912). The rationale for employing forward-mode autodiff for$\psi_t$ partial derivatives is that the input dimension is relatively small—either 1 (for time) or $d$ (for spatial coordinates)—compared to the output dimension, which can be $n \times d$ or, in the case of images, $H \times W$. In such cases, forward-mode autodiff is both computationally and memory-efficient compared to backward-mode autodiff.

We sincerely thank the reviewer for their prompt response and valuable suggestions.

In line with the recommendations, the revised paper now includes visualizations of the learned reconstruction flow. A key challenge in generating these visualizations is that our framework does not explicitly model the reconstruction flow, nor it is accessible with existing datasets. To address this, we created a synthetic training dataset with dynamics governed by a predefined flow. These experiments validate the effectiveness of the ReMatching objective in recovering the underlying flow. In addition, it emphasizes the significance of the rematching procedure, as illustrated in Figure 13, which shows that the matched velocity fields of solutions not trained with the rematching loss deviate significantly from the ground-truth generating velocity fields.

Furthermore, we have addressed the reviewer’s feedback on improving readability by adding a summary paragraph to the introduction, highlighting our key contributions more clearly.

W4: The segmentation compared with SAM is not proper and confusing. Since SAM segments objects according to semantics but this paper segments according to motions, it’s not proper to punish SAM’s over-segmentation.

A: Our approach is not intended for semantic segmentation, nor do we make any claims in this regard. The qualitative comparison with SAM was conducted to highlight the non-triviality of the scene decomposition task. This comparison shows that, within the context of dynamic reconstruction, leveraging a foundation model such as SAM to generate scene decompositions that align with the underlying scene geometry may not always be the most effective approach. Furthermore, including this comparison also adds value due to the fact that utilizing SAM for 3D-related tasks has gained popularity in recent works, such as [1] and [2].

[1]: Zhou, Yuchen, et al. "Point-SAM: Promptable 3D Segmentation Model for Point Clouds." arXiv preprint arXiv:2406.17741 (2024).

[2]: Cen, Jiazhong, et al. "Segment Anything in 3D with Radiance Fields." arXiv preprint arXiv:2304.12308 (2023).

W5: Another question is about the bouncing ball example shown in the paper. As I know this scene includes some moving shadow on the ground, so the motion should not segment the shadow part as the same as the static ground. How is the segmentation applied?

A: We understand the reviewer concerns, but it seems that there might be some confusion. We claim that shadow effects are not part of the geometry, but rather arise as a consequence of the specific geometric configuration and lighting conditions. To illustrate this, consider the reanimation of the bouncing balls scene: shadows are not independent entities capable of arbitrary deformation. The ability of our model to accommodate shadow effects into the static part (e.g., the floor in the bouncing balls scene) represents a significant success for our approach, highlighting its capability to improve the disentanglement of geometry from appearance.

W6: The performances are only comparable, it’s hard to tell whether the priors are really helpful or not.

A: Our approach effectively reduces unrealistic distortions, such as issues in the dynamic movements of fingers in the jumping jacks scene and structural artifacts in moving parts like the leg in the T-Rex scene, the knife in the lemon scene, and the slicer in the banana scene. These points are discussed in lines 415-417 and 463-466 of the main text. It is worth noting that our improvements primarily enhance image areas involving motion, which can skew the results based on the extent of movement presented in an image within each scene. To strengthen these claims, the latest revision now includes additional qualitative evidence in the supplementary material. This material features novel-view video reconstruction results that compare our model with baseline approaches, providing a clearer illustration of the advantages highlighted in the main text.

W7: There is a lack of an illustrative figure to show the main architecture of the proposed method, so the reader can only know how the idea is working till the end of the paper.

A: Thank you for this suggestion. We added such a visualization in the implementation details in the appendix section, included in the latest revision. We note that our framework contribution is not in proposing an architecture, in fact applicable to several types of model functions.

W8: There are too many equations or derivations in the main text. Although they are clear and I appreciate the details, too many derivations in the main text make the paper lose its main focus.

A: We revised the method section to make it as concise as possible, which led to some simplifications in the mathematical descriptions of the derivations. However, we could not completely eliminate any of them. We would be happy to consider more specific examples if available.

W9: There are inconsistent baseline names in text and tables, for example, 3D Geometry-aware Deformable Gaussians is called DAG in main text but GA3D in the table.

A: Thank you for taking time to spot this. We fixed these inconsistencies, incorporated into the latest revision.

We thank the reviewer for a detailed review. Before addressing the specific weaknesses and questions raised, we would like to clarify and respond to some points made in the reviewer’s summary.

S1: “This paper aims to solve dynamic reconstruction task. It proposes a new loss called ReMatching loss, in order to inject some physical priors into the learned deformation field”

A: We do not claim to aim at solving the dynamic reconstruction task. Instead, acknowledging the inherently ill-posed nature of this problem—which necessitates the integration of priors—we propose a novel method for incorporating such priors (see lines 26–33). The goal of the ReMatching framework is not to “inject physical priors”, but rather to suggest a novel scheme to leverage priors to improve generalization to unseen frames, without compromising solutions’ fidelity levels (see lines 34-35). This is done by suggesting an optimization objective mimicking the APM (see lines 60-63) .

S2: “ The paper builds the model based on the deformable-3d-gaussian paper, and tests the performance for three of the five categories”

A: The suggested framework is not formulated with respect to a specific instantiation of a time-dependent function (see lines 38-39, 100-107). This is also supported empirically by applying our framework on time-dependent image pixel values (see lines 39, 510-515). The first two categories of priors, while not directly matching any of synthetic scenes due to their simplicity, are effectively integrated and tested through the combined classes.

S3: “showing comparable performance to baselines.”

A: Our approach has advantages in reducing unrealistic distortions, such as visible issues in the dynamic movements of fingers in the jumping jacks scene and structural artifacts in moving parts like the leg in the T-Rex scene, the knife in the lemon scene, and the slicer in the banana scene. These points are discussed in lines 415-417 and 463-466 of the main text. Notably, our improvements primarily enhance image areas involving motion, which can skew the comparison based on the extent of movement presented within each scene. Nevertheless, our approach ranks first in 11 out of 13 scenes across at least two metrics. To strengthen these claims, the latest revision now includes additional qualitative evidence in the supplementary material, featuring novel-view video reconstruction results comparing our model with baseline approaches.

W1: In synthetic experiments, type IV and type V priors are selected for each scene. They should be thoroughly tested respectively to show the applicability under different scenarios.

A: We respectfully disagree with the value of such an evaluation. Priors should be thoughtfully designed and applied in alignment with the specific dynamics exhibited by the scene. For instance, in scenes without static components, using the class $P_V$ is less suitable (e.g., the jumping jack scene). Similarly, the classes $P_I$ and $P_{II}$ , while not directly matching any of D-NeRF synthetic scenes due to their simplicity, are effectively integrated through the combined classes $P_{IV}$ and $P_{V}$.

W2: In real-world experiments, it is not clear which type of priors are used. And also, all of the priors should be tested to show the abilities.

A: Thank you for this suggestion. We added an evaluation for both of the relevant priors $P_{III}$ and $P_{IV}$, incorporated in table 2 in the latest revision.

W3: The velocity branches in the model only serve as a constraint part, but not used directly. The paper should demonstrate whether the learned velocity is meaningful or not. For example, using the velocity to advect the scene.

A: We respectfully disagree with the reviewer’s suggestion. Our framework does not explicitly model a velocity branch as proposed. Employing a velocity field from the prior class to directly advect the scene is inconsistent with the principles of the ReMatching framework. The ReMatching loss is designed to optimize the dynamic reconstruction model, which operates via direct single-step evaluations only (see lines 142–143), to align as closely as possible with the prior class without compromising reconstruction fidelity. Importantly, elements from the prior class may inherently conflict with reconstruction quality (see lines 54–62 and 149–182 in the text for further explanation).

We sincerely thank the reviewer once again for their detailed review.

We have uploaded a second revision that includes the GA3D evaluation for the HyperNeRF dataset in Table 2. This additional result does not significantly alter the quantitative evaluation of this benchmark, with our approach still ranked first in at least two metrics across all five scenes.

Furthermore, Section 8.4 now includes qualitative visualizations of the ReMatching framework’s ability to recover the underlying reconstruction flow. We hope this addresses the reviewer’s comment regarding the need for clearer visualizations. We are happy to provide further clarification if necessary.

We thank the reviewer for a detailed review. Below we address the main comments and questions expressed in this review.

W1: It's not so clear what specific problem the paper is trying to tackle and overcome what challenges in the literature.

A: We divide our answer into two parts.

“what specific problem the paper is trying to tackle”

The paper aims to improve the accuracy of dynamic reconstruction models, a long-standing open research challenge. This difficulty is exemplified in the qualitative results of existing baselines showcased in the paper, which highlight their limitations in capturing realistic dynamics. The core challenge arises from the ill-posed nature of the task: dynamic reconstruction relies on discrete (sparse) samples of a continuous phenomenon (e.g., objects moving continuously over time). This inherent sparsity makes it essential to incorporate prior knowledge to guide any reconstruction model. This answer can be found in the text in lines 26-34.

“overcome what challenges in the literature”

The Ill-posedness of the reconstruction problem necessitates the use of prior knowledge, which is inherent to any algorithm addressing this problem. However, since in many cases priors do not match exactly, enforcing prior assumptions for many existing methods often compromises fidelity to the input data. Our work tackles this issue not by proposing new types of priors—already abundant in the literature—but by introducing the ReMatching loss, which optimizes for solutions that remain as close as possible to the desired prior class, rather than strictly enforcing membership in the prior class. This approach minimizes the tradeoff, preserving fidelity while benefiting from the priors.

Additionally, we address two more challenges:

Generality across models: Our solution is designed to be independent of any specific reconstruction model. This generality allows the framework to unify diverse components, such as geometry representations and image rendering functions, under a common approach. For instance, by extending its applicability to rendered images, our framework can be incorporated to the training of any differentiable dynamic reconstruction model, making future advancements in this framework broadly relevant to many models.

Flexibility in prior design: We enable users of the framework to mix prior classes to tailor the method to their specific instance of the reconstruction problem, enhancing its adaptability across different scenarios.

This answer can be found in the text in lines 34-42..

W2: “Killing application? The method theoretically sounds good, but it seems not so clear what problem the method is mainly targeting… Experiments (Table 1 and 2, Figure 2) show the benefit of the method for better rendering quality, but are there any major challenges that the method targets to solve?”

A: Models enabling photorealistic renderings of novel views are pivotal in fields like the film industry, video gaming, and AR/VR. Their utility extends beyond entertainment also to scientific fields like biology, physics, and geography, where accurate visualization of dynamic phenomena is crucial (see [1] for further discussion). Moreover, dynamic reconstruction models play an essential role in downstream applications such as scene editing and scene reanimation. For example, reanimation is possible by utilizing the model’s learned geometry to control novel, plausible movements. Improving dynamic reconstruction models, as our work aims to do, has the potential to significantly benefit such downstream applications as well. Our evaluation on novel-view rendering provides a robust proxy for assessing the quality a model achieves in learning meaningful geometry and appearance representations of dynamic scenes. Thus, the improvements in novel-view rendering of our approach hold promise for improving the quality of downstream applications as well—which we identify as an important future research direction.

[1]: Yunus, Raza, et al. "Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes." Computer Graphics Forum. 2024.

Thank you for reviewing our rebuttal and updating the score!

Thank you,

The Authors

We thank the reviewer for a detailed review. Below we address the main comments and questions expressed in this review.

W1: no ablation study is presented to isolate the effects of the loss function parameters.

A: Thank you for this suggestion. We note that some of the effects of the ReMatching loss are evaluated in the paper through a comparison with D3G (see Tables 1 and 2), as D3G uses the same rendering model and training protocol, making the ReMatching loss the only difference. Additionally, the fact that we used a consistent hyperparameter value of $\lambda = 0.001$ (see line 386) across all experiments suggests a positive indicator of the ReMatching loss’s robustness.

However, we agree with the reviewer on the importance of a more comprehensive ablation study, testing a range of values for the ReMatching loss weight $\lambda$ and, in cases involving the adaptive prior class as in $\mathcal{P}_{\mathrm{IV}}$, the maximum number of possible pieces $k$. To address this, we included such an analysis in the appendix of the latest revision. The results indicate that:

i) The loss weight $\lambda$ is robust within the range $[5\mathrm{e}{-4} , 5\mathrm{e}{-3}]$, consistently improving results, while $\lambda \lt 5\mathrm{e}{-5}$ aligns with the baseline. Larger values, $\lambda \gt 1\mathrm{e}{-2}$, may compete with the reconstruction loss, leading to suboptimal solutions.

ii) The ablation study for the parameter $k$ demonstrated stable performance across a relatively wide range of choices, offering flexibility in selecting $k$ based on leveraging prior knowledge about the expected number of moving parts in the scene. This, in our view, makes $k$ a relatively user-friendly hyperparameter to configure for effective use of the loss.

W2: A new loss function raises questions about its impact on training stability and convergence speed. Additional insights such as runtime comparisons, convergence plots, or error analysis across iterations.

A: Thank you for this suggestion. We agree with the reviewer on the value of such analysis, which we have now included in the appendix of the latest revision. Regarding runtime comparisons, the results validate our claims about the additional computational cost (see lines 214-215) compared to the baseline. For the convergence analysis, the results indicate that the ReMatching loss exhibits similar qualitative behaviour to the baseline reconstruction loss.

W3: The paper asserts that the fact that the method is simulation-free helps avoid potential numerical instabilities. However, it's unclear if this claim has been experimentally validated or remains a theoretical assumption.

A: We agree with the reviewer that our previous statement on the limitations of simulation needed refinement. Indeed, a more accurate characterization is that the error in flow simulation correlates with the computational budget allocated to the simulation steps (i.e., in the idealized case of an infinite computational budget, simulation error would be eliminated). Thus, a more precise claim is that our approach, being simulation-free, fully avoids the need for potentially costly simulation procedures. This clarification has been incorporated into the latest revision.