Neural Polynomial Gabor Fields for Macro Motion Analysis

(cont'd)

Related work

Phase-PGF introduces a completely different motion analysis framework to tackle the macro motion analysis task. Specifically, we use phase to capture the macro motion information, differing from other works significantly.

Reproducibility

We will release code and data upon acceptance.

Details on human preference evaluation

As indicated in Appendix B and the supplementary webpage, for human evaluation, over 100 ratings are obtained for each video. As indicated, please see our supplementary webpage for the actual user study form we used.

Further applications

We demonstrate moving object discovery for video analysis. To automatically discover moving objects, we introduce a quantity, phase score, which quantifies how much a subband contributes to the final rendering. The subbands associated with the highest phase scores represent the dynamic/static objects in the scene. We discuss the automatic phase association in Appendix C.2. We show that our model can discover moving objects in a real video in Appendix D.6.

Thank you for your constructive review and insightful suggestions!

We address all your concerns and questions as below.

Addressing slight artifacts

One possible reason for the slight artifacts in motion magnification is that the spatial support is insufficient for the magnified object trajectory. As we discussed in the limitation section, this might be addressed by adding more Gabor basis which requires more computational resources, or by introducing spatially-adaptive Gabor basis in future work.

Plan for scaling to complex 3d dynamic scenes

As dicussed in the limitation section, an essentialy bottleneck in scaling our method to large-scale complex 3D dynamic scene is adding more Gabor basis and subbands, which reuiqres increased compute. An alternative solution is spatially adaptive Gabor basis, yet we remain this for future work.

Further experimental evaluation:

We have included additional supporting evaluations in the revised manuscript appendix D, as detailed below:

Additional examples. We include 6 more examples of more complex macro motions:

• Non-rigid motion: We include an Internet video, Jelly, where a jelly is wobbling on a seat. This example demonstrates non-rigid non-uniform shearing motion.
• Interaction between objects: We include an example of two balls colliding with each other and then changing moving directions.
• Multi-object motion: We include a scene with three balls vibrating with different frequencies and magnitudes.
• Non-periodic motion: We synthesize three videos, Damping, Bouncing, and Projectile, contain different non-periodic motions. In Damping and Bouncing, we simulate a damping motion of a ball. In projectile, we simulate a ball being projected outward, subjecting to gravity.

We show further descriptions, results, and discussion in Appendix D.1. In summary, our model is able to represent and interpret the more diverse motions in these additional examples. Video results are also available at our supplementary webpage.

Quantitative results. We also include two quantitative metrics to evaluate all the examples, including:

• Normalized Cross-Correlation (NCC) to measure for phase correlation and phase interpretability. The NCC is a standard metric in time-series analysis to measure the similarity of two 1-D temporal signals, which meet our need in measuring how the generated phase align with the motion is a dynamic scene. In Table 2, we show that our method surpasses the baseline significantly in this metric.
• Frechet Inception Distance (FID) to measure the quality of the phase-manipulated videos. In particular, we compute FID between the manipulated video and the original video. Since our motion manipulation only changes the motions presented in the scene, a lower FID then suggests less artifacts and higher visual fidelity. In Table 3, we show that the result produced by our method has better visual quality.

We include more disucssion on quantitative metrics in Appendix D.2.

Phase space analysis and moving object discovery. To provide more analysis, we also add an analysis on phase space. In particular, we introduce a quantity, phase score, to analyze and automatically select phases that correspond to significant motions in the dynamic scene. We discuss phase space in Appendix C.2. We show that our model can discover moving objects in a real video in Appendix D.6.

Ablation study. In Appendix D.3, we include ablation study on decoder and adversarial training. In Figure 12 of the revised manuscript, we show that the introduced components contribute significantly to our model's high visual fidelity.

Comparison to point tracking-based method. We include one additional point tracking-based baseline method. Specifically, we extract points using PIPs++ [1], a state-of-the-art point tracking method. To analysis motion, we use PCA to reduce its dimensions. We show a qualitative result in Figure 13. We observe that the motion space from point tracking is not structured or interpretable, and thus it does not support intrepreting or manipulating motions in a meaningful way. This suggests that the unstructured point-space representation does not easily support abstracting a low-dimensional manipulatable motion representation.

We believe these additional experiments can better support our method.

Network architecture:

In Appendix C.1, We include all our additional technical details, including

• Phase Generator
• Spatial Polynomial Gabor Fields
• Basis Modulation and Phase-PGF
• Neural Rendering
• Feature Decoding and Rendering
• Adversarial Training

We also added Table 4,5,6,7 that further specify the hyper-parameters and network architectures we used.

Thank you for your constructive review and insightful suggestions!

We address all your concerns and questions as below.

Quantitative metrics

Following your suggestion, we use quantitative metrics to evaluate phase interpretability and video quality. In particular, we use:

• Normalized Cross-Correlation (NCC) to measure for phase correlation and phase interpretability. The NCC is a standard metric in time-series analysis to measure the similarity of two 1-D temporal signals, which meet our need in measuring how the generated phase align with the motion is a dynamic scene. In Table 2, we show that our method surpasses the baseline significantly in this metric.
• Frechet Inception Distance (FID) to measure the quality of the phase-manipulated videos. In particular, we compute FID between the manipulated video and the original video. Since our motion manipulation only changes the motions presented in the scene, a lower FID then suggests less artifacts and higher visual fidelity. In Table 3, we show that the result produced by our method has better visual quality.

We include more disucssion on quantitative metrics in Appendix D.2.

More complex motions

We include 6 more examples of more complex macro motions:

• Non-rigid motion: We include an Internet video, Jelly, where a jelly is wobbling on a seat. This example demonstrates non-rigid non-uniform shearing motion.
• Interaction between objects: We include an example of two balls colliding with each other and then changing moving directions.
• Multi-object motion: We include a scene with three balls vibrating with different frequencies and magnitudes.
• Non-periodic motion: We synthesize three videos, Damping, Bouncing, and Projectile, contain different non-periodic motions. In Damping and Bouncing, we simulate a damping motion of a ball. In projectile, we simulate a ball being projected outward, subjecting to gravity.

We show further descriptions, results, and discussion in Appendix D.1. In summary, our model is able to represent and interpret the more diverse motions in these additional examples. Video results are also available at our supplementary webpage.

Separate more than two motions

In the Multi-object motion example above, we demonstrate separation of three balls.

Automatic moving object discovery

To automatically discover moving objects, we introduce a quantity, phase score, which quantifies how much a subband contributes to the final rendering. The subbands associated with the highest phase scores represent the dynamic/static objects in the scene. We discuss the automatic phase association in Appendix C.2. We show that our model can discover moving objects in a real video in Appendix D.6.

Clarification on representing motion direction

We clarify that our method does represent motion direction by the orientation of the Gabor basis. In particular, we do decompose motion directions into $x$ and $y$. This can be seen intuitively from the projectile example in Figure 11.

Clarification on extracting motion representation

We set small timesteps to sample the implicit function values and discretize the continuous implicit function to a discrete phase sequence. Specifically, we use a timestep $2/N$ in a normalized scale, where $N$ is the number of input frames.

Evaluation metrics

To allow automatic quantitative measures, we additionally include the following evaluation metrics:

• Normalized Cross-Correlation (NCC) to measure for phase correlation and phase interpretability. The NCC is a standard metric in time-series analysis to measure the similarity of two 1-D temporal signals, which meet our need in measuring how the generated phase align with the motion is a dynamic scene. In Table 2, we show that our method surpasses the baseline significantly in this metric.
• Frechet Inception Distance (FID) to measure the quality of the phase-manipulated videos. In particular, we compute FID between the manipulated video and the original video. Since our motion manipulation only changes the motions presented in the scene, a lower FID then suggests less artifacts and higher visual fidelity. In Table 3, we show that the result produced by our method has better visual quality.

We include more disucssion on quantitative metrics in Appendix D.2.

More extenisve quantitative evaluation

As mentioned above, we add quantitative measurements and additional examples in Appendix D of the revised manuscript. We believe these additional examples and evaluations add to the extensiveness of our experiments.

Comparison to point tracking

We follow your suggestion to compare with a point tracking-based method. In particular, we extract points using PIPs++ [1], a state-of-the-art point tracking method. To analyze motion, we use PCA to reduce its dimensions. We show a qualitative result in Figure 13. We observe that the motion space from point tracking is not structured nor interpretable, and thus it does not support intrepreting or manipulating motions in a meaningful way. This suggests that the unstructured point-space representation does not easily support abstracting a low-dimensional manipulatable motion representation.

The working range of the proposed method

Besides the five additional examples we have shown above, we also add failure cases to demonstrate the working range. In particular, we find that our approach does not work with motions of shapeless objects such as fire (Figure 15), because it violates our assumption on time-invariant basis functions to represent the video appearances. Another example is chaotic unpredictable motions. We show a hair example in Figure 15 where the hair motion is unpredictable and it might not allow a frequency-based low-dimensional representation.

Amplitude of phases

We clarify that the motion amplitudes can be captured by the generated phase amplitudes within a single motion. This can be observed from the Damping example in Figure 7.

Yet, we note that the phase amplitudes are not comparable among different motions. For a scene with multiple motions, such as the two bouncing balls, although each ball's motion amplitude can still be captured, the relative phase amplitudes of the two balls do not reflect their relative motion amplitudes, because each phase signal is generated from an implicit function (phase generator), and an implicit function can have an arbitrary output scale.

[1] Zheng et al., PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking, ICCV, 2023

Thank you for your constructive review and insightful suggestions!

We address all your concerns and questions as below.

Exposition on phase space and phase generator

We add detailed exposition on the formulation and architecture of the Phase Generator in Appendix C.1 of the revised manuscript, and detailed discussion on phase space in C.2. In short, we have $K$ phase sequences, i.e., our phase space is $K$-dimensional, where $K$ denotes the number of subbands. Each of the $K$ generated phase sequences modulates a subband (we have also added details of the modulation in Appendix C.1). All $K$ subbands add up to collectively represent the dynamic scene.

How to initialize the number of phases

We add a detailed discussion in Appendix C.2 of the revised manuscript. The main idea is that, $K$ can be set either according to prior knowledge of how many motion components there are in the scene, or to a large number. When we set a large $K$, our model can automatically associate appropriate subbands (i.e., phases) to the motion components in the scene, thus automatically discovering moving objects without manually specifying the number of phases. Then, all other redundent phases have little or no effects on representing the dynamic contents.

To do this, we introduce a phase score metric to automatically discover the associated subbands. The phase score quantifies how much a subband contributes to the final rendering. The subbands associated with the highest phase scores represent the dynamic objects in the scene. We show that our model can discover moving objects in a real video in Appendix D.6.

Clarification on manipulating phases

We clarify that several of our experiments on macro motion analysis indeed manipulate phases to manipulate motions. In particular:

• Motion separation (Figure 4): We used band-limit filters on the generated phase sequence, and then we isolated the motion of interest (the motion of the blue ball) according to the frequency. In this way, we manipulated (mollified) the motion of the blue ball without affecting the motion of the red ball.
• Motion intensity adjustment (Figure 5): We introduced a phase augmentation technique in Section 3.2., Equation (5) and (6), to magnify the motion in the video.
• Motion smoothing (Figure 6): We show that by removing high-frequency bands of the generated phase sequence, we can smooth out the high-frequency motions in the macro motion.

We also include video results of phase-based manipulation in our supplementary webpage.

Details on implementation

In Appendix C.1, We include all our additional technical details, including

• Phase Generator
• Spatial Polynomial Gabor Fields
• Basis Modulation and Phase-PGF
• Neural Rendering
• Feature Decoding and Rendering
• Adversarial Training

We also added Table 4,5,6,7 that further specify the hyper-parameters and network architectures we used.

Adapting to diverse motion complexities

We include 6 more examples of more complex macro motions:

• Non-rigid motion: We include an Internet video, Jelly, where a jelly is wobbling on a seat. This example demonstrates non-rigid non-uniform shearing motion.
• Interaction between objects: We include an example of two balls colliding with each other and then changing moving directions.
• Multi-object motion: We include a scene with three balls vibrating with different frequencies and magnitudes.
• Non-periodic motion: We synthesize three videos, Damping, Bouncing, and Projectile, contain different non-periodic motions. In Damping and Bouncing, we simulate a damping motion of a ball. In projectile, we simulate a ball being projected outward, subjecting to gravity.

We show further descriptions, results, and discussion in Appendix D.1. In summary, our model is able to represent and interpret the more diverse motions in these additional examples. Video results are also available at our supplementary webpage.

Ablation study

In Appendix D.3, we include ablation study on decoder and adversarial training. In Figure 12 of the revised manuscript, we show that the introduced components contribute significantly to our model's high visual fidelity.

(cont'd in the next response)

Dear Reviewer,

Thank you again for your insightful review and constructive feedback on our submission. We hope our responses have effectively addressed all your concerns. Please be informed that the author-reviewer discussion period ends today. Should there be any unresolved issues, we would appreciate your prompt feedback. If your concerns have been addressed, please kindly consider updating your score. Thank you!

Best, Authors

Thank you for your constructive review and insightful suggestions!

We address all your concerns and questions as below.

Additional supporting experiments

To include additional supporting experiments, we extensively include additional contents.

Additional examples. We include 6 more examples of more complex macro motions:

• Non-rigid motion: We include an Internet video, Jelly, where a jelly is wobbling on a seat. This example demonstrates non-rigid non-uniform shearing motion.
• Interaction between objects: We include an example of two balls colliding with each other and then changing moving directions.
• Multi-object motion: We include a scene with three balls vibrating with different frequencies and magnitudes.
• Non-periodic motion: We synthesize three videos, Damping, Bouncing, and Projectile, contain different non-periodic motions. In Damping and Bouncing, we simulate a damping motion of a ball. In projectile, we simulate a ball being projected outward, subjecting to gravity.

We show further descriptions, results, and discussion in Appendix D.1. In summary, our model is able to represent and interpret the more diverse motions in these additional examples. Video results are also available at our supplementary webpage.

Quantitative results. We also include two quantitative metrics to evaluate all the examples, including:

• Normalized Cross-Correlation (NCC) to measure for phase correlation and phase interpretability. The NCC is a standard metric in time-series analysis to measure the similarity of two 1-D temporal signals, which meet our need in measuring how the generated phase align with the motion is a dynamic scene. In Table 2, we show that our method surpasses the baseline significantly in this metric.
• Frechet Inception Distance (FID) to measure the quality of the phase-manipulated videos. In particular, we compute FID between the manipulated video and the original video. Since our motion manipulation only changes the motions presented in the scene, a lower FID then suggests less artifacts and higher visual fidelity. In Table 3, we show that the result produced by our method has better visual quality.

We include more disucssion on quantitative metrics in Appendix D.2.

Phase space analysis and moving object discovery. To provide more analysis, we add an analysis on phase space in Appendix C.2. Following the analysis on phase space, we introduce a phase score metric to automatically discover the associated subbands. The phase score quantifies how much a subband contributes to the final rendering. The subbands associated with the highest phase scores represent the dynamic/static components in the scene. We show that our model can discover moving objects in a real video in Appendix D.6.

Ablation study. In Appendix D.3, we include ablation study on decoder and adversarial training. In Figure 12 of the revised manuscript, we show that the introduced components contribute significantly to our model's high visual fidelity.

Comparison to point tracking-based method. We include one additional point tracking-based baseline method. Specifically, we extract points using PIPs++ [1], a state-of-the-art point tracking method. To analysis motion, we use PCA to reduce its dimensions. We show a qualitative result in Figure 13. We observe that the motion space from point tracking is not structured or interpretable, and thus it does not support intrepreting or manipulating motions in a meaningful way. This suggests that the unstructured point-space representation does not easily support abstracting a low-dimensional manipulatable motion representation.

We believe these additional experiments can better support our method.

More video results on webpage

We add video visualizations for video results for motion analysis, motion smoothing, motion magnification, motion separation, and failure cases in our supplementary webpage.

Quality for video results

As summarized above, we add a quantitative metric FID for video quality and show results in Table 8. We achieve higher quality compared to baseline methods.

More complex motions

As summarized above, we add several examples in more complex motions. Among them, the Jelly example demonstrates not only more complex motion that includes non-rigid shearing, but also more complex object, i.e., a colorful soft body. We show that our model can reconstruct well the video and generate an interpretable motion representation including a dominant phase in Figure 8.

Systematic evaluation on phases

As summarized above, we use a metric, Normalized Cross Correlation, to measure phase correctness. Please refer to appendix D.2 for more details.

[1] Zheng et al., PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking, ICCV, 2023