Motion PointNet: Solving Dynamic Capture in Point Cloud Video Human Action

We thank the reviewer for recognizing the novelty and effectiveness of using PDEs to solve point cloud video problems and affirming the efficiency of our method.

Response to [Comment 1]: The core component in our PDEs-solving module is the spectral method which we detailed in Section 3.2 (PDEs-solving Module, Eq.10-12). We would like to emphasize several points to respond to the reviewer's question more clearly.

1. Superiority of spectral method: In implementing our idea, which formulates the dynamic modeling as a PDEs-solving problem, there are several options for the design of our PDEs-solving module. For instance, a pure Transformer method [1]. As discussed in previous work [2, 3], such transformer-based design attempts to learn the operator as a whole to approximate input-output mappings. However, dealing with input-output mappings in high-dimensional space can be challenging for a simplistic deep model to cover adequately while still maintaining network efficiency. To tackle the complex mappings problem, the spectral methods that we use decompose complex nonlinear mappings into multiple basis operators (Eq.10), which also holds the universal approximation capacity with theoretical guarantees.

2. Ablation results: Nevertheless, the strong perceptivity inherent in the attention mechanism cannot be overlooked. We design our PDEs-solving module base on the combination of both the Transformer model and the spectral method. We conducted ablation experiments in Section 4.3 (Tab. 8) to make a comparison with the regular Transformer. After removing the spectral method, our PDEs-solving module can be viewed as a pure Transformer model with an encoder-decoder structure. Results in Tab. 8 highlight a substantial decline in performance upon removing the spectral method, indicating its critical importance in our model.

Due to page limitations in the main text, we will augment our discussion in the supplementary material.

Response to [Comment 2]: We appreciate the reviewer's suggestion for a more comprehensive elaboration on the PDEs.

Firstly, traditional methods relying solely on global supervision from the action labels lack temporal guidance, potentially leading to an inadequate representation of dynamics in the point cloud video action recognition. To address this limitation, we decouple the spatial-temporal feature and build a PDEs problem by a directive temporal-to-spatial mapping. Together with the contrastive loss (Eq. 15) in the pretrain stage, our PDEs-solving module refines the focus of dynamic capture to a distinct objective. By doing so, we expect to provide temporal guidance.

Another insight is that PDEs can directly incorporate the feature variations over time as a variable in the modeling process. This process is visualized as Eq. 10-12 in our proposal, where we use multiple basis operators to optimize the complex temporal-to-spatial mapping. This allows a more accurate and nuanced representation of how point cloud features evolve over time compared to previous temporal modeling approaches, which is crucial for recognizing and interpreting human actions accurately.

Due to page limitations in the main text, we will expand on our methodology in the supplementary material to offer a clearer explanation.

Response to [Questions 1]: We appreciate the reviewer's suggestion regarding the potential use of our PDEs-solving module in traditional video understanding. While theoretically feasible, given that both point cloud videos and traditional videos share spatial-temporal characteristics, adapting this module for structured pixel-based video data poses significant challenges. Unlike point clouds, traditional videos represent data in a highly structured, grid-like pixel format, which poses different demands on data processing and feature extraction. Adapting the PDEs-solving approach, initially tailored for the unstructured nature of point clouds, to this structured format would require significant modifications. These might include redefining the way spatial relationships are perceived and processed within the PDE framework. Additionally, the encoding of temporal information in pixel-based videos is substantially different from point clouds, necessitating a reevaluation of how temporal dynamics are captured and integrated using PDEs in this new context. Given these challenges, exploring the application of our PDEs-solving module to traditional video understanding represents an intriguing and valuable direction for our future research.

Reference

[1] Liu, Xinliang, et al. "Ht-net: Hierarchical transformer based operator learning model for multiscale pdes." arXiv preprint arXiv:2210.10890 (2022).

[2] Wu, Haixu, et al. "Solving High-Dimensional PDEs with Latent Spectral Models." International Conference on Machine Learning (2023).

[3] Karniadakis, George Em, et al. "Physics-informed machine learning." Nature Reviews Physics 3.6 (2021): 422-440.

We are grateful for the insightful feedback provided by the reviewer and would like to address the concerns raised:

Response to [Comment 1]: Thanks for this insightful comment. The contrastive objective is indeed the sole loss during pre-training, but it does not rely on random features. Instead, it utilizes the inherent spatial and temporal coherence within the point cloud video data itself. Our contrastive loss encourages the model to learn a representation that captures the underlying structure of the point cloud sequence, which, while initially random, quickly becomes structured as the model begins to learn the spatial-temporal dynamics inherent in the data. This aligns with the principles of self-supervised learning, where the data provides its own supervision.

Response to [Comment 2]: We appreciate the reviewer's insightful view of using pure MLP or transformer-like networks for the PDEs-solving module design. Indeed, it is feasible to remove the spectral method in our proposal. However, this will harm the performance in a great extent (see Section 4.3, Tab. 8). We emphasize that the spectral method with multiple basis operators (Eq.10) holds a stronger universal approximation capacity with theoretical guarantees than those rough deep models attempt to learn the operator as a whole. We would like to claim that the superiority of the design of our PDEs-solving module is supported by both theoretical and experimental analysis. We explain the motivation for this design in our response to [Reviewer RMAg 's Comment 1 & 2]. Please check for more details.

Response to [Comment 3 & 4]: Thanks for these two insightful comments. The two-stage training process is designed to first allow the network to learn robust spatio-temporal features from the encoder and then to refine this representation using the classification head in the second stage. In this way, we can keep our encoder as lightweight as possible while still enforcing its strong learning ability. However, we acknowledge the potential benefits of exploring different training settings and concur with the reviewer's point. We have reported the results of training on the encoder only in Section 4, Tab. 7. We also add the two baseline comparisons below. This will demonstrate the effectiveness of our proposed method beyond the complexity of the architecture and training procedure. The experiments are conducted on the MSR-Action3D Dataset with 24 frames. Due to page limitations in the main text, we will augment the experimental results and discuss more in the supplementary material.

Here the pretrain + finetune is the setting used in our paper, setting 1 indicates the suggested baseline 1, and setting 2 indicates the suggested baseline 2.

Response to [Question 1]: The high-response points are selected based on the magnitude of the feature response, with a threshold set to emphasize those points that most significantly contribute to the action recognition task. The binary representation was chosen for clarity in visualization. We have revised the descriptions for the generation of visualization in the caption of Fig. 3. We will also provide comparative visualizations with methods like PointNet++ to better illustrate our model's ability to capture dynamics in the future appendix.

Response to [Question 2]: The PDEs-solving module is tailored for the task of point cloud video action recognition to address the issue we mentioned in Section 1 (pages 1-2). However, the unsupervised nature of the first training stage does indeed mean that the representations learned are not tied to a specific label set, allowing for potential generalization to other domains. However, limited time for rebuttal does not allow us to conduct extant experiments to further evaluate its variegation. Therefore, we plan to explore the potential for quick adaptation to other domains in our future research.

We thank the reviewer for the opportunity to improve our paper and hope our responses and planned revisions will address the concerns satisfactorily.

We thank the reviewer for recognizing the novelty and quality of our approach and for acknowledging the clear presentation and significance of our work.

Response to [Comment 1 & 7]: We acknowledge the importance of a rigorous mathematical foundation and a better interpretability of the proposed methods. We reorganize Section 3.2 in the revision paper to include a more thorough derivation of the PDEs and the reasoning behind our chosen formulations. Please check the revision paper for more details.

Response to [Comment 2]: We appreciate this insightful comment.

Firstly, we designed our PDEs-solving module based on the combination of the attention mechanism and the classic spectral methods for PDE. We explain the motivation for this design in our response to [Reviewer RMAg 's Comment 1]. Please check for more detail.

Secondly, we use a contrastive-based InfoNCE loss in our pretrain stage to refine the focus of dynamic capture to a distinct objective. By treating the $Feature \in \mathbb{R}^{T\times M}$ (getting from the encoder) before applying spatial/temporal pooling as the negative sample, we force the model to learn the implicit mapping between the spatial and temporal space, instead of reconstruct the de-pooling feature by closer both Ft and Fs to $Feature$ (which against the learning objectives). We have conducted several ablation studies in Section 4.3 (Tab. 8) to underscore the superiority of our loss design. Additional discussions will be included in the Section 3.2 to justify our choices.

Response to [Comment 3]: We addressed this by including a comparison with recent works like PointMapNet, showcasing the advancements our model brings to the field. These comparisons are added to Section 4.2 to strengthen our argument of the proposed method's superiority.

Response to [Comment 4]: We agree that evaluating our approach on a diverse set of tasks could demonstrate its generalizability. But, limited time for rebuttal does not allow us to conduct extant experiments to further evaluate its variegation. Therefore, we plan to extend our evaluation to tasks such as segmentation and detection for our future research.

Response to [Comment 5]: Yes, our PDEs-solving module can also improve the performance of other encoders. We have conducted ablation studies the PST-Transformer encoder to demonstrate the broad applicability of our PDEs-solving approach in Section 4.3 (Tab. 7).

Response to [Comment 6]: Thanks for this comment. To demonstrate the actual speed and offer a clearer picture of our model's efficiency, we provide runtime comparisons and will include this part in Section 4 (Tab. 2). The experiments is conducted on single NVIDIA A100 GPU, on the MSR-Action3D Dataset with 24 frames.

We are confident that the revisions and additions proposed in this rebuttal will address the concerns raised by the reviewer and improve the overall quality and impact of our work.