Sparse Mask Representation for Human-Scene Interaction

Q1: The novelty is limited to sparsifying the input for the human-scene interaction. Using sparse inputs by itself is not new, but it has not been studied for this task before.

The use of sparse masks to enhance model speed has been investigated and established as a path within Sparse Coding solutions, as outlined in our Related Work. However, as you mentioned, its effectiveness on the human-scene interaction task has not been studied before. Unlike other methods that mostly apply the sparsity for improving inference speed, our method can increase both accuracy and speed. Despite the simplicity, our method outperforms recent work by a significant margin, for example, we outperform MIME [1] with 2.72% higher accuracy and 60 times faster.

Q2: In Figure 6 and Table 4. It seems the model gets worse when using 50 or 10 masks which is strange. Why would using 3 masks be better than 10 or 50 masks? Why would it be even better than using the full dense tensor?

We value the feedback provided by the reviewer. The POSA backbone operates on dense tensor inputs. However, not all vertices contribute meaningfully to predicting human-scene contacts, leading to redundant information that can induce overfitting during the learning process. Sparse masks aim to mitigate this issue by increasing sparsity and selecting useful information. Nonetheless, an excessively large set of sparse masks can introduce a similar problem of redundant information, which adversely affects results when employing 50 or 10 masks.

Among these, the kept 3-mask setup refined by our introduced Sparse Mask Refinement technique yields the best results. This refinement process helps eliminate redundant sparse masks, enabling even better performance than the baseline which still suffers from redundant information issues. For further detailed insights, please refer to Section 4.4 'How do sparse masks help reduce input information?'

Q3: The paper attributes the COO representation to "Choy 2020". The COO format is much older than that.

We appreciate the reviewer for pointing this out and concur that the original COO format has been appropriately cited. To the best of our knowledge, the initial representation of a sparse tensor using the COO format is documented by [2] (Chou et al., 2018), and subsequent extensions to operations on sparse tensors are detailed in [3] (Choy et al., 2020). Please do let us know if we missed any citations.

Q4: Figure 4 is hard to see. Human bodies are too small.

We appreciate your feedback. We have enhanced the size of Figure 4.

Q5: The method is basically learning mesh subsampling. I wonder about how the method compares to classic subsampling methods.

We appreciate the reviewer for mentioning the comparison with mesh subsampling methods. We agree that these methods also enhance the inference speed, with much of the mesh subsampling process occurring at runtime.

The table below illustrates the comparison between our proposed SMR and other Mesh Subsampling methods, including typical algorithm-based methods and deep-based ones. Across all cases, our method significantly outperforms other approaches in terms of both speed and accuracy. This result is easily explainable, as most mesh subsampling methods do not adhere to any specific algorithms during the subsampling process to preserve model performance. Additionally, their primary objectives revolve around finding a better discrete representation of a mesh with triangles of equal edge length, rather than catering to human-scene interaction tasks. Besides, some deep-based mesh simplifier methods also take time to finish their sampling process.

|Methods|Mesh Subsampling|Reconstruction Accuracy(%)|Speed (s/sample)| |:-|:-:|:-:|:-:| |Baseline (POSA)|Algorithm-based|91.12|0.28| |Isotropic Remeshing [4]|Algorithm-based |82.18(-8.94)|0.18($\downarrow$ 1.56)| |Vertex Clustering [5]|Algorithm-based | 78.67 (-12.45)|0.13($\downarrow$ 2.15)| |Incremental Decimation [6]|Algorithm-based | 79.45 (-11.67) | 0.09 ($\downarrow$ 3.11)| |Neural Mesh Simplification [7]|Deep-based|85.56(-5.56)|0.47($\uparrow$ 1.68)| |CoMA [8]|Deep-based|89.22(-1.90) |0.23($\downarrow$ 1.22)| |Ours|-|93.69(+2.57)|0.01($\downarrow$ 28)|

Result comparison between SMR and other mesh subsampling methods. Results are benchmarked on the PROXD dataset.

Dear Reviewer BCwn,

We appreciate your time and feedback on our paper. We have responded to all the concerns you raised. Please let us know if you have any further questions.

Best regards, Authors.

Q1: The current version appears to be an application of the Choy, 2020 citation. Clarification on the contribution beyond this should be provided.

• Our method is an input representation technique that centers on a learned set of sparse masks to generate zero-filtered data points.
• COO is a matrix reformatting technique used to restructure the inputs. The indexing characteristic of the COO enables us to safely eliminate the mentioned zero points while guaranteeing the incorporation of the remaining data into the network.
• Simply speaking, our method learns and ranks the sparse masks, while the COO ensures sparsity implementation. Our experiment shows that we need both the sparse masks and the COO to obtain the best results.

Q2: As mentioned, the acceleration could be attributed to two factors. First, a sparse body mesh with 90% fewer vertices is used. Second, the sparse network works. Ablation should be conducted on the sparse mesh only and the sparse network only.

We appreciate the reviewer's question. The table below shows some results regarding the contribution of sparse masks and a sparse network. POSA serves as our baseline, and if setups involve masks, three masks are used. It is evident that when we use sparse masks without the COO format, the learning is not effective since there are too many zero points from the input without being removed, leading to no improvement in speed and, in fact, a decrease in accuracy. Applying a sparse network to original inputs improves speed, but the trade-off for accuracy is noticeable, as discussed in many previous papers. When we apply the COO format to the original inputs, the differences in speed and accuracy are not significant compared to the original baseline. If sparse masks and the COO format are both used without the sparse network, we must retain all zero points, even in the COO format, to match the input shape for the network. Consequently, no improvement in speed is observed, and the model's effectiveness is limited. With our introduced Sparse Mask Refinement that works on COO format, we can preserve the performance of the model but the speed improvement is not guaranteed. Ultimately, when we use COO inputs obtained from sparse masks and a sparse network together, with the stored indexes in COO format, the input shape problem can be addressed, achieving optimization in both speed and accuracy.

|Network|COO Format|Sparse Masks|Sparse Mask Refinement|Reconstruction Accuracy (%)|Inference Speed (s/sample)| |:-:|:-:|:-:|:-:|:-:|:-:| |Dense||||91.12|0.28| |Dense||x||85.72(-5.4)|0.28($\downarrow$ 1.0)| |Sparse||||83.41(-7.71)|0.09($\downarrow$ 3.11)| |Dense|x|||91.02(-0.1)|0.27($\downarrow$ 1.04)| |Dense|x|x||85.46(-5.66)|0.28($\downarrow$ 1.0)| |Dense|x|x|x|93.27(+2.15)| 0.28($\downarrow$ 1.0)| |Sparse|x|x|x|93.69(+2.57)|0.01($\downarrow$ 28)|

SMR Component Analysis. Results are benchmarked on the PROXD dataset. POSA is used as a backbone. The kept 3-mask setup is used when Sparse Masks are available.

Dear Reviewer gK1o,

Many thanks for your time and feedback on our paper. As the authors-reviewers discussion will end soon, please let us know if you have any further questions or comments.

Best regards,

Q1: How the sparse masks are defined, especially in the context of task dependency?

The sparse masks are randomly generated based on the input format used for human-scene contact prediction. Each mask is a tensor whose shape is similar to the shape of the input. In the contact prediction task, each input is represented by using a tensor with shape $N_v \times N_S$, where $N_v$ is the number of vertices to represent the human mesh, and $N_S$ is the number of features (coordinate and contact label) of each vertex.

Q2: Are the sparse masks randomly generated, meaning do the 0 and 1 values occur at random locations? Or are the masks specifically tailored to the task at hand?

Yes, sparse masks are randomly generated. During the learning process, the model relies on the mask score $\alpha$ to identify the contribution of masks. Only masks that show a high enough contribution are kept.

Q3: How does your approach handle fine-grained contacts, such as situations where the tips of the fingers come into contact with other objects?

The datasets used in our task (PROXD, GIMO, BEHAVE) do not consider fine-grained contacts. For instance, a part of the human body, such as a hand or foot, is assumed to make contact with at most one object. We believe our method will work effortlessly with fine-grained contacts, however, the current limitation in datasets does not allow us to verify the results of this case.

Q4: Could you elaborate on how your approach addresses videos?

Our contribution is the sparse mask representation, and it works both with frames or video-based backbone. For example, to manage consecutive frames in the Scene Synthesis task, we simply employ the algorithm-based ContactFormer backbone. In a standard configuration, ContactFormer typically receives only one input from the contact predictor to compute contextual dependencies and the dynamic evolution of interactions. However, in our proposed SMR, ContactFormer can consider multiple contacts with associated weights. This capability enables it to establish more robust contextual dependencies, leading to improved reconstruction results. A recap of our approach and other methods in the mentioned task can be found in Section A.2 of our Appendix.

Q1: About the methodology contribution.

We thank the reviewer for the comment, we verify our contribution as below:

• We propose a simple yet effective method that focuses on the representation of inputs used for human-scene interaction. Our method reduces processing time while enhancing model effectiveness by eliminating redundant information. Existing works on human-scene interaction have not yet investigated this problem.
• We not only utilize sparse masks as in previous work but also propose a refinement strategy to create zero-filtered inputs.
• Despite simplicity, our method outperforms other state-of-the-art methods significantly. For example, our method outperforms MIME [1] with 2.72% higher accuracy and 60 times faster.

Q2: I can see that from Figure 6. it shows optimal performance for k = 3 mask with 90% sparsity, but there is no clear pattern or correlation between sparsity ratio vs accuracy. Could authors give an explanation on this trend?

We appreciate the reviewer's feedback. In the original submission, we have shown the correlation between the sparsity ratio, number of masks, accuracy, and running time in Figure 6. That figure shows that using only one mask results in a significant drop in performance due to missing information. Increasing the number of masks helps retain essential information and enhances the overall model performance. However, an excessive number of sparse masks (50 masks) can introduce redundant and inconsistent information, potentially lowering the overall accuracy. By maintaining the sparse masks at appropriate values (which can be easily achieved using Sparse Mask Refinement), we can optimize performance while significantly speeding up the evaluation process.

As per your suggestion, we have further added a comprehensive discussion about the mentioned information in Section E.1 of our Appendix, providing a more comprehensive exploration. This includes an in-depth analysis of the correlation between the number of sparse masks and sparsity concerning accuracy. Additionally, another experiment examining inference speed is also conducted within this section.

Dear Reviewer fpDr,

Please do let us know if you have any further questions before the authors-reviewers discussion period.

Best regards, Authors

Dear Reviewers fpDr, t62A, gK1o, BCwn

We sincerely appreciate the time and effort throughout the reviewing process of our submission. As the author-reviewer discussion is due soon, please let us know if you have further questions about our submission.

Once again, thank you in advance, and we look forward to your feedback.

Best regards, Authors.