Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation

(Q1) Are the readings from 6 sensors belonging to Region 8 simply added up as the pressure of Region 8?

Instead of summing the values from the 6 sensors within Region 8, we define the maximum value among those sensors as the pressure for that region. Consequently, individuals with smaller hands might not activate all 6 pressure sensors in Region 8 during certain grasp postures. Therefore, we opted for the maximum value among the 6 sensors rather than the sum. Similarly, we chose the maximum over the average for the same reason. When only a subset of the 6 sensors are activated due to hand size discrepancies, using the average would result in an artificially low pressure reading due to the inactive sensors.

(Q2) Is it possible to briefly explain how the quality would be influenced by muscle fatigue as mentioned in L158?

Muscle fatigue is the decline in a muscle's ability to generate force. It happens when a muscle is repeatedly contracted or held in a sustained contraction for an extended period. Imagine holding a heavy book at arm's length – after a while, your arm will start to shake, and it will become increasingly difficult to keep the book up. This is muscle fatigue. The connection between muscle fatigue and EMG-based control is that muscle fatigue can negatively impact the quality of EMG data used for hand pressure estimation. As muscles tire, their electrical activity changes. This can manifest as increased signal amplitude (RMS) and shifted frequency content[R4]. These changes make it harder for the model to accurately distinguish between different hand postures and pressure levels. Moreover, fatigue leads to less consistent muscle activation patterns, making it challenging for the model to learn reliable relationships between EMG signals and exerted pressure[R5]. While muscle fatigue is recognized as an important topic in EMG-based control, it falls outside the scope of this study. To minimize its effects on our results, we ensured sufficient rest periods between data collection trials for both training and testing. To convey this consideration more explicitly, we will include a statement regarding this after Line 930 in Section B.3 Data Collection Protocol.

(W1) Additional dataset reference

As you suggest, we will cite and discuss in related work section as follows: A highly relevant work is the ActionSense dataset, which focuses on capturing multimodal data of human activities in a kitchen environment using wearable sensors. Similar to our approach, ActionSense emphasizes the importance of rich, synchronized, multi-modal data streams for understanding human actions and gathers . However, while ActionSense provides a valuable resource for understanding general kitchen activities, our work focuses on utilization of 3D hand posture into muscular force learning for understanding hand pressure estimation.

(W2) Additional cross-user evaluation

Please refer (Joint Response 2).

(W3) Additional results regarding absolute errors

We have included the Mean Absolute Error (MAE) results in the rebuttal PDF file. Table R6 and Table R7 in the rebuttal PDF file correspond to Table 2 and Table 3 in the main manuscript, respectively, and present the MAE metrics for each comparison.

(W4) Providing temporal information for predicted forces

• “A curve showing the predicted force changing with time would be preferable.”: We acknowledge the reviewer's insightful suggestion regarding the need for temporal information in our results. To illustrate the performance of our model over time, we will include figures showing the temporal evolution of both ground truth and predicted pressure values. Figure R1 in the rebuttal PDF file provides an example of such visualization. The figure presents the ground truth pressure and the pressure predicted by our model for all 9 hand regions during TM-Press and Medium Wrap actions. The horizontal axis represents time, each row corresponds to a specific hand region, and the vertical axis in each plot shows the pressure values. In this example, the user performs the TM-Press action for 10 seconds, followed by the Medium Wrap action for another 10 seconds. As evident from the figure, the location of pressure exertion and prediction shifts according to the action performed. We believe that these temporal visualizations will enhance the readers' understanding of our framework by clearly demonstrating how data is collected and predictions are made over time. We will include these figures and the corresponding discussion in a new Section D.4 titled "Comparison between Predicted Pressures and Ground Truth along with time" in the revised manuscript. Moreover, we’ll add one of those figures in the main body of the paper as in (Line 299) of Section 5.2.1.
• “Quantitative metrics on temporal consistency would be a bonus if possible.”: We would appreciate it if the reviewer could provide specific examples or references for "quantitative metrics on temporal consistency". This would allow us to better understand their suggestion and provide a relevant analysis.

(W5) Additional experiments and results for glove-less evaluation

Please refer (Joint Response 1).

(W6) quantitative prediction errors for different postures

You correctly pointed out that our study does not demonstrate the performance of our framework on unseen gestures during the training phase. We acknowledge this limitation. While constructing our dataset, we strived to collect data encompassing a wide range of hand interactions, including plane interactions, pinch interactions, and grasp interactions, to facilitate comparisons with previous works [R1,R3]. However, we also had to consider the limited time and potential fatigue of our participants. Therefore, we made a selection of representative postures, as shown in Figure 8 and Figure 9. Collecting additional data for novel gestures with new participants is infeasible in this rebuttal phase, preventing us from conducting further experiments on generalization. However, as highlighted in Lines 908 and 152, our selection of postures covers a considerable portion of representative movements from the reference grasp taxonomy and those presented in [R3]. Further research with new gestures and objects is certainly needed to solidify these points. Following this point, we will add this text into the limitation section:

• While our dataset covers a range of representative hand postures, it is limited in its exploration of novel gestures and object interactions unseen during training. Further research is necessary to investigate the generalizability of our framework to new objects and gestures. Expanding the dataset with a broader range of hand-object interaction scenarios and evaluating the model's performance on these unseen examples would provide valuable insights into its robustness and adaptability.

(Q3)Will the data glove introduce different interaction patterns from those without data gloves?

As outlined in Section 4.4 of the main text, we describe how we canonicalize 3D hand pose data derived from a vision-based hand pose estimator in Section B.4.2, 3D Hand Pose. During inference, we align the 3D hand pose estimated from visual data with the training data representation. This process involves rescaling the hand pose to match the bone lengths used during training, rotating it to align with the kinematic tree of the MANO hand model, and translating it to ensure consistent root joint positioning. This canonicalization minimizes the discrepancy between hand pose data from the glove and the vision-based estimator, improving the generalizability of our model during inference. Therefore, if the inference of hand pose is good enough, there will be no difference between wearing gloves and not. As noted in the Section E. Limitations and Future Works in the appendix, our framework inherently relies on the accuracy of off-the-shelf hand pose detectors during inference (Line 1097). This dependency is acknowledged as a limitation of the proposed modality and inference pipeline.

Due to limitations on the length of our rebuttal, we would like to address (Q1) and (Q2) in seperated comment.

(M1) Datasets for Hand Pressure Estimation — do these datasets include pressure, many of these appear to be object interaction papers

You correctly pointed out that the line between object interaction papers and hand contact/pressure papers can be blurry, and Table 1 and Section 3.1, Vision-based Hand Pressure Estimation, reflect this. While some datasets focus solely on hand contact, others include pressure information, and some delve into object interaction to varying degrees. Rather than attempting a strict categorization, which can be challenging due to the inherent interconnectedness of these areas, we chose to provide a more comprehensive overview in this section. This approach allows for a broader discussion of related work and a better understanding of the research landscape.

(M2) Fig 4: can you clarify if this participant was in the training dataset?

Yes. Of course, this testing session on display does not overlap with the training session.

(M3) L302 ‘significant’ - were significance tests run here?

No. Thanks to Reviewer YYDP, we understand that the term ‘significant’ was understood differently for readers. We’ll remove that expression to avoid misunderstanding

(M3) Table 3 - what are the relevant error bars for this table?

Table R5 presents the same data as Table 3 in the main manuscript, but includes the standard deviation for each reported value. We will update Table 3 in the revised manuscript to include these standard deviations, providing a more complete picture of the variability in our results.

(M4) Table 2:unclear what errors are over

As guided in the Experiments section (Line 248), we present the evaluation metrics in Appendix D.2. To assess our model's performance, we utilize three metrics: Coefficient of Determination (R²) which measures the proportion of variance in the actual pressure explained by our model, Normalized Root Mean Squared Error (NRMSE) which quantifies the precision of our model's pressure predictions, and Classification Accuracy, assessing the model's ability to correctly identify whether pressure is being applied to each region of the hand. We will add this summary to Line 248 of the main text.

(M5) Table 3: unclear what value is being plotted

The caption for Table 3 indicates that metric is NRMSE, but if there is still any ambiguity, please let us know again.

(M6) all exocentric?

Yes. In this research, we use an exocentric camera for camera modality. However, we believe it can be extended to usage of egocentric cameras if the hand pose estimation model is working well.

(M7) nit: Figure 3 pixelated

It’s a mistake. We’ll fix it.

(M8) Manuscript overstates at multiple points L316 "exceptional"

We’ll remove that expression to avoid overstatement. “Specific actions such as I-Press, M-Press, and R-Press exhibit high accuracy and low NRMSE,showing the model’s superior performance in simpler press interactions.”

(W1) Additional experiments for cross-user evaluation

Please refer (Joint Response 2).

(W2-1) Clarification on the comparison with PressureVision++ You are correct that the current comparison might not be entirely fair, and we acknowledge that our framework currently performs best when all fingertips are tracked. However, we'd like to clarify our approach in two ways. First, this limitation is inherent to the PressureVision and PressureVision++ methods themselves. As you pointed out, PressureVision++ relies on visual cues like color and shape changes on the hand caused by pressure. Consequently, it can only estimate hand pressure when all relevant visual features are visible in the camera image. In contrast, our approach doesn't require full hand visibility; as long as sufficient hand posture information can be inferred, pressure estimation is possible. Second, while we aim for comprehensive comparisons, PressureVision and PressureVision++ represent the only viable options available in current research for vision-based hand pressure estimation. While not a perfect comparison, we believe it offers valuable insights given the limited alternatives.

(W2-2) “The quantitative comparison between the two would be informative.”

Please refer (Joint Response 1).

(W3-1) Limitation on behavioral generalization to new objects and gestures

You correctly pointed out that our study does not demonstrate the performance of our framework on unseen gestures during the training phase. We acknowledge this limitation. While constructing our dataset, we strived to collect data encompassing a wide range of hand interactions, including plane interactions, pinch interactions, and grasp interactions, to facilitate comparisons with previous works [R1,R3]. However, we also had to consider the limited time and potential fatigue of our participants. Therefore, we made a selection of representative postures, as shown in Figure 8 and Figure 9. Collecting additional data for novel gestures with new participants is infeasible in this rebuttal phase, preventing us from conducting further experiments on generalization. However, as highlighted in Lines 908 and 152, our selection of postures covers a considerable portion of representative movements from the reference grasp taxonomy and those presented in [R3]. This suggests a degree of generalizability, but further research with new gestures and objects is certainly needed to solidify these points. Following this point, we will add this text to the Limitation section (Line 1107):

• While our dataset covers a range of representative hand postures, it is limited in its exploration of novel gestures and object interactions unseen during training. Further research is necessary to investigate the generalizability of our framework to new objects and gestures. Expanding the dataset with a broader range of hand-object interaction scenarios and evaluating the model's performance on these unseen examples would provide valuable insights into its robustness and adaptability.

(W3-2) Clarification on matching the glove and vision-based hand pose estimation framework

You are correct, and we don't claim this to be a perfectly fair comparison. However, we'd like to provide some context on our rationale in two points. First, the authors of PressureVision++ specifically highlight its effectiveness in interacting with diverse surfaces (3rd, 4th, and 6th paragraphs of Introduction [R1]), even presenting training and data collection methodologies designed for such scenarios. This led us to believe that PressureVision++ would be a suitable candidate for comparison with our vision-based method. Second, when considering vision-based methods for comparison, we found PressureVision and PressureVision++ to be the only viable options available in existing research. While it may not meet your standards for a completely fair comparison, we believe we made the best possible choice given the available options.

(Q2) Why does utilizing hand angles not improve the model performance? Based on the current results of our ablation study, we believe that hand angles alone lack sufficient inductive bias to effectively capture the spatial information inherent in hand postures.

(Q3) How was 3D hand pose acquired in cases without the glove? We specify the utilization of an off-the-shelf hand pose detector in Section 5's introduction and Section 4.4, explicitly stating that we chose to employ ACR (Line 244). Additionally, as mentioned earlier, Section B.4.2, 3D Hand Pose, describes the canonicalization process for transforming the results of vision-based pose estimation into a standardized 3D hand pose representation (Lines 943-975).

(Q4) Can you show raw traces of the discretized pressure to give an intuition for what the real pressure data looks like?

To visually illustrate the performance of our model over time, we will include figures showing the temporal evolution of both ground truth and predicted pressure values. Figure R1 file provides an example of such visualization. The figure presents the GT pressure and the pressure predicted by our model for all 9 hand regions during TM-Press and Medium Wrap actions. The horizontal axis represents time, each row corresponds to a specific hand region, and the vertical axis in each plot shows the pressure values. In this example, the user performs the TM-Press for 10 seconds, followed by the Medium Wrap for another 10 seconds. We believe that these temporal visualizations will enhance the readers' understanding of our framework by demonstrating how data is collected and predictions are made over time. We will include these figures and the corresponding discussion in a new Section D.4. Moreover, we’ll add one of those figures in the main body of the paper as in (Line 299) of Section 5.2.1.

Due to limitations on the length of our rebuttal, we would like to address the minor points in separated comment.

(W1) Clarifications on the novelty of the proposed framework

• We acknowledge that there might be room for improvement in terms of methodological novelty. In this paper, however, our primary objective of this paper was to pioneer the use of electromyography and 3D hand posture data simultaneously and demonstrate its efficacy. To achieve this, we presented a data collection platform and a pipeline for using our model in inference without a data glove. Subsequently, to validate the approach experimentally, we deliberately opted for the simplest form of machine learning modeling approach instead of focusing on introducing a new model architecture by incorporating geometric structures of the model and data, our work lays a strong foundation for future research in this direction. We will add this point in the Limitations and Future Work section (Line 1107) as follows:
• “While our current framework demonstrates the value of combining sEMG and 3D hand pose data, it primarily focuses on leveraging global features from both modalities. Future work could explore incorporating fine-grained spatial information and prior structural knowledge of hand poses into the model architecture to further enhance pressure estimation accuracy and robustness.”

(W2) Clarification on the comparison with PressureVision++

• We agree that this is not a perfectly fair comparison. Regarding this, we would like to clarify the rationale for our comparison and explain the additional quantitative comparison with PressureVision++ we carried out during our rebuttal. Unlike PressureVision, the authors of PressureVision++ specifically highlighted its effectiveness in interacting with diverse surfaces (3rd, 4th, and 6th paragraphs of Introduction in [R1]), even presenting training and data collection methodologies designed for such scenarios. This led us to believe that PressureVision++ would be a suitable candidate for comparison with our vision-based method, which is why the primary reason we chose PressureVision++[R1] over PressureVision[R2]. When considering vision-based methods for comparison, we found PressureVision and PressureVision++ to be the only viable options available in existing research. While it may not meet your standards for a completely fair comparison, we believe we made the best possible choice, given the available options. We carried out a quantitative comparison with PressureVision++ to support our claim with a fairer condition.

(W3) Clarification on utilizing vision-based hand pose estimation model for testing setup

• As outlined in Section 4.4 (Line 228), we canonicalize 3D hand pose data derived from a vision-based hand pose estimator (More details shown in Section B.4.2, Line 942). During inference, we align the 3D hand pose estimated from visual data with the training data representation. This process involves rescaling the hand pose to match the bone lengths used during training, rotating it to align with the kinematic tree of the MANO hand model, and translating it to ensure consistent root joint positioning. This canonicalization minimizes the discrepancy between hand pose data from the glove and the vision-based estimator, improving the generalizability of our model during inference. Also our framework inherently relies on the accuracy of off-the-shelf hand pose detectors during inference (Line 1097). We acknowledge that this dependency is a limitation of the proposed modality and inference pipeline and discuss this limitation in Section E. Limitations and Future Works (Line 1077).

(Q1) Benefits of utilizing predicted hand poses as input for training

• While using predicted hand poses during training might seem beneficial, our training dataset requires participants to wear a glove to collect the ground truth hand pressure data. We utilize a pressure-sensing glove (TactileGlove, Pressure Profile Systems) for this purpose. Unfortunately, most current hand pose detectors are not trained to work on hands wearing gloves, or their performance significantly degrades in such situations. Therefore, it is hard to incorporate predicted hand poses during the training phase due to the need for more reliable hand pose estimation for gloved hands.

(Q2) Have you tried other multimodal information fusion methods?

• We haven't explored alternative multimodal information fusion methods in this work. However, we acknowledge that there are numerous potential improvements to be made in both the model design and implementation. Investigating different fusion techniques is a promising direction for future work, and we believe it holds the potential to enhance the accuracy and robustness of our hand pressure estimation framework. We will add this point into limitation and future works section as follows: “While our current framework demonstrates the value of combining sEMG and 3D hand pose data, it primarily focuses on leveraging global features from both modalities. Future work could explore incorporating fine-grained spatial information and prior structural knowledge of hand poses into the model architecture to further enhance pressure estimation accuracy and robustness.”

(W1) Support for advantages of integrating sEMG signals with 3D Hand Posture data

• We would like to clarify that we performed an ablation study (Table 2 and 3) and discussed study results in the corresponding sections. These tables compare the performance of our model using: (1) sEMG data only[R3], (2) sEMG data with hand posture data using angle representation, and (3) sEMG data with 3D hand pose data(Ours). We primarily discussed this analysis in Section 5.2.1 highlighting the benefits of combining sEMG and 3D hand pose information. We acknowledge that our initial explanation of the motivation might not be sufficiently convincing for readers. To provide stronger empirical motivation for our approach, we will add a new figure illustrating cases where similar sEMG patterns are observed despite pressure being applied to different parts of the hand. Figure R4 demonstrates this phenomenon. The figure displays the time-series representation of 8-channel sEMG signals over 5 seconds for two pairs of hand postures: (1) I-Press and M-Press, (2) TI-Pinch and TM-Pinch. Each row represents a different sEMG channel. In both pairs, the pressure is applied differently, targeting either the index fingertip or the middle fingertip. However, we observe very similar sEMG patterns between I-Press and M-Press, and between TI-Pinch and TM-Pinch. This observation demonstrates that while sEMG signals provide valuable information about force, they may not be sufficient for fine-grained pressure localization on their own. This highlights the potential of incorporating hand posture information to guide pressure estimation. We will incorporate this discussion and Figure R4 into the revised manuscript, adding a new second paragraph in Section 4.1 (Llines 172-173) and a new Appendix section titled "Empirical motivation of our framework" to further clarify the rationale behind our approach.

(W2) Comparison with vision-based datasets/techniques extracting 3D hand posture info

• We agree that we did not comprehensively cover various vision-based datasets or methodologies for 3D hand estimation as related works or comparative methodologies. However, please take it into consideration that the ultimate goal of our research is to estimate hand pressure, not to derive 3D hand posture. For this reason, we selected comparative methodologies and datasets aimed at estimating hand contact or hand pressure rather than 3D hand posture estimation. Since the ultimate goal of our research is hand pressure estimation, we believe that comparisons with 3D hand pose estimation methodologies and datasets may not be contextually consistent. However, we also found out that we missed highly related dataset paper such as ActionSense from NeurIPS 2022 which will be added as part of our related works.

(W3) Clarification on the meaning of the robustness in our paper

• In our manuscript, the 'robustness' of the model refers to the model’s capability to ensure high accuracy across various postures over different hand regions. Here, we have detailed the performance across different hand regions (Table 3) for various postures (Figure 3) to show our model’s robustness. Thanks to Reviewer 3PTe, we understand that the term 'robustness' was understood differently by readers, we will revise it to clearly convey our intended meaning. Specifically, we will revise the following lines as follows:

• (Line 313) “This consistent correlation between predicted values and actual pressure measurements highlights the model’s robustness and adaptability in handling a wide range of hand movements.” → “This consistent correlation between predicted values and actual pressure measurements highlights the model’s ability to maintain high accuracy and reliability across a diverse range of hand parts and postures, thereby demonstrating its robustness.”

• (Line 331) “Figure 4b shows demo video footage illustrating our framework’s robustness in estimating hand pressure while continuously changing hand posture, pressure levels, and the objects being grasped.” → “"Figure 4b shows demo video footage illustrating our framework’s capability in accurately estimating hand pressure while continuously changing hand posture, pressure levels, and the objects being grasped."