Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion

We thank Reviewer 7SiD's insightful comments and suggestions. We are also glad to see that Reviewer 7SiD acknowledges the contributions of our work, such as “The data processing ... shows a well-designed filtering strategy”, and “The model design is novel”.

W1: statements of mutual attention module

R1: Thanks for your comments. We observe that during the modeling of interactive gestures, the two speaker movements are invariant to the order of audio signals, holistically. For example, if we exchange the input audio signals C_a and C_b into their counterpart denoisers, the generated results would not be influenced. We will further clarify this.

Q2: Raw data Release

R2: Thanks for your suggestion. Our dataset is being organized, and we will open-source our dataset and processing code as soon as possible.

Thanks to the authors for their feedback and consideration regarding the data release.

For the TIM models, since the authors demonstrate the interaction performance is better than the baseline by user study, it is acceptable that the proposed model is relatively straightforward for a dataset paper, such as the baselines in AIST++, BEAT etc, so it may not be overly criticized.

For the dataset, I think it has the potential to be beneficial to the community and could accelerate further research, since there are no concurrent works on in-the-wild video data, previous in-lab datasets like TalkingWithHands, ConvoFusion are still too small to have enough gesture patterns. Also, they lack original RGB video, facial expressions, and body shapes compared with this dataset. This makes me believe the paper is valuable for acceptance. 

Considering the mixed scores, I raised my score to 10 to encourage dataset work to enter the discussion phase. The dataset presented, especially the raw data and the sharing of filtering, is potentially impactful, and may need further discussions.

We thank reviewer ryqB for your valuable time and insightful comments. We have tried to address your concerns in the updated manuscript and our rebuttal is below.

W1: Dataset Quality

R1: Thanks for your comments and valuable suggestions.

1) Considering the complex interaction between two hands and the human body, it is quite challenging to obtain precise postures when facing body occlusion. Therefore, as we claimed in Lines 815-817, we conduct five strict criteria (eg, no missing joint of the upper body) to filter low-quality videos. Moreover, we also recruit human inspectors to manually check the visualized results following a ratio of 5:1. In this manner, the quality of our dataset is greatly improved. Despite this, the hand self-penetration problem is still worth exploring and improving in pose estimation research, especially during mesh-based demonstration. Therefore, inspired by [A] [B] we directly model the human body joint movements in our experiments to address the self-penetration impact. As suggested, we will further provide more comprehensive example videos and add corresponding discussion in the revision.

2) During experiments, we follow the convention of [C][D][E] to resample FPS as 15. In our dataset, we retain all the meta data (eg, video frames, poses, facial expressions) within the original FPS (ie, 30) of talk show videos. We will release our full version data and pre-processing code, thereby researchers can obtain various FPS data according to their tasks.

3) We set the lower body to unified seat postures while preserving standard shape both for dataset and generated videos during visualization (post-processing). For dataset construction, we obtain comprehensive parameters of SMPLX including whole body joints, shape, root global orientation, etc. For the generation, as reported in Lines 862-863, considering the complex and variable positions of the two speakers of in-the-wild videos, similar to [A][B], we just model the upper body joint movements (without shape or root position). Moreover, the body shape and root position are fixed due to the weak correlation with human speech. For example, it is quite challenging to model from only audio inputs whether the two speakers are tall or short, sitting or standing.

[A] Generating holistic 3d human motion from speech, in CVPR 2023

[B] Towards Variable and Coordinated Holistic Co-Speech Motion Generation, in CVPR,2024

[C] BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis, in ECCV 2022

[D] EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling, in CVPR 2024

[E] Learning hierarchical cross-modal association for co-speech gesture generation, in CVPR, 2022.

W2: Comparison with the previous dataset

R2: Thanks for your constructive suggestions. As reported in the Table below, we conduct a comparison with [2]. The duration of [2] is validated based on the conversational corpus based on the officially released GitHub link. Although [2] contains concurrent gestures, our dataset achieves more than 4x larger than it. Moreover, the extensive multi-modality attributes (eg, facial expression, phoneme, etc) in our dataset significantly facilitate research on various downstream human dynamics-related tasks like talking head generation. We will add this discussion to the revision.

W3: Interaction Metric

R3: Thanks for your insightful comments. Indeed, an interaction-specific metric would definitely improve our work. Actually, we tried our best to conduct literature reviews and could not find any metric that can be directly applied to our task. Therefore, we conduct the user study including a specific-designed interaction coherency perspective to evaluate the quality of interactions between two speakers. As reported in Figure 5, our method achieves the best performance against all competitors. We will focus on addressing this issue in the future.

W4: Audio process

R4: Thanks for your suggestions. Overlapping speech is handled by exploiting both the speaker-specific features in separated audios $C_{a}$ and $C_{b}$ and global information in $C_{mix}$. We leverage separated human speech as guidance for bilateral branches to generate corresponding gestures. With respect to $C_{mix}$, we fed it into the audio encoder for better focus on interaction information. In this fashion, the synthesized posture retains the rhythm of specific audio while preserving interactive coherency with the conversation partner. As suggested, we conduct the ablation study to verify the effectiveness of the original mixed audio signal $C_{mix}$. By subtracting the original mixed audio, the indicators FGD and BC present much worse performance. These results verify the mixed audio signal displays effectively enhance the interaction between two speakers. We will add this discussion and more examples of overlapping speech in the revision.

W5: Ambiguous Figure

R5: Thanks for your comments. The figure is correct. As we stated in W4, our audio encoder actually takes $C_{mix}$, $C_{a}$, and $C_{b}$ as inputs. The original mixed audio signal can be directly obtained by the addition of $C_{a}$ and $C_{b}$. Before feeding these audio signals into the audio encoder, we convert them into mel-spectrograms as unified dimension $128 \times 186$. We will clarify this in the revision.

Q1: Separation of Speaker Audio

A1: We obtained speech separation results with an accuracy of 95%. Here we defined correct instances as those audio clips with accurate speech segmentation, correct text recognition, and accurate alignment. During our audio preprocessing, the audio is initially segmented by pyannote-audio technique to achieve 92% accuracy. Then, the accuracy of text recognized by WhisperX is 96%. Once we obtain the initial audio segments and corresponding recognized text transcripts, we recruit professional human inspectors to manually align the audio segments with spatial-wise speakers. Meanwhile, the human inspectors would further check the rationality of segmentation and text recognition results from the perspective of human perception. In this step, we set two groups of inspectors for cross-validation to ensure the final alignment rate is 98%. We will provide more examples of audio separation in the revision (refer to our webpage: https://anonymous.4open.science/w/Co3-F300/).

Q2: Only Upper Body & Body Shape and root

A2: As refers to the response of W1, considering the complex root positions and low correlation with human speech, we only model the upper body joint movements without shape in the experiments.

Q3: Unclear Representation

A3: Thanks for pointing these out. Regarding the body shape, please refer to the responses of W1, W4, and W5. The dimension of input audio mel-spectrograms ($C_{a}$) is $128 \times 186$. Our experiments only contain upper body joints without facial expressions and shape parameters. The upper body ($x_{a}$) contains 46 joints (i.e., 16 body joints + 30 hand joints) of each speaker as we stated in Lines 343-345. Additionally, the dimension of the generated motion sequence is $\mathbb{R}^{90\times 276}$, where 90 denotes frame number and $276 = 46 \times 6$ means upper body joints. The order of each joint follows the original convention of SMPL-X. We will provide a detailed joint order list in the revision.

Thanks again for your great efforts and constructive comments in reviewing this paper. As for your raised concerns, we add the point-by-point response below.

Quality of the dataset

Since our data is collected from in-the-wild talk show videos, the spatial positions of the two speakers are complex and varied, influenced by the dialogue scene. For example, as shown in the demo pager, the two speakers may sit parallel to each other, or they may sit on chairs of different heights. Meanwhile, the spatial distance between the two speakers also changes. It is quite challenging to model such complex and various spatial positions and distances from the conversational speech of two speakers. Therefore, we fixed the root joints of speakers in the experiment to facilitate model training.

To better present our dataset, we try our best to visualize the rendered 3D postures incorporated RGB video frames. However, 20% of the dataset includes more than 6K samples within 14 hours, which is quite difficult to present on a web page during the rebuttal period. We will try our best to show more visualization samples (eg, 5-20 cases) during the rebuttal, hoping to meet your requirements.

Investigation and discussion of prior work

As suggested, we compare our work with the TWH16.2 dataset in Table 1 of the revised paper (already submitted) and add the corresponding discussion. The TWH16.2 is only composed of joint-based body movements. In contrast, our dataset contains whole-body meshed SMPL-X human postures, which are more convenient for avatar rendering and various downstream tasks (eg, talking face).

In the GENEA Workshop 2023 Challenge, the participants aim to model the single-person gesture from the conversational corpus incorporated with interlocuter reaction movements[A][B]. This experimental setting is different from ours in that synthesizes the concurrent gestures of both speakers. The corresponding discussion is added to the revision.

[A] Diffugesture: Generating human gesture from two-person dialogue with diffusion models. In GENEA Workshop 2023 Challenge, 2023.

[B] The GENEA Challenge 2023: A large-scale evaluation of gesture generation models in monadic and dyadic settings. In Proceedings of the 25th International Conference on Multimodal Interaction (pp. 792-801).

Interaction Metric

Indeed, we have tried our best to conduct literature reviews and didn't find any metrics that can be directly applied to the concurrent gesture generation. Therefore, we conduct an ablation study to specifically evaluate the interaction coherency of two speakers. With regard to this point, we add the corresponding discussion in the revision and will put more effort into designing specific metrics in the future.

Quality of the manuscript

1. With regard to $C_{mix}$, we have updated our statements in the revision. Actually, $C_{mix}$, $C_{a}$, and $C_{b}$ are all raw audio waves. Direct addition of separated audio would not impact the quality of mixed audio signals.

2. We have added the corresponding clarifications and citations in the revision.

We hope our answers and updated paper can help to address the concerns raised in the initial reviews. We would highly appreciate active discussions from the reviewers and we are happy to clarify any further questions.

We thank Reviewer CYKi's for the valuable comments and suggestions. We further clarify and correct the problems of the experimental setup and ablation results analysis.

W1: Spatial relationships between the two speakers && sitting postures

R1: Thanks for your comment.

1) Considering the asymmetric spatial movements of two speakers, we model interactive gesture dynamics via two bilateral cooperative branches. Due to the complex and various positions of the two speakers in our source talk show videos, we label the relative spatial positions in the data processing. By assigning the specific spatial position (ie, left or right) to the speaker identities in the diarization, we finally obtain the separated audio signals with corresponding body movements, as we stated in Figure 2 and Lines 852-854.

2) Similar to [A][B], we just model the upper body joint movements in the experiments. The lower body including legs are fixed (eg, seated) while visualizing due to the weak correlation with human speech. For example, it is quite challenging to model whether the two speakers are sitting or standing from only audio inputs. We will clarify this in the revision.

[A] Generating holistic 3d human motion from speech, in CVPR 2023

[B] Towards Variable and Coordinated Holistic Co-Speech Motion Generation, in CVPR,2024

W2: Pre-trained models for competitors

R2: Thanks for your suggestions. To ensure as fair a comparison as possible, we only leverage the pre-trained audio feature extractors of some competitors that follow their original statements. These audio feature extractors are usually utilized as universal components in the network of the gesture generation community (just like CLIP in text feature extraction). Specifically, in DiffSHEG, we follow the convention of the original work to utilize the pre-trained HuBERT[C] for audio feature extraction. In TalkSHOW, we exploit the pre-trained Wav2vec[D] to encode the audio signals following the original setting. Apart from this, the remaining components for gesture generation in DiffSHEG and TalkSHOW are all trained from scratch on the newly collected GES-Inter dataset. In this fashion, the reliability of results yielded by these methods would not be impacted. We will clarify this in the revision.

[C] Hubert: Self-supervised speech representation learning by masked prediction of hidden units, in TASLP, 2021.

[D] wav2vec 2.0: A framework for self-supervised learning of speech representations, in NeurIPS, 2020.

W3: Ablation study for TIM with MLP layer

R3: Thanks for your valuable comments. As suggested, we conduct an ablation study that replaces the TIM via a simple MLP layer. The FGD and BC display the obvious worse impact as shown in the Table below. The results verify that our TIM effectively enhances interactive coherency between two speakers. We will add this discussion in the revision.

W4: Qualitative results for ablated models

R4: Thank you for your suggestion. As we illustrated in Figure 7 in the supplementary material, we showcase the keyframes of the ablation study to demonstrate qualitative comparison. The full version of our method displays the best performance against different variants. Moreover, we will incorporate additional visualization demonstrations in the revision (refer to our webpage: https://anonymous.4open.science/w/Co3-F300/).

W5: User study with InterGen

R5: As the detailed scores of the ablation study reported in Table below, our method even achieves 10% ((4.4-4.0)/4=10%) improvement over InterGen in Naturalness. Meanwhile, the results generated by InterGen in comparison videos contain some obvious unreasonable postures (eg, the neck of the left speakers, and the right hand of the right speakers). Moreover, we conduct a significant analysis for the user study as referred to Q3. The results demonstrate that our method outperforms InterGen with a large marginal improvement. We will further highlight these areas for better demonstration in the revision.

Q1: Baseline methods utilized pre-trained models

A1: As referring to the response of W2, we only incorporate the pre-trained audio feature extractor in DiffSHEG and TallSHOW, which follow the original experimental settings.

Q2: Additional ablation studies incorporating simple fusion modules

A2: As referring to the response of W3, we conduct the additional ablation study to demonstrate the effectiveness of the proposed TIM.

Q3: Could the authors perform a significance analysis for the user study?

A3: As suggested by your constructive comments, we conduct a significant analysis of the user study using t-test, focusing on three key aspects: Naturalness, Smoothness, and Interaction Coherency. The results verify our method surpasses all the counterparts with significant improvements, including sub-optimal InterGen. In particular, for all the comparisons between our model and the other models, we formulate our null hypotheses (H0) as "our model does not outperform another method". In contrast, the alternative hypothesis (H1) posits that "our model significantly outperforms another method," with a significance level (α) set as 0.05. Here, we perform a series of t-tests to compare the rating scores of our model against each of the other competitors individually and calculate all the t-statistics shown in the table below. Since we recruit 15 volunteers, our degree of freedom(df) for every analysis is 14. Then, we look up the t-table with two tails and find out all the p-values are less than 0.05 (α). Therefore, we reject the null hypotheses, indicating that our model significantly outperforms all the other methods in every aspect.

Thank you for your response and the additional experiments. Most of my concerns have been adequately addressed. However, I recommend that the authors discuss the limitations related to the lack of precise spatial relationships in the revised manuscript. Based on these revisions, I am inclined to raise my score from 5 to 6.

W5: Discussion about visual results and further highlighting

R5: Thank you very much for your constructive suggestions. As refers to the response of W4 (b, c), our method models the 90-frame gesture movements in the experiments. Thus, all the demo videos are 6 seconds. The jittery effects are mostly caused by the blurring of speakers moving quickly in consecutive video frames. As stated in Lines 825-827, during our data processing, we translate the joint of arm postures as Euler angles with x, y, and z-order. Then, if the wrist poses exceed 150 degrees on any axis, or if the pose changes by more than 25 degrees between adjacent frames (at 15 fps), we discard these abnormal postures over a span of 90 frames. Moreover, we conduct manual detection with a ratio of 1:5 to eliminate unnatural instances. Despite the huge effort we put into data preprocessing, the automatic pose extraction stream may influence our dataset with some bad instances. For better demonstration, we will follow your suggestion to report the length quantification of motion fractions in demo videos. In addition, we will zoom in on the details of the gesture for highlighting. The corresponding discussion will be added in the revision.

Q1: Some typos

A1: Thank you for pointing it out. We will correct the typo in the revision.

We thank Reviewer 2ueC very much for the high evaluation that ''the proposed method is technically sound'' and ''the proposed dataset is carefully curated and thoughtfully designed''. We further clarify the problems of the experimental setup and results analysis.

W1: Some additional dataset preprocessing details

R1: Thanks for your comments.

a) In these preprocessing phases, due to the large collected raw video and labor consumption, we conduct initial filtering via some automatic techniques without manually checking each video. Specifically, we leverage YOLOv8 for human detection, discarding clips that do not show realistic people (eg, cartoon characters). The English conversational corpus is directly filtered by meta information provided by downloaded videos. Then, in the following data processing steps, the initial processed data will be further filtered (including pose filtering, audio processing, manual checking, etc). We will add these details in the revision.

b) After initial video filtering and scene cuts, we leverage the advanced pose estimator PyMAF-X to conduct automatic occlusion determination. In particular, we drop video clips that do not contain complete upper body parts of two speakers. Then, as stated in Lines 838-847, we conduct a manual review to further eliminate instances where two speakers are significantly occluded during interaction (eg, if the joints of one arm are obscured). Due to the strict keyword selection in raw video crawling, our dataset rarely contains two speakers standing or walking around. If there are several clips including the aforementioned postures, we will filter them out to ensure our dataset maintains unified posture representation.

W2: Some details and motivations of the proposed approach

R2: Thank you for the valuable suggestions

a) Considering the asymmetric dynamics of two speakers, we leverage separated human speech as guidance for bilateral branches to generate corresponding gestures. Moreover, we utilize the original mixed audio signal of two speakers to indicate the interaction information to ensure the synthesized posture retains rhythm with specific audio while preserving interactive coherency with the conversation partner. Actually, the original mixed audio signal can be directly obtained by addition $C_{mix} = C_{a} + C_{b}$. As suggested, we conduct the ablation study to verify the effectiveness of the original mixed audio signal $C_{mix}$. By subtracting the original mixed audio, the indicators FGD and BC present much worse performance. These results verify the mixed audio signal displays effectively enhance the interaction between two speakers. We will add this discussion in the revision.

b) As reported in Lines 862-863, considering the complex and variable positions of the two speakers of in-the-wild videos, similar to [A][B], we just model the local body joint movements (without global translation or root position) in the experiments. The lower body including the legs is fixed while visualizing due to the weak correlation with human speech. For example, it is quite challenging to model whether the two speakers are sitting with their legs crossed or standing from only audio inputs. We will add corresponding clarification in the revision.

[A] Generating holistic 3d human motion from speech, in CVPR 2023

[B] Towards Variable and Coordinated Holistic Co-Speech Motion Generation, in CVPR,2024

c) Thanks for your insightful comments. Inspired by [C][D], we introduce foot contact loss to ensure the physical reasonableness of the generated gestures. Since we only model the upper body joints in experiments, we complete the lower body joints as T pose in forward kinematic function during calculate loss. As suggested, we conduct the ablation study to verify the effectiveness of $L_{foot}$. As illustrated in the Table below, the exclusion of the $L_{foot}$ results in FGD and BC are obviously worse than the full version framework. This indicates that our foot contact loss displays a positive impact on the generated postures. We will add the corresponding discussion in the revision.

[C] Human Motion Diffusion Model, in ICLR, 2023

[D] InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions, in IJCV, 2024

W3: Implementation of counterparts and quantitative comparison

R3: Thanks for your comments. The details are listed point by point as below.

a) To ensure as fair a comparison as possible, we only leverage the pre-trained audio feature extractors of some competitors that follow their original statements. These audio feature extractors are usually utilized as universal components in the network of the gesture generation community (just like CLIP in text feature extraction). Specifically, in DiffSHEG, we follow the convention of the original work to utilize the pre-trained HuBERT[E] for audio feature extraction. In TalkSHOW, we exploit the pre-trained Wav2vec[F] to encode the audio signals following the original setting. Apart from this, the remaining components for gesture generation in DiffSHEG and TalkSHOW are all trained from scratch on the newly collected GES-Inter dataset. In this fashion, the reliability of results yielded by these methods would not be impacted. For other methods, we modify their final output layer to match the dimensions of our experimental settings. We will clarify this in the revision and release the code of all the models as soon as possible.

[C] Hubert: Self-supervised speech representation learning by masked prediction of hidden units, in TASLP, 2021.

[D] wav2vec 2.0: A framework for self-supervised learning of speech representations, in NeurIPS, 2020.

b) Thanks for your constructive suggestions. We will incorporate additional visualization demonstrations in the revision (refer to our webpage: https://anonymous.4open.science/w/Co3-F300/). The current demo videos can provide a comparison with InerX and InterGen. For example, the results generated by InterGen in comparison videos contain some unreasonable postures (eg, the neck of the left speakers, and the right hand of the right speakers). We will further highlight these areas for better demonstration in the revision.

W4: Some user study details

R4: Thanks for your comments. The details are listed point by point as below.

a) For each method, we randomly select two generated videos in the user study. Hence, each participant needs to watch 16 videos.

b) In our experiments, the length of the generated gesture sequence is 90 frames with 15 FPS. Thus, all the demo videos in the user study have the same length of 6 seconds.

c) For fair comparison in user study, the demo videos are randomly selected. As suggested, we count the motion fractions length of two speakers upon all the 16 demo videos. We adopt elbow joints as indicators to determine whether the motion occurs. Empirically, when the pose changes by more than 5 degrees between adjacent frames, we nominate the speakers who are moving now. Therefore, among 16 demo videos with 6 seconds, the average motion fraction lengths of the two speakers are 4.3 and 3.1 seconds. We will report this statistical result in the revision.

d) As suggested, we report the detailed mean and standard deviation for each method as shown in the Table below. Our method even achieves a 10% ((4.4-4.0)/4=10%) large marginal improvement over suboptimal InterGen in Naturalness. Meanwhile, our method displays a much lower standard deviation than InterX and InterGen. This indicates the much more stable performance of our method against competitors. We will add the corresponding discussion in the revision.