Q1: Clarifications for Method

1. The dimensions of F and T are provided from the beginning (L137).

2. We have re-examined our code and confirm that the L1 loss function ($L_{PQ}=||P-{\hat{P}}||_1$) was indeed used throughout our experiments. While this was a typographical error in the manuscript, it did not affect our results or conclusions.

3. Different ‘k’ in the paper.

   • $k_i$ (Eq.2, italicized, not bold) is a scalar, specifically the $i$-th entry to the codebook.
   • $\mathbf{k}$ (Eq.2 and L185, non-italicized, bold) is a vector, denoting the token indices which are quantized from token features by FSQ.
   • $\mathbf{k}_s$ (L193), where $s\in\{0,1,...,S\}$, is the token indices $\mathbf{k}$ at different discrete step $s$.

   While the same letter ‘k’ is used, the distinctions are made clear by the use of italics, bolding, and subscripts. Moreover, the consistent use of $\mathbf{k}$ across two stages ensures uniformity and avoids using unnecessary new symbols.

4. Model the relationship between 𝑁=100 tokens? We only model the relationship between adjacent joints within a sub-structure, as shown in Fig._R 1 in pdf. However, we do not further model the relationship between different sub-structures.

We will update all these typos and further polish our presentation.

Q2: Differences between Di²Pose and [1]

Admittedly, Di²Pose and [1] follow a mainstream scheme that utilizes a quantization step combined with a diffusion process, explored in various domains [2-4]. However, Di²Pose addresses a different problem with specific motivations and objectives.

• Distinct Motivation and Problem Domain: Di²Pose aims to solve occluded 3D HPE by leveraging the discrete nature of 3D poses and the strengths of diffusion models in handling uncertainty and indeterminacy. Our goal is to enhance the robustness and accuracy of 3D HPE under occlusion.

  In contrast, [1] focuses on text-to-image generation task, specifically addressing the weaknesses of Autoregressive models like DALLE.

• Distinct Mechanism: While the "Mask and replace" in [1] and our "Occlude and replace" share a similar transition matrix implementation, their purposes are fundamentally different. The strategy in [1] is designed to mitigate unidirectional bias and accumulated prediction errors. Our pose quantization step produces a token sequence representing pose substructures, enabling the "Occlude and Replace" mechanism. "Occlude" simulates the occlusion of a substructure, and "Replace" solves the uncertainty under occlusions, where a single occluded region may correspond to multiple potential 3D poses. This effectively simulates the transition from occluded to recovered, integrating occlusion impacts into the estimation process.

• Contribution to the Field: Di²Pose provides a new paradigm for tackling occluded 3D HPE. By designing a specific pose quantization step and leveraging discrete diffusion tailored for this task, we contribute a novel approach for addressing occlusions effectively.

We acknowledge the importance of distinguishing our contributions from prior work and will clarify and discuss these differences in the Method section.

Q3: Dimensions of search space

In the introduction, we mention that continuous diffusion models need a larger search space to achieve optimal generative outcomes, whereas discrete diffusion models do not. It is noteworthy that our emphasis is on the size of the search space rather than the size of the model's dimension. For clarification, please refer to the detailed analysis in “A general response to common concern” in the global response.

Q4: Extra experiments

• Replace backbone: We add experiments replacing the ViT backbone with a CNN-based backbone [6] and evaluate on the 3DPW. As shown in Table_R 2 in pdf, it shows that Di²Pose maintains its superiority even with a CNN backbone.

• Repeated experiments: We repeat experiments three times and evaluate on the 3DPW. As shown in Table_R 3 in pdf, we report the “Mean $\pm$ Std” of MPJPE and PA-MPJPE across multiple runs. It shows robustness of Di²Pose, and the mean results still achieves SOTA.

• Multiple inference results: In our paper, we focused on single inference outputs for practical 3D HPE applications. However, we acknowledge the diversity of outputs possible with diffusion models due to different initializations. Thus, we add experiments with multiple inferences from different initializations. The “Mean $\pm$ Std” is reported in Table_R 3 in pdf, which shows that Di²Pose produces relatively stable results.

  Additionally, we visualize the diverse outputs of Di²Pose from a single input image in Fig._R 4 in pdf, demonstrating the model's ability to generate varied predictions under occlusion.

• Running times: The training time and inference speed are shown in Table_R 4 in pdf.

Q5: Continuous latent representation + Quantization before decoding?

The idea is indeed very interesting. However, this specific approach is beyond the scope of our current paper. We appreciate your comments and will consider exploring this idea in the future.

Q6: Limitation

Environmental considerations: We add potential negative environmental impacts as follow. The diffusion-based model has a longer runtime compared to other CNN or GCN-based methods, causing more computational resources and energy consumption.

[1] Vector Quantized Diffusion Model for Text-to-Image Synthesis. CVPR 2022

[2] Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generation. ICCV 2022

[3] Priority-centric human motion generation in discrete latent space. ICCV 2023

[4] Layoutdm: Discrete diffusion model for controllable layout generation. CVPR 2023

[5] Deep High-Resolution Representation Learning for Human Pose Estimation. CVPR 2019

We sincerely appreciate your thorough consideration of our responses and are delighted to address all questions and concerns.

Q1:Dimensions of 𝐹 and 𝑇

We apologize for this misunderstanding. The confusion arose due to the word limit in our rebuttal, which prevented us from clearly illustrating this point. We originally intended to convey that we fully agree with your suggestion, and we will revise the original manuscript to include explicit descriptions of the dimensions of $\mathbf{F}$ and $\mathbf{T}$ (e.g., ) starting from the beginning (L137), such as $\mathbf{F} = (\mathbf{f}_1, \mathbf{f}_2, \cdots, \mathbf{f}_N)$ ($\mathbf{f}_i \in \mathbb{R}^{D}$), and $\mathbf{T}=(\mathbf{t}_1, \mathbf{t}_2, \cdots, \mathbf{t}_N)$ ($\mathbf{t}_i \in \mathbb{R}^{D}$) . This will make our presentation clearer and more intuitive.

Q2: Dependencies between sub-structures?

We agree that considering the dependencies between sub-structures is an important problem. Our approach follows the codebook learning paradigm (i.e., the encoder-decoder framework in VQ-VAE style), where an encoder encode the complete pose into discrete tokens, and a decoder decodes these tokens back to a reconstrcuted pose. Within this encoder-decoder framework, the interdependencies among sub-structures are mainly considered by the decoder, e.g., the discrete tokens (local sub-structure) can be reconstructed into the original 3D pose (global structure). Since our pose quantization step adheres to this pipeline, we have not implemented explicit strategies to further enhance the dependencies between sub-structures.

The motivation behind the design of the Local-MLP (L143-L145): Our primary goal was to address a specific limitation observed in previous work [20 in the original paper], where MLPs were used to globally model the relationships between all joints. In the context of Di²Pose, our focus is on learning tokens that effectively represent different pose sub-structures. For each token, the key is to capture the relationships between the joints within a sub-structure. This is why we designed the Local-MLP to specifically model the dependencies between adjacent joints within each sub-structure.

Q3: Reply to the comment on "Q2"

We sincerely appreciate your insights on this matter. We completely agree with your point, and we have acknowledged this in our rebuttal where we stated, “We acknowledge the importance of distinguishing our contributions from prior work and will clarify and discuss these differences in the Method section.”

Regarding your concern about the presentation of the method section, we would like to clarify that when writing this paper, we recognized that discrete diffusion models are a popular and widely-used framework. Our "Occlude and Replace" mechanism was specifically designed for occluded 3D HPE, and we chose this name to reflect its targeted application. However, we did not sufficiently discuss previous discrete diffusion model techniques in our original manuscript. This oversight could indeed cause some confusion, and we sincerely apologize for that.

To address the issues mentioned above, we will provide several revisions as follows:

1. In the Related Work section, we will include a discussion of the current development and applications of discrete diffusion models (such as [1-4]), highlighting both the connections and differences between our work and prior research.
2. In the Method section, to avoid potential confusion, we will clearly state that our "Occlude and Replace" transition matrix is inspired by prior work [1]. We will also emphasize the distinctions and specific contributions of our approach in comparison.

Once again, we are grateful for your constructive comments, which have greatly contributed to improving the clarity and quality of our paper. If you have any further questions or concerns, please don't hesitate to discuss them with us.

[1] Vector Quantized Diffusion Model for Text-to-Image Synthesis. CVPR 2022

[2] Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generation. ICCV 2022

[3] Priority-centric human motion generation in discrete latent space. ICCV 2023

[4] Layoutdm: Discrete diffusion model for controllable layout generation. CVPR 2023

Thank you for the detailed comments. We are willing to address all your questions.

Q1: Effectiveness of pose quantization for addressing data dependency

The pose quantization step is designed to convert a 3D pose into multiple quantized tokens, which can be modeled in the latent space by the discrete diffusion model. The keypoint of addressing data dependency is that leveraging discrete diffusion process model discrete tokens rather than exploiting continuous diffusion model diffuses 3D pose in continuous space. For clarification, we provide the detailed analyse in the global response. Please refer to “A general response to common concern”.

Q2: Clarification for the target of our paper

First, we want to emphasize that the primary goal of our Di²Pose framework is to address the occlusion problem in 3D HPE. The framework consists of two stages, both designed to tackle occlusions effectively:

• Pose Quantization Step transforms a 3D pose into multiple quantized tokens. Our Local-MLP and FSQ mechanisms are introduced to learn a better codebook, ensuring that each quantized token represents an effective substructure of the pose. This step is crucial for the subsequent discrete diffusion process, where the Occlude and Replace strategy is applied. By encoding poses into meaningful tokens, we facilitate the handling of occlusions using the discrete diffusion model.
• During Discrete Diffusion Process, we use the Occlude and Replace strategy to address occlusions. Each Occ token represents an occluded sub-structure, while the replacement mechanism mitigates uncertainty. This process relies on the quantized tokens learned in Stage 1, ensuring that the model can simulate occlusions and accurately recover the complete pose.

Regarding the data dependency issue, our discussion aims to highlight an additional bonus of using the discrete diffusion model over continuous ones. This is beneficial but secondary to our main objective of solving the occlusion problem.

Q3: Clarification for the definitions of "Continuous" and "Discrete"

Discrete Representation of 3D Poses: In this paper, we focus on the discrete representation of 3D poses. As described in the Introduction, existing 3D HPE methods tend to represent poses using coordinate vectors or heatmap embeddings. These methods are both considered as discrete representations of 3D poses. Our pose quantization step converts a 3D pose into multiple discrete tokens, which still belong to discrete representations.

Continuous and Discrete Diffusion Models: For the continuous and discrete diffusion models, we distinguish them by their initialization patterns at the beginning of the reverse process:

• Continuous Diffusion Model: Continuous diffusion models initialize the 3D pose from random noise, where each joint can be sampled from the continuous 3D space. This aligns with the general understanding of continuous diffusion processes, such as those modeled by continuous SDEs or ODEs.
• Discrete Diffusion Model: The discrete diffusion model, on the other hand, initializes the 3D pose from limited quantized tokens. The range of each token index depends on the size of the codebook, which is a finite number.

Q4: Clear explanation of the Local-MLP

We have redrawed Figure 3 from the original paper with more detailed structures, as shown in Fig._R 1 in pdf.

In Local-MLP, the key component is the JS-Block, which captures local interactions among X joints. Both Linear Proj. 1 and Linear Proj. 2 are implemented as Conv1d layers with a stride and padding of 1, which allows for information fusion along the joint dimension.

1. Linear Proj. 1 integrates features within each joint in $\mathbf{P_{emb}^\top} \in \mathbb{R}^{D \times J}$.
2. Joint Shift Operation shifts features of adjacent joints into the same channel (Figure b(1) → Figure b(2)). The central part (green box) remains stationary, while the adjacent parts (blue and orange boxes) shift in opposite directions.
3. We extract the section indicated by the red dashed box (Figure b(2) → Figure b(3)), resulting in each green dashed box containing features of different adjacent joints.
4. Linear Proj. 2 integrates features within each green dashed box, combining adjacent joint features (Figure b(3) → Figure b(4)).
5. Channel MLP further integrates information across each channel by expanding the dimension D by a factor of 4 and then mapping it back to D dimensions.

Q5: Visualization of “Replace” mechanism

We visualize the relevant results in Fig._R 2 in pdf. First, Di²Pose uses the pose quantization step to represent a 3D pose as multiple discrete tokens, where each number represents an index of the codebook. To demonstrate the effect of the "Replace" operation in the discrete diffusion process, we replace a token at a specific position with other available tokens, while keeping the tokens at other positions unchanged. The resulting token sequence is then decoded by the Pose decoder to obtain a 3D pose. It can be seen that replacing a certain token with other available tokens consistently changes the same sub-structure, which is circled in red.

Q6: Ablation studies on the Local-MLP

We have addressed this issue in the Ablation Study of our paper.

In Table 3(a), we present the ablation study results for different local joint numbers X in the Joint Shift operations. It is noteworthy that the case where X=1 corresponds to not using the joint shift operation at all, which is regarded as a vanilla VQ-VAE. The results indicate that our Local-MLP, which incorporates the joint shift operation, provides better pose representation performance.

Q7: Potential negative societal impact

In Appendix F.2, we have discussed the potential negative societal impacts of our work, including the risk of malicious applications such as illegal surveillance and video synthesis.

Thank you for the detailed comments. We are willing to address all your questions.

Q1: Extended experiments on more complex datasets

Clarification. In 3D skeleton-based HPE task, the 17-joint annotation of the Human3.6M is a widely-used and standard benchmark. Most mainstream methods validate their methods on this dataset. For fair comparisons, we also follwed this setting in our experiments. As described in [3], this limited number of joints helps discard the smallest links associated to details for the hands and feet, going as far down the kinematic chain to only reach the wrist and the ankle joints." This has led subsequent works to predominantly use the 17 joints for experiments rather than the 32 joints.

Extra results on a new dataset: We acknowledge the reviewer's concern that the 17-joint Human3.6M dataset is relatively simple. Thus, we utilize a recent H3WB dataset [4], an extended annotation of Human3.6M with 133 keypoints covering the body, hands, and face, significantly increasing complexity. The experimental results are shown in Table_R 1 in pdf. Di²Pose still achieves comparable results with SOTAs in terms of “All” and “Body”, but slightly underperforms on the "Face" and "Hand". This may be because the joints within the “face" and "Hand" are densely distributed and highly correlated, unlike the torso. Separate pose quantization for "Face," "Hand," and "Body" might be needed. Due to the limited rebuttal time, we did not have the opportunity to try such idea.

Di²Pose with SMPL, SMPL-X, and STAR and add experiments on new mesh-based datasets? It is important to clarify the differences between skeleton-based (3D coordinates) and mesh-based (e.g., SMPL) 3D HPE tasks. Di²Pose is specifically designed for skeleton-based HPE. SMPL models, on the other hand, require estimating both pose and shape parameters, necessitating a redesign of existing Di²Pose. This level of complexity is beyond the scope of the current paper. However, we are very interested in adapting Di²Pose for mesh-based models. We will try to explore this direction in future work.

Q2: Better visualizations

Render the 3D skeleton into the image? In 3D skeleton-based HPE, projecting the 3D skeleton onto a 2D plane can lead to overlapping joints and visual confusion, making it difficult to assess estimation quality. This is different from mesh-based methods, where shape parameters allow for more accurate visual alignment with human silhouettes. Consequently, skeleton-based methods usually visualize results directly in 3D space [1,2].

Comparison with GT in the same 3D space. To solve the reviewer's concern about the difficulty in discerning differences in human poses. We visualize the predictions and ground truth in the same 3D space, as shown in Fig._R 5 in pdf.

Visualization on multiple views: We add visualization results on multiple viewpoints, as shown in Fig._R 3 in pdf.

We will add these visualizations in the final version.

Q3: Clarification for the question about “VQ-VAE+Diffusion”

Although Di²Pose follows the “codebook learning + discrete diffusion process” pipeline, it is not as simple as “VQ-VAE + DiffPose”.

• Codebook learning: The primary challenge in the first stage is to learn effective and representative tokens, with each representing a sub-structure. Simply applying vanilla VQ-VAE does not capture the relationships between adjacent joints within a sub-structure and is prone to causing codebook collapse. These issues would render the subsequent discrete diffusion process ineffective due to invalid tokens, preventing occlusion simulation and recovery. Specifically, we design the Local-MLP and exploit FSQ to ensure successful sub-structure representations.
• Discrete Diffusion Process: While our second stage indeed involves a diffusion process, it differs significantly from DiffPose, which employs a continuous diffusion model. Di²Pose operates on a discrete token sequence, while DiffPose diffuses 3D poses in continuous space. Moreover, Our process uses a transition matrix for state transitions, while DiffPose employs a Gaussian Mixture Model to model the uncertainty distribution.

Importantly, Di²Pose seamlessly integrates both stages to address occluded 3D HPE. Quantized tokens from the first stage are processed by the discrete diffusion model to simulate occlusions and recover the full pose. This mechanism effectively incorporates occlusions into the HPE process, offering valuable insights for tackling the occluded 3D HPE task.

Comparsion with [5]: Regarding the comment "In terms of design, it is not even as clever as [1]," we would like to highlight the following points:

• Thanks for bring this paper to our attention. Firstly, we want to mention that this paper [5] was released to arXiv on May 30, 2024, which is after the NeurIPS submission deadline (May 22, 2024).
• Both Di²Pose and [5] utilize a pipeline involving codebook learning and a diffusion process. However, [5] focuses on body-motion generation, a task involving sequential signals that require inter-frame information. In contrast, Di²Pose addresses occluded 3D HPE, targeting single-frame setting. The differences in the problems being solved naturally lead to different design choices and frameworks, making a direct comparison of the methods' cleverness inappropriate. We will add these discussions and comparisons to our paper.

[1] Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregation. In ICCV, 2023

[2] GLA-GCN: Global-local Adaptive Graph Convolutional Network for 3D Human Pose Estimation from Monocular Video. In ICCV, 2023

[3] Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. In TPAMI, 2013

[4] H3WB: Human3.6M 3D WholeBody Dataset and Benchmark. In ICCV. 2023

[5] Stratified Avatar Generation from Sparse Observations. In CVPR. 2024

Thank you for the detailed comments. We are willing to address all your questions.

Q1: Detailed explanations and insights about the two main parts: pose quantization and discrete diffusion.

The proposed Di²Pose is a two-stage framework designed to address the challenges of occluded 3D human pose estimation (HPE).

• In the first stage, we train a pose quantization step that transforms a 3D pose $\mathbf{P}$ into $N$ discrete tokens $\mathbf{k}$. Each token represents a sub-structure of the whole pose. This pose quantization step leverages the discrete nature of 3D poses and represents them as quantized tokens by capturing the local interactions between joints. This quantization step is crucial for the subsequent discrete diffusion process, as it allows the discrete diffusion model to simulate occlusions of specific sub-structures of the 3D pose. The quantized tokens $\mathbf{k}$ serve as a vital link that binds pose quantization and the discrete diffusion process together, ensuring coherent interaction between the two stages.

• In the second stage, we model tokens in the discrete space using a discrete diffusion process, which consists of a forward and a reverse process.

  Forward Process: During the forward process, each token of $\mathbf{k}$ is probabilistically occluded with an Occ. token or replaced with another available token. The occluded token represents the occlusion of the corresponding sub-structure of the 3D pose. The token replacement mechanism is designed to enhance the diversity of potential sub-structures, reflecting the indeterminacy in occluded parts.

  Reverse Process: In the reverse process, the pose tokens are initially occluded or randomly initialized. The denoising diffusion process estimates the probability density of pose tokens step-by-step based on the input 2D image until the tokens are completely reconstructed. Each step leverages contextual information from all tokens of the whole pose as predicted in the previous step, facilitating the estimation of a new probability density distribution and the prediction of the current step’s tokens. This sequential approach ensures a detailed and accurate reconstruction of 3D poses from occluded scenes.

This two-stage framework allows Di²Pose to effectively handle occlusions by breaking down the pose into meaningful sub-structures and reconstructing them through a probabilistic diffusion process. The integration of pose quantization with the discrete diffusion process significantly contributes to the observed improvements in handling occluded poses.

Q2: Comparison with related works

We compare our Di²Pose with the mentioned related works as follows.

• Pose Relation Transformer (PORT) [1]: PORT is designed to address the HPE problem, mitigating the effect of occlusions inspired by the sentence completion task in NLP. The similarity between PORT and our Di²Pose is that both methods recognize the importance of capturing local context between adjacent joints. PORT leverages an attention mechanism to aggregate adjacent joint features, while Di²Pose proposes a Local-MLP to capture local relationships within a sub-structure of the 3D pose. Moreover, PORT introduces a Masked Joint Modeling (MJM) approach to reconstruct randomly masked joints, which helps refine occlusions.

  However, MJM randomly selects joint indices for masking and trains PORT to reconstruct the masked joints, explicitly simulating occlusions and treating the masked joints as independent. In contrast, Di²Pose uses a discrete diffusion process to implicitly model occlusion within the latent space, enhancing its understanding of how occlusions affect human poses. Additionally, we found that PORT is specifically designed for 2D HPE, which differs from our focus on 3D HPE. Thus, we cannot directly compare these two methods by experiments.

• InfoGCN [2]: InfoGCN is proposed to solve the human skeleton-based action recognition task. This novel method focuses on embedding physical constraints and intention information into the latent representations of human actions. The similarity between InfoGCN and Di²Pose is that both methods aim to learn informative latent representations from raw data. InfoGCN introduces a novel learning objective based on the information bottleneck theory, which aims to learn an efficiently compressed latent representation of an action. However, Di²Pose proposes a pose quantization step, which leverages VQ-VAE to convert a 3D pose into multiple discrete latent tokens, compressing information in different ways. Moreover, InfoGCN focuses on the human skeleton-based action recognition task, which is different from 3D HPE. Therefore, we also cannot directly compare our method with InforGCN by experiments.

Although the above works focus on different tasks, we recognize the importance of discussing these methods to provide a comprehensive context for our work. To address this, we will cite these two works in the Related Work section and add these discussions about their contributions and differences with our approach. Additionally, we will further investigate other relevant works to enhance the completeness and context of our study.

Q3: Limitations

We have illustrated the limitations of our method in detail in the Appendix Sec.F, including failure cases, physically reasonable outcomes, and frame-based limitations.

[1] Pose Relation Transformer Refine Occlusions for Human Pose Estimation. In ICRA, 2023

[2] InfoGCN: Representation Learning for Human Skeleton-based Action Recognition. In CVPR, 2022