Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation

We thank the reviewer for the insightful comments and feedback. We have addressed each comment carefully and provided a point-by-point response below.

C#1: although this method shows good results…it has performance drop on image quality and text alignment.

We thank the reviewer for this insightful comment. We followed the common metrics widely adopted in the current generative AI field, however, despite the rapid advancements in generative AI, existing metrics have not evolved to provide a more accurate evaluation [ref1]. We summarize some of the issues below.

1. CLIP score: it relies on cosine similarity between the model's semantic understanding and the given text, which may not align with pose assessments or the relevance of generated images. Additionally, this score is sensitive to the arrangement and composition of elements in images; even minor changes can result in significant fluctuations in the score, which may not accurately reflect the overall generative quality. [ref2] suggests a Dino-based score.
2. FID: it estimates the distance between a distribution of Inception-v3 features of real images and those of images generated by the generative models. However, Inception's poor representation of the rich and varied content generated by modern text-to-image models incorrect normality assumptions and poor sample complexity [ref1]. Thus, the FID score does not account for semantic correctness or content relevance—specifically pose—in relation to the specified text or conditions.

Relying solely on FID and CLIP scores does not comprehensively assess the generative model. Therefore, in light of your comment and similar concerns from the other Reviewers, we further evaluated our method with a new state-of-the-art metric CMMD [ref1], which is based on richer CLIP embeddings and the maximum mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased estimator that does not make any assumptions on the probability distribution of the embeddings, offering a more robust and reliable assessment of image quality. As shown in the table below, our method achieves better CMMD value compared to HumanSD, demonstrating comparably high image quality.

Thanks to this comment, we add a subsection in Appendix A.4 listing the current challenges associated with pose evaluation via these metrics and additional results using CMMD in our final manuscript.

[ref1] Jayasumana et al. Rethinking fid: Towards a better evaluation metric for image generation. In CVPR 2024.

[ref2] Ruiz, et al. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR 2023.

C#2: compared to controlnet and other…no significant contribution both in training recipe and architecture.

We respectfully hold a different perspective on the assessment of our contributions. Stable-Pose is a light-weight adapter that can be easily integrated into any pre-trained T2I diffusion models to effectively enhance pose control, even in challenging conditions. It achieves so by synergistically capturing both local pose details with a coarse-to-fine PMSA ViT, and global semantics via a pose encoder. Its novelty and efficiency are confirmed by other reviewers (pWbK), and Reviewer RKmi noted ‘the robustness, generalizability, and collaborative potential of the proposed method’.

To further validate its generalizability, we conducted additional experiments by applying Stable-Pose on top of a pre-trained HumanSD model. As shown in the table below, the inclusion of Stable-Pose considerably improved the baseline HumanSD by over 10% in AP and 12% in KID, highlighting its high generalizability and effectiveness in enhancing both pose control and image quality.

Q#1: It is interesting to see when pose-mask guidance strength is larger than 7, the AP for pose actually got dropped.

We share the reviewer’s perspective on the slight decrease in AP for higher values of the pose-mask guidance strength (α), which is less than 1%. Increasing α in our proposed loss means putting more attention on the foreground pose regions. However, if α is too large, it forces the model to learn irrelevant texture information like clothing, which negatively impacts training and slightly decreases AP. Despite this, Stable-Pose still outperforms others across an α range of 1-10. Notably, increasing α has a significant impact on FID, worsening it from 11.0 at α=5 to 13.0 at α=10. This indicates that focusing solely on the pose regions may decrease the quality of generated content in non-pose regions. Thus there exists a slight trade-off in selecting α to maintain both high pose accuracy and image quality, in which a value around 5 to 6 turns out to be optimal. Thanks for this insightful comment, we will include it in our final manuscript.

Q#2: for all the examples shown in the paper, are they using the same seed for each of the methods? (seed is important)

Thank you for pointing out this very important aspect in current generative AI. Yes, we indeed always kept the same seed list for all techniques, including ours, during both training and inference time, to ensure fair comparison and reproducibility. We appreciate your attention to this matter and will add this information to our final manuscript to make it more clear. We will also make our code available on GitHub upon acceptance to support the reproducibility of our results.

Once more, we appreciate the Reviewer's insightful suggestions and hope our detailed responses have addressed all concerns. If any further questions arise, we invite the Reviewer to discuss them during the August 7-13 discussion period. We value your comments and look forward to further engagement.

We express our sincere appreciation to the reviewer for providing valuable comments and suggestions. We have addressed the comments point-by-point, as outlined below.

C#1: (No consideration of different persons & parts)

Thanks for the insightful discussion. The pose information is encoded in two ways in Stable-Pose:

1. The pose encoder (β): It encodes the RGB pose skeleton input into high-level contextual information. Through color encoding, β captures semantic information such as the differentiation of limbs and the number of people.
2. PMSA-ViT: the patchified latent features are processed through several coarse-to-fine pose-masked self-attentive ViTs, which focus on accurately modeling the poses shown in the pose skeleton and the interaction between different pose parts.

Thus, through these two modules, Stable-Pose synergistically integrates both local and global contextual information (including the number of people and body parts) from the pose skeleton naturally.

To further validate the design, we specifically add an ablation study by removing either β or PMSA-ViT, retaining only one type of encoding. The results in the table below show that using only PMSA-ViT yields an AP of 36.48, which is expected due to the absence of color-coding information for distinguishing body parts. While using β alone increases the AP to 45.03. However, the most significant improvement is observed when integrating both local and global information encoding into the Stable-Pose architecture, achieving the highest AP of 48.88.

Thanks to this comment, we will add these results and discussion to our final manuscript and we appreciate this valuable comment.

C#2: (Inferior PCE)

While PCE is useful for evaluating the number of humans in generated images, it has limitations such as the sensitivity to outliers. Given that Stable-Pose is designed to enhance SD's ability to generate accurate content in pose-mask regions, a pre-trained SD may still produce human-like figures in the background. For example, in the first row of Fig. R1 of our attached PDF, Stable-Pose generates an additional half-shaped person in the background on the right side. Such cases affect the overall PCE measurement. Therefore, foreground counting should be prioritized over background counting. However, PCE treats both equally.

Despite this, Stable-Pose still reaches the best PCE score among adapter-based methods (e.g., ControlNet) in LAION-Human. In addition, applying Stable-Pose on top of a pre-trained HumanSD model (a fine-tuning-based method) considerably improves the PCE score compared to using Stable-Pose with SD. While it results in a marginal error of only 0.05 more/less people per image compared to the HumanSD method, it substantially improves the pose accuracy in AP by 10%.

Thanks to this comment, we further discuss this aspect of PCE in Appendix A.4 and would love to hear your opinion about it.

C#3: (Add module architecture ablation studies) & C#4: (Add FPS results)

Thanks for your advice in improving the paper. In fact we have addressed these concerns in our supplementary material as detailed below:

1. In Table A.6 and A.7, we demonstrated ablation studies of the module structure, including the number of the PMSA-ViT blocks and different patch sizes in the ViT, respectively.
2. In Table A.1, we reported the inference time (FPS) and training duration (in hours) of our method compared to the other state-of-the-art approaches.

Due to the page limit we could not include these results in the main manuscript, but only in the appendix. That said, we plan to move some parts mentioned above to the main draft, as there is an additional page available for the final submission.

C#5: (Add failures)

Thank you for your insightful comment. We attached a PDF file with some failure cases in Fig. R1, such as generating the wrong number of people in very crowded scenes. Stable-Pose enhances SD's accuracy in pose-mask regions, but a pre-trained SD may still produce human-like figures in the background. For example, in the first row of Fig. R1, Stable-Pose generates an extra half-shaped person on the very right side. We will add a dedicated subsection (A.9) in the Appendix to address this issue.

C#6: (Opinions on masked generation)

We highly appreciate this insightful comment. In our implementation, we indeed incorporate the pose masks during inference by default to enhance control. To further validate their effectiveness, we additionally conducted experiments with removing the pose masks during inference. As shown in the table below, this led to approximately a 3% drop in AP. This could be due to two main reasons:

1. The pose masks provided additional guidance, thus enhancing control.
2. The inclusion of pose masks maintains consistency between the model's behavior during training and inference.

Thus, including pose masks benefits pose control in the generation. Thanks to this comment, we will include this result in our final manuscript.

We highly appreciate the valuable suggestions and input from the Reviewer. We hope our responses address the reviewer’s concerns about Stable-Pose, with details covered in the manuscript and appendix. We look forward to any further questions during the reviewer-author discussion period from August 7-13.

We highly appreciate the Reviewer for the valuable comments and suggestions. We have carefully addressed each comment, providing a point-by-point response below.

C#1: The coarse-to-fine approach does not appear promising…The improvement from constant variation is negligible.

Thanks for this comment. As the coarse-to-fine approach is one part of the PMSA design, the strength of PMSA-ViT lies in its synergistic integration of the masking algorithm and kernel designs. As shown in Table 4 of our manuscript, using coarse-to-fine pose masks in PMSA improves the baseline results by approximately 7% in AP. Table 5 also highlights the significance of the order of kernels and how it can significantly impact the results. We believe that this coarse-to-fine approach can also be effectively generalized and incorporated into the loss definition of future diffusion models. Thank you again for the feedback and we will include the discussion in this regard in our final manuscript.

C#2: Limitations and failure cases are not adequately discussed.

Thank you for the comment. We have attached a PDF file that includes some failure cases of our method, shown in Fig. R1, such as the incorrect number of generated humans in very crowded scenes. Given that Stable-Pose is designed to enhance SD's ability to generate accurate content in pose-mask regions, a pre-trained SD may still produce human-like figures in the background. For example, in the first row of Fig. R1 of our attached PDF, Stable-Pose generates an additional half-shaped person in the background on the right side. We will also include a dedicated subsection (A.9) in the Appendix to address this issue with the visualization.

Additionally, in Sec. 5, we discussed some limitations of Stable-Pose, including its slightly longer inference time due to self-attention (illustrated in the appendix Table A.1), and the need for evaluation under other conditions, such as edge maps. Due to the page limit we could not elaborate it extensively in the main manuscript, however, we plan to include deeper discussion on the limitations and failure cases in the final manuscript as there is an additional page available for the improvement of the final submission.

Q#1: The claim…stating that pose-guided image generation requires paired human data during training, is not entirely accurate. Previous works, such as [1], [2], [3] etc., do not require paired information… It would be beneficial to revise this section to reflect these nuances and include the relevant citations.

Thank you for providing the insightful information and feedback. We agree with the Reviewer that the requirement of paired information is not necessarily a disadvantage in some applications. In our work, we would like to differentiate from this line of methods by focusing on a pose-guided T2I model that offers higher flexibility in training and content generation without constraints on the appearance. We also acknowledge the importance of discussing unsupervised training methods with appearance constraints, as highlighted by the Reviewer. We will include the relevant citations and revise our final manuscript accordingly. Thank you again for your valuable input.

Q#2: How do the results appear for extreme poses, such as bending the upper body backward?

Thank you for this insightful suggestion. We have illustrated some extreme pose cases in Fig. R2 in our attached PDF file, such as bending the upper body backward in some dancing poses. As shown in the figures, Stable-Pose still maintains very high pose accuracy on generated images under these challenging scenarios, whereas ControlNet fails to depict the correct pose and body structures. We will also add a dedicated subsection (A.10) in the Appendix to address this concern.

We deeply appreciate every comment and suggestion of the Reviewer. We hope we have satisfactorily addressed every point. Your input is valued, and we invite the reviewer to raise any additional questions during the upcoming discussion period from August 7-13.

We appreciate the Reviewer’s comments and insightful suggestions. Below, we provide a point-by-point response to each comment.

C#1: The proposed method is more like an incremental work…instead of a new innovative pipeline.

We respectfully hold a different perspective on the novelty assessment. Stable-Pose is a light-weight adapter that can be easily integrated into any pre-trained T2I diffusion models to effectively enhance pose control, even in challenging conditions. It achieves so by synergistically capturing both local pose details with a coarse-to-fine PMSA ViT, and global semantics via a pose encoder. Its novelty and efficiency are confirmed by other reviewers (pWbK), and Reviewer RKmi noted ‘the robustness, generalizability, and collaborative potential of the proposed method’.

To further validate its generalizability, we conducted additional experiments by applying Stable-Pose on top of a pre-trained HumanSD model. As shown in the table below, the inclusion of Stable-Pose considerably improved the baseline HumanSD by over 10% in AP and 12% in KID, highlighting its high generalizability and effectiveness in enhancing both pose control and image quality.

C#2: Although the generated images follow the pose guidance better, the image quality (FID) and T2I Alignment (CLIP Score) are worse…

& Q#1: Could the authors discuss more about why image quality (FID) and T2I Alignment (CLIP Score) are worse than those SOTA methods?

We thank the reviewer for this insightful comment. We followed the common metrics widely adopted in the current generative AI field, however, despite the rapid advancements in generative AI, existing metrics have not evolved to provide a more accurate evaluation [ref1]. We summarize some of the issues below.

1. CLIP score: it relies on cosine similarity between the model's semantic understanding and the given text, which may not align with pose assessments or the relevance of generated images. Additionally, this score is sensitive to the arrangement and composition of elements in images; even minor changes can result in significant fluctuations in the score, which may not accurately reflect the overall generative quality. [ref2] suggests a Dino-based score.
2. FID: it estimates the distance between a distribution of Inception-v3 features of real images and those of images generated by the generative models. However, Inception's poor representation of the rich and varied content generated by modern text-to-image models incorrect normality assumptions and poor sample complexity [ref1]. Thus, the FID score does not account for semantic correctness or content relevance—specifically pose—in relation to the specified text or conditions.

Relying solely on FID and CLIP scores does not provide a comprehensive assessment of the generative model. Therefore, in light of your comment and similar concerns from the other Reviewers, we further evaluated our method with a new state-of-the-art metric CMMD [ref1], which is based on richer CLIP embeddings and the maximum mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased estimator that does not make any assumptions on the probability distribution of the embeddings, offering a more robust and reliable assessment of image quality. As shown in the table below, our method achieves better CMMD value compared to HumanSD, demonstrating comparably high image quality.

Thanks to this comment, we add a subsection in Appendix A.4 listing the current challenges associated with pose assessment via these metrics and the additional results using CMMD in our final manuscript.

[ref1] Jayasumana et al. "Rethinking fid: Towards a better evaluation metric for image generation." In CVPR 2024.

[ref2] Ruiz, et al. "Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR 2023.

Q#2: Why the proposed methods are better at distinguish the left and right of the pose condition?

Thanks for this comment. Indeed, the PMSA-ViT focuses on exploring relationships between human parts using Transformers, capturing local pose details without differentiating body parts. Yet simultaneously, Stable-Pose includes a pose encoder (β) to encode color-coded pose skeleton information, capturing semantic details like limb positions and the number of people. This enables Stable-Pose to naturally and flexibly enhance pose accuracy by synergistically integrating both local pose details and global human conditions.

To further validate the design, we specifically add an ablation study by removing either β or PMSA-ViT, retaining only one type of encoding. The results in the table below show that using only PMSA-ViT yields an AP of 36.48, which is expected due to the absence of color-coding information for distinguishing body parts. While using β alone increases the AP to 45.03. However, the most significant improvement is observed when integrating both local and global information encoding into the Stable-Pose architecture, achieving the highest AP of 48.88.

Thanks to this comment, we add these results and discussion to our final manuscript and we appreciate this valuable discussion.

We thank you once again for every comment. Our responses aim to address all concerns. If there are further questions or suggestions, we invite the Reviewer to discuss them during the August 7-13 reviewer-author period. We value your input and look forward to productive discussions.

I have carefully reviewed the other reviewers' evaluations and the full rebuttal from the authors. As I mentioned earlier, the innovative aspect of the paper lies in how it encodes pose information into the diffusion model. The results, including those presented in the rebuttal, demonstrate that their proposed method outperforms the state-of-the-art in adhering to that control signal. I agree with the other reviewers that this approach may cause a marginal degradation in performance in terms of FID evaluation. However, I do not believe that having an FID value approximately 1 point higher than other methods is a significant issue that should penalize the proposed method. In fact, I doubt this difference is even perceptible visually, and there could be several factors contributing to it. Overall, I believe the module proposed by the authors provides effective pose control, which requires a certain level of innovation, and there are sufficient experiments to support its efficacy. Therefore, I would support the acceptance of this paper. But of course, I am open to discussion if there are different opinions.