DisCo: Disentangled Control for Realistic Human Dance Generation

Weakness:

• Ethics issues. Ethics reviewers are required to review whether double-blind regulations have been violated.
  • In code config/__init__.py, I found there exist Chinese characters and some codes like MSRA PC Node. As we know, MSRA means Microsoft Research Asia. This would make the reviewer to infer the authors’ nationality and possibly Microsoft affiliation.
  • The authors did not discuss possible ethical issues with this study. Including issues such as race and gender bias.
• About Citation
  • It seems that authors are not clear on the difference between \cite and \citep. For example, “… ControlNet branch with the pre-trained U-Net weight following (Zhang & Agrawala, 2023).” should be“… ControlNet branch with the pre-trained U-Net weight following Zhang & Agrawala (2023).” There are more than one similar problem. During rebuttal, please list all of similar issues and revise them, I will check them one by one.
  • I would like to point out serious issues with the wrong citation on Grounded-SAM. The first implementation of Grounded-SAM is the https://github.com/IDEA-Research/Grounded-Segment-Anything, which has been accepted as an ICCV demo. Please cite it correctly. Besides, if authors used it, please cite Grounding DINO.
  • Instructpix2pix was accepted CVPR, not an arxiv paper. Please cite it correctly. There is more than one similar problem. During rebuttal, please list all similar issues and revise them, I will check them one by one.
  • Missing comma in the equation of Section 3.1.
  • When summarizing contributions, To address this problem, ... should be To address these problems, ....
• Reproduction
  • When I try to run the code. I found it missing README and I cannot run the code correctly. I am not sure whether the codes are appropriate.
  • Missing video demo. This is very essential for me to check the results. And the video comparison with baselines is needed.
• Technical Design
  • It is hard for readers to check the technical designs in Figure 2. The CLIP feature is fed into TM module (ResBlock and TransBlock). How do the lines without arrows connect to the module? How do the lines with arrows connect to the module?
  • Do the features output by each layer need to be processed by BG ControlNet and Pose ControlNet and then added to the middle layer? If there are $X$ down layers, in the UNet middle layer, will features be added about $2X$ times?
• Why not compare with Follow-your-pose?

My main concern comes from ethics issues and writing issues. There is something confusing that makes readers hard to follow this work. I spent about 8 hours checking this submission and tried to run the codes (but failed). I plan to rate it as 4 but the system only supports 3 or 5. Therefore, I rate it as 3 now. After the rebuttal and reviewer discussions, I will revise my rating to provide a clear rating of accept or reject.

1. The technical contributions of this paper are limited. The proposed DISCO is mainly a combination of Image-variation Stable Diffusion, Background ControlNet, and Human pose ControlNet.
2. The authors only provide images or frames in paper, but not videos. Therefor, the temporal consistency of synthesized videos is unconvincing.

I am not an expertise in video synthesis, here are my feelings about this work.

First of all, the authors didn't provide the video demonstration for their work, which is supposed to largely decrease the validity of this work, since the whole highlight is about dance generation.

1. Though the title is about "dance generation", I feel the emphysis of this work, including technical design, is more on pose-based image transfer. There is little thing about "sequence" modeling for dance.
2. I won't say the generated dances are realistic (as claimed in title). There are many temporal inconsistency and jittering, though I acknowledge the single image-based editting is realistic. I guess for better video modeling, we should pay more attention on the temporal consistency. (I found the video somewhere else.)
3. Upper-body video generation is kind of limited. Is there full-body dance dataset available?
4. Since in diffusion model, generating each image requires hundreds of steps, I am curious how long does it take to generate a dance video. Will that be a limitation of this work?
5. For video generation, it's necessary to have comparisons with baselines.
6. Some discussion about limitations are desirable.

1. Since the proposed task of the paper is generative, it requires a human evaluation of the perceived plausibility and overall generation quality. Quantitative metrics do not capture these aspects, and they are commonly covered through various types of user studies in related work. Without such an evaluation, it is hard to assess the impact of the proposed method fully.

2. Have the authors explored or encountered any incompatibilities between the reference images and the target poses? For example, any significant differences in the relative body shapes between the reference and the target, or backgrounds that may not realistically match the target poses? A discussion of such scenarios, or other potential limitations, is important to understand the full scope of the proposed method.