SKDream: Controllable Multi-view and 3D Generation with Arbitrary Skeletons

1. I appreciate that the authors have created baselines based on SDEdit and SDEdit+COSAG. However, since the method illustrated in the pipeline shows that the skeleton still needs to be projected into 2D images, I wonder what if we just apply the 2D skeleton + text-conditioned image generation from ControlNet generation and then feed it into any single-view image-to-3d model, e.g. zero123? How much would performance gain be from the proposed multi-view images input to LRM?

2. Could the author provide a comprehensive experiment and discussion on comparison with SDS-based method e.g. DreamGaussian? For example, taking a skeleton + text conditioned image as input to 3D generation?

3. A user study would be helpful for judging the comparison of motion control and appearance control quality.

1. There exists grammar mistakes in the paper, please polish the writing.

2. In the skeletal extraction section, the author is missing a lot of details, me personally is quite curious about how to build the graph from curves. Maybe add a little more algorithmic details in the appendix part will be better.

3. The appearance refinement is not introduced clearly. I'm still wondering why we need to maintain a learnable texture map u and what's the motivation of this?

4. The quantitative experiments seems a little inadequate, can you add some more baselines for comparison?

1. The current method is based on a highly constrained environment, thus the major concern is its practical usability, in particular:

(i) Since the synthetic skeleton is created by MCF and graph partition, the resulting skeleton does not have a clear semantic meaning at each node, which is contradict to practical pipelines where each joints can be clearly defined and followed consistently. Therefore it's unclear how should the skeleton conditions be created in practice. (ii) The work lacks demonstration of results from arbitrary manual created skeleton inputs, where skeletons with the same curve but varying node positions often exist. Experiments with manually created skeletons that have the same overall structure but different node placements and should be included and quantified to show how sensitive the current method is to these variations, which would provide valuable insights into the robustness of the method. (iii) Specifying 3D skeleton conditions is more challenging than 2D skeletons with optionally relative depth, the authors should include a comparison with baselines that use 2D skeleton inputs, and leaving the depth ambiguity to be resolved by simply learning from the distribution of the data.

2. The current methods present limited insights in learning from skeleton correlations. Most modules are a simple adaption of the attention-based methods, while lacking in-depth analysis of its effects. The author should include more ablation studies on the effects attention-based modules, e.g. visualizations of the learned correlations, or showing the results after removing these modules.

3. The current evaluation also heavily biased on the mesh skeletonization, while containing limited evaluations and ablations in the generalization results. As above, the author may test on out-of-distribution skeleton types or evaluations on real-world datasets. This would help assess the practical applicability of the method beyond the synthetic dataset

Overall, I find this paper lacks enough technical contributions for acceptance.

The paper lacks novelty. Although the authors claim their pipeline for skeleton generation is novel, it simply combines two traditional approaches (MCF and DP) to extract skeletons from a mesh, without providing adequate motivation or context to demonstrate why this combination is innovative. Additionally, the paper lacks detail regarding the skeleton extraction process, such as the number of hyperparameters involved and guidance on setting these parameters. The comparison is limited to a learning-based method, RigNet, even though the proposed approach is not learning-based. The authors should expand the related work on mesh skeletonization and compare their method with more traditional mesh skeletonization techniques.

The multi-view and 3D generation pipeline also lack novely. For example, AnimatableDreamer already employs a skeleton-based 3D generation approach, contradicting the authors' claim that “their work is the first to achieve arbitrary skeletal-conditioned generation.” The use of a diffusion model in this approach is also not novel; while skeleton conditioning is applied, it is not unique, as AnimatableDreamer similarly conditions its diffusion model on skeletons during training.

The experimental validation is insufficient. In mesh skeletonization, the proposed approach is only compared to SDEdit, excluding other established methods such as those by Tagliasacchi et al. (2012) and Bærentzen & Rotenberg (2021), which are cited in the related work. For 3D generation, no experimental comparisons are made with state-of-the-art (SOTA) methods, even though numerous 3D generation techniques, including ProlificDreamer, MVDream, Farm3D, Text2Video-Zero, and AnimatableDreamer, should be included in the evaluation to better demonstarte the performance of the proposed approach.

• Several details in the presentation require improvement:
  • The overall objective function should be explicitly stated. If not an end-to-end model, the supervision signals and objective functions for each stage should be clearly described
  • Regarding the skeletal condition representation experiments, qualitative results in Figure 8 should be aligned with Figure 9, showing comparisons between w/o CCE-D (Raw), CCE (color only), and CCE-D (color+depth) as conditions. The corresponding relationship between ablation experiments and notation in Section 6.1 should be clarified accordingly
• Otherwise, the work is relatively self-contained without significant issues