Self-Conditioned Diffusion Model for Consistent Human Image and Video Synthesis

1. As far as I understand, the proposed method should be quite efficient compared to previous works since there isn't any copied UNet structure. Is there any discussion or comparison of the efficiency, e.g., trainable parameters and inference time for a single batch?

2. What's the difference between the proposed strategy and a "trainable version" of Reference-Only ControlNet, from here? I believe Reference-Only ControlNet also proposed a similar share-weight structure for appearance control. Any detailed discussion on the architecture design?

3. Metric for video generation evaluation. I understand the authors follow previous works and adopt FVD as the video evaluation metric. However, this metric has recently been widely criticized by the community because of its inaccuracy in reflecting the overall quality. I wonder what the performance comparison would be if debiased FVD is used for evaluation. From here

4. Are there any side-by-side video visualization comparisons between this work and recent baselines? E.g. MagicPose, Champ? It would be better to judge the temporal consistency of the video quality.

5. How does the model generalize to out-of-domain real human identities? E.g. Old people?

6. The denoising network has been fine-tuned on real human datasets and 3500 self-collected dance videos only, but the identity preservation for cartoon-style images in Figure 11 and the supplementary video is quite good. Is there any explanation for this? Do the self-collected videos contain any cartoon characters?

I'm more than willing to raise my score if my concerns are addressed.

1. No technical contribution. The technical novelty is limited. The significance of this paper is not expounded sufficiently. The author needs to highlight this paper’s innovative contributions to prior-guided I2I/ I2V generation.
2. Overclaim and SOTA. The experimental comparisons are outdated, comparing against older methods while claiming "state-of-the-art" status. The work overlooks recent 2024 publications and lacks quantitative evaluations against recent work. Authors should add necessary discussions and comparisons about some of the following: "Controlnext", "MimicMotion", "Cinemo", "PoseCrafter(ECCV'24)", "Mimo", "X-portrait(SIGGRAPH'24)", "PoseAnimate(IJCAI'24)", "DynamiCrafter(ECCV'24)", "SparseCtrl(ECCV'24)", "LATENTMAN(CVPR'24)" and "Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis (CVPR'24)"......

My main concern is that the technical contributions of this paper appear to be incremental.

In the context of controllable human animation generation, I am only knowledgeable about several widely studied works, such as Animate Anyone and Champ. To me, the approach of directly concatenating reference latents and target noise latents spatially for a unified SD diffusion process is new. However, the "inpainting motivation" and spatial conditioning strategy are common in image-to-video generation [1] and multi-view 3D generation tasks [2, 3, 4]. Given that the quantitative improvements are minimal and there are insufficient qualitative comparisons—since the supplemental videos are solely produced by this paper—it is challenging to draw definitive conclusions about the effectiveness of the proposed method.

Regarding the causal feature interaction strategy, it provides only slight improvements, as shown in Table 2 (PSNR: 18.64 vs. 18.59). Based on Figure 6, it seems that the causal feature interaction strategy may not be effective. In fact, it appears that the full model introduces artifacts in the connection region of the shoulder and neck compared to the model that does not utilize the causal feature interaction strategy.

[1] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. [3] One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion. CVPR'24 [4] InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models [5] CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model. ECCV'24

1. The logic behind why inpainting is advantageous (Ln221-223) is unclear and requires further clarification. Simply framing the task as inpainting does not inherently address how it enhances appearance consistency.
2. The proposed "causal feature interaction" lacks novelty. It is intuitive that target features should query information from the reference, while reference features should query only from themselves; this approach feels too trivial to be considered a novel contribution.
3. The description of the method in Ln238-287 is overly redundant, especially regarding the use of self-attention in diffusion to achieve content consistency. This observation has already been well-documented in previous video generation research.
4. There are performance concerns. In Table 1, the FID score is significantly higher than other methods, suggesting suboptimal quality. Furthermore, in Table 2, a straightforward spatial conditioning approach without causal feature interaction achieves a lower FID and FID-VID, along with a higher SSIM, which suggests that the main claimed contribution—"causal feature interaction"—does not improve results. In fact, pure spatial conditioning seems sufficient for content consistency. Additionally, Figure 6 shows that results "without causal interaction" are visually closer to the ground truth. Could the authors provide more video-format visual results to clarify?
5. The paper has instances of careless writing (e.g., Ln261) and inconsistencies between titles and tables (e.g., Table 2), which detract from readability and clarity.

1. While the insight of inpainting manner is reasonable, it heavily relies on the capabilities of the foundation model. Since this work takes SD1.5 as the base, which isn’t fully perfect for generating humans, there doesn’t appear to be a mechanism to address the situation when the base model lacks such an ability. This raises reasonable doubt that the effectiveness of the results is largely due to fine-tuning the base model with the dataset rather than overcoming inherent issues. Additionally, base models indeed perform outpainting, but this does not mean their results are consist, there is still a gap.
2. The method primarily focuses on the spatial aspect, with no special treatment on the temporal dimension for video generation—just following the AnimateDiff. So, how to improve the consistency in temporal?
3. The observed phenomenon (line247-267) , whether it is too model-specific (SD) or architecture-specific (UNet-base), this phenomenon may not be universally present. If so, please provide more observation results of models and architectures (DiT), like SD3.5.
4. Dedicating too much of the introduction to detailing previous methods, making it difficult to quickly grasp the main contributions of this paper. It is recommended that the authors reorganize this section, using concise language to summarize the primary limitations of prior work and clearly present contributions of this work.