Tradiffusion++：Hierarchical Guidance for Fine-Grained Trajectory-Based Image Generation

• I generally find "trajectories" less intuitive and user-friendly than "masks," particularly for articulated objects like humans or animals. Drawing a trajectory or skeleton for such objects requires some level of artistic skill to envision a natural pose, which can be challenging. On the other hand, masks don’t impose strict control on the model, giving it the freedom to generate a natural appearance for the object. For example, how would one know that the "lambda" shape in Figure 6 is suitable for a cat? Similarly, the teddy bear in Figure 8. It is evident in Figure 15 (the kid and bus) that unintuitive trajectories can lead to distorted objects. I believe that the usability of trajectories needs to be investigated compared to masks through a user-study.

• The main contribution, consisting of the three losses, is relatively minor and heavily inspired by [1]. The main difference is using a trajectory mask rather than bounding boxes (rectangular masks) to optimize different cross-attention maps. In fact, at low-resolution layers, it becomes identical to that of [1] for small trajectories, as illustrated in Figure 5. Therefore, I find the technical novelty of the proposed approach quite limited.

• The insights mentioned in section 3.2 are not new and were discussed in detail in several earlier works such as [2-4]. How do the insights provided in the paper differ from those in [2-4]?

• The "Coordinate Transformation" operation in Figure 3 dilates the trajectories, making them closely resemble segmentation masks. Consequently, a comparison with segmentation-based approaches, such as those in [5,6], is necessary to determine whether "trajectories" offer any advantage over segmentation maps.

[1] Chen, Minghao, Iro Laina, and Andrea Vedaldi. "Training-free layout control with cross-attention guidance." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

[2] Hertz, Amir, et al. "Prompt-to-prompt image editing with cross attention control." arXiv preprint arXiv:2208.01626 (2022).

[3] Tang, Raphael, et al. "What the daam: Interpreting stable diffusion using cross attention." arXiv preprint arXiv:2210.04885 (2022).

[4] Liu, Bingyan, et al. "Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[5] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7701–7711, 2023.

[6] Guillaume Couairon, Marlene Careil, Matthieu Cord, Stephane Lathuiliere, and Jakob Verbeek. ´ Zero-shot spatial layout conditioning for text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2174–2183, 2023.

1. More elaboration of control loss compared to [1] is needed. What is the difference?
2. More qualitative ablation studies can be done, using the same text and trajectory prompt, to verify the effectiveness.
3. More visulization results are expected, such as more complex trajectories with overlapping

[1] Chen, Minghao, Iro Laina, and Andrea Vedaldi. "Training-free layout control with cross-attention guidance." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

I mainly have concerns of the experiment comparison and the discussion of a series of training-free methods.

Experiment comparison. The authors mainly treat TraDiffusion as a baseline, a training-free based method. However, it is still necessary to compare with other popular baselines like ControlNet and InstanceDiffusion etc. Despite they are not training-free methods, the ControlNet is already well-developed enough and very easy to use when just input the layout/skeleton images. The authors can also consider making them integrated into ControlNet to show the orthogonal ability of the proposed method. Moreover, the authors also should compare with other training-free methods especially leveraging the guidance such as [1] or some feature injection methods [2]. I recommend the authors do a more comprehensive comparison to other training-free methods.

More importantly, the authors point out a good point that it is an energy function method (Line 185). I suggest the authors could elaborate more about what the differences between energy-based methods and the other gradient-based method [1], and the advantage compared to feature injection methods like ControlNet and [2] etc. The authors can consider to analog the diffusion generation to the SGD, which is pointed out in [3]. These discussion will make the paper more theoretically-sound.

[1]. Universal Guidance for Diffusion Models [2] Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation [3] Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering

• Insufficient experiments. The main weakness of this version of the paper is the lack of comparisons with existing work. The main table (Table. 1) only compares the paper to TraDiffusion and plain stable diffusion, but no other types of controllable diffusion generation methods. There are immediate questions of how TraDiffusion++ compare to these baselines
  • Control methods that require training such as ControlNet and ControlNext. Can these methods be adapted for scribble-based generation?
  • Training-free methods such as [A] support generation with bounding boxes. Can they be adapted? Even if the answer is not, one very intuitive baseline is to use a bbox to bound the scribbles, and run the generation with the bounding boxes.
• More diverse generation examples. Though the proposed method seems generic and the paper includes many qualitative samples. They focus primarily on generation of scenes involving animals. Inclusion of other objects may be interesting.

[A] Chen, Minghao, Iro Laina, and Andrea Vedaldi. "Training-free layout control with cross-attention guidance." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.