Spade : Training-Free Improvement of Spatial Fidelity in Text-to-Image Generation

1. The authors listed ControlGPT [1] and LayoutGuidance [2] in the introduction section to demonstrate the point that existing approaches either require expensive training or labeled annotations. However, this summarization seems to miss two related works: LLM-grounded Diffusion [3] and LayoutGPT [4]. While the pipeline of [4] requires a reference annotated dataset, the pipeline of [3] does not include training (other than stable diffusion training itself) or using extra labeled annotations. Furthermore, this work uses a pre-collected 3D asset library, which adds an extra requirement that previous methods [1,2,3,4] do not need. Since the requirements of [3] are similar to this work (i.e., Stable Diffusion and GPT-4 without additional training), an additional discussion between the current work and [3] is needed. The authors are also encouraged to compare with [3,4] in terms of spatial fidelity.
2. The work proposes a database named SPADE as one of the main contributions, but the database has 80 MS-COCO classes, with hand-collected 3D model assets and limited 2D spatial relationships. Therefore, it is hard to know whether the proposed LSAI method, based on SPADE, can generalize to categories that are rare and relationships that are different from the ones in the dataset (e.g., A is far away from the camera than B).
3. VISOR only considers simple relationships (e.g., above/under/left/right) with limited classes (COCO classes), which the synthetic database considers. More evaluation on challenging benchmarks that test diverse object types and relationships, such as the spatial part of [5], is needed to explain the improvements of this method in more challenging scenarios.
4. The user study only asks the users questions about whether the generation is correct. However, whether this method will introduce degradation in terms of visual quality is also a very important part in the evaluation. If this is not considered, then an example with simple copy-and-paste can achieve high scores in the evaluation.
5. This work lacks clarity in presentation in the method introduction. For example, it is unclear how Eq. 1 relates to LSAI in addition to being a part of the standard denoising diffusion framework. It is also unclear how the proposed LSAI, which leverages SPADE generator that requires explicit spatial keywords and object names, is able to parse arbitrary user inputs, including those that do not have spatial keywords or have other types of special keywords (e.g., near/far/inside/outside).
6. This work seems to generate floating objects that do not seem to compose well. For example, in Fig. 5, shadows are missing in the generated image.

[1] Controllable Text-to-Image Generation with GPT-4. https://arxiv.org/abs/2305.18583 [2] Training-free layout control with cross-attention guidance. https://arxiv.org/abs/2304.03373 [3] LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models. https://arxiv.org/abs/2305.13655 [4] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models. https://arxiv.org/abs/2305.15393 [5] T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation. https://arxiv.org/abs/2307.06350

• The technical contribution is limited. ControlNet as well as SPEdit are used on top of a synthesized scene using Blender.
• The method is limited to 2 objects and 1 spatial relation.
• Missing comparison with (cited) other layout-guided diffusion models. It is obvious that the presented model will achieve better scores when evaluated on spatial fidelity against purely text2image models.
• The generated images are similar to the reference images and hence often look unrealistic and stitched (unrealistic composition of objects and background).

Side note: SPADE is not a good name, the first thing that comes to mind is the very influential semantic image synthesis model https://arxiv.org/abs/1903.07291

1. The intuition behind SPADE is not explained well. The author should explain why the 3D rendering engine is needed here. Because the paper only focuses on vertical and horizontal direction, 2D is enough.
2. Some comparison experiment is missing. For example, LayoutGPT also focuses on the relationship issue, the comparison result should be given.
3. More visualization results are needed to validate the generalization ability. The OOD experiment is interesting, however, the author should show the result with different substitutes.

Despite the strengths, the paper has the following weaknesses that limit the significance of the contribution

• The diversity of spatial relationships. Though the paper presents the method as able to deal with multiple objects with multiple relationship terms, I think the spatial relationships that the paper uses are not diverse enough. Given that the dataset is generated with real 3D objects in 3D spatial coordinates, there are actually more spatial relationships than the four types given. Please take a look at [1] section 4, which includes 17 types of spatial relationships. Clearly, the proposed method can not reflect many other relationships including "inside/outside", etc. Other methods, LayoutGPT for example, are able to generate more diverse spatial relationships.
• The diversity of objects. Though the paper presents that the model is able to generalize to OOD objects, I still doubt the diversity of the objects that the method can generate. Given the database only provides 80 objects and the generated process is based on a retrieval manner, I wonder if the dataset is able to generate imagined objects or if daily objects look significantly different than the dataset assets. For example, "a lamp next to a spider".
• Photorealism. Despite the method's ability to perform well on the VISOR benchmark, it sacrifices too much in terms of photorealism.
• Background controllability. Layout-guided generated methods like layoutGPT, layout Guidance, and ControlGPT, are able to synthesize a reasonable background given the objects in the foreground. The proposed method seems to lose the controllability for background synthesis.

[1] 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment