Cycle-Consistent Learning for Joint Layout-to-Image Generation and Object Detection

• About the methodology:
  • Are the operations between the L2I generator and the object detector all differentiable or do you use gradient estimation methods like STE?
  • In line 244, to implement single-step sampling, the authors opt not to do Gaussian sampling as usual. So how to sample the special noise $x_t^{pert}$?
  • For unpaired circumstances in Sec. 3.2.3, since GT images are not available, I believe you need to run the whole T-step denoising for each sample in the training time, right?
  • Do you modify the layout encoding manner, or maintain the same with your baseline methods?
• About experiments:
  • Initialization: do you train from Stable Diffusion or GeoDiffusion/ControlNet? If the latter, you should clarify it more clearly in Sec. 4.1. Moreover, for the results of GeoDiffusion/ControlNet in Table 1-4, it would be better if you can fine-tune the pre-trained checkpoints also for 2/3 epochs for a fair comparison.
  • In Table 6(a), what does the variant "Baseline + $L_{ldm}$" mean, since the baseline has utilized the LDM loss?
  • In Table 6(c), do you use the same GDCC (trained with Faster-RCNN) to run experiments with different detectors?
• Overall, I think this is a solid paper, but there are several details requiring further clarification and refinement. I hope my reviews can help revise the paper.

(1): While the authors tested the benefits of Cycle-Consistent learning for the Layout-to-Image (L2I) task using two advanced generative models, GeoDiffusion and ControlNet , they employed relatively outdated detectors such as Faster R-CNN to evaluate the object detection task, yielding improvements of less than 1%. More experimentation with state-of-the-art detection models is needed to verify whether GDCC substantially enhances detection performance. (2): Lack of Analysis on Iterative Cycle-Consistency: The paper does not provide an analysis of the number of iterations required for Cycle-Consistency nor address the potential risk of mode collapse over multiple iterations. Such an analysis would help clarify the stability of the iterative cycle process. (3): Limited Evidence of Unpaired Data Benefits: The results in Table 7 indicate only a 0.3% improvement with unpaired data, which is insufficient to demonstrate its effectiveness for enhancing detection performance.

• Overall, the proposed method uses an iterative training procedure, fixing one and updating the other, which will need to address the instability issues as commonly observed in training traditional GANs, although the proposed two motivations in the training procedure help prevent observing these issues in the experiments.
• In terms of how detectors help generators based on the proposed layout translation cycle loss (Eqn. 7), there are missing details. Eqn.7 requires the number of predicted bounding boxes and that used in layout-to-image generation are the same, N, which can not be always guaranteed. In the appendix, the ``calculate box loss" in Algorithm 1 is not elaborated. In experiments, small objects (<2% of image area) are ignored, which might show up in the detection. The COCO-Stuff train dataset also contains categories that are not in the COCO pretrained Fast R-CNN. In addition, the smooth L1 loss seems not effective in helping improving the fidelity of synthesized object instance. Overall, it brings doubts on how exactly detectors help generators.
  • The Equation indexes in Algorithms seem to be an older version, not match the main text.
• It is also not very clear how the generator helps the detector based on the image translation cycle loss (Eqn. 10). Considering the iterative nature of the proposed method, the similarity between $x_1^{syn}$ and $x_2^{syn}$ will entail less diversity of the generator.
• In the tables, some details are missing. E.g., how are the detection mAPs computed for baseline methods in comparisons. The number of epochs is not directly comparable considering the fine-tuning nature of the proposed method.