MoveAnything: Controllable Scene Generation with Text-to-Image Diffusion Models

• Clarity: the idea of the paper is not well laid out in my opinion - i.e. the paper could be written significantly better. E.g. the caption of Figure 2 (the main method idea) is basically not understandable without reading the paper itself (small things: in the optimization box, nothing changes with optimization? I think you should change the features as they're updated, e.g. from f_1 to f'_1; also both Figure 1 and Figure 2 use rectangular masks, which on first reading, I took as a limitation of the paper). In both the paper and the caption the core idea should be laid out first (i.e. multiple views, but then pooling/averaging the resulting features). Further, a theoretical, or intuitiv, motivation should be given as to why this is believed to work well (after all the authors must have had something in mind when starting the study) - e.g. the averaging removes unwanted hallucinated artifacts (I'm 'hallucinating' here of course).
• The experiments are quite limited and only applied to a single dataset. Further, this dataset is generated, and not a ground-truth image dataset. It seems the method is not applicable to any real image not generated from a diffusion process? If so, this is a severe limitation that should be pointed out. Also, as the dataset is not to be published(?), this hampers scientific reproducibility.

Smaller things:

• p.4. "Diffusion Models", you cite Ho et al., which is great, but there has been work before (e.g. [1])
• p.6. 'implicit expression': why is this implicit? I'd argue it's explicit.

[1] Deep Unsupervised Learning using Nonequilibrium Thermodynamics. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, Surya Ganguli. Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:2256-2265, 2015.

1. Contribution of the proposed method is overclaimed.

• The authors assert that their work is "one of the first works to study diffusion-based controllable scene generation and spatial image editing". However, there have been numerous studies on these two research topics, such as [1] (controllable scene generation) and [2] (spatial image editing). The authors may change the expression to specifically refer to "moving objects." Nonetheless, there are some concurrent works, such as [3] and [4], which are not adequately discussed in the manuscript.
• The movement of objects in experiments is limited to [-0.2, 0.2] of image size, and the provided visualizations suggest that the proposed method's object movements are minor, restricting the proposed method's applications. Is it hard to significantly alter the spatial positions using the proposed method?

2. Experiments are not sufficient and rigorous.

• Although Figure 1 and 2 illustrate an example of moving multiple objects in a scene image, the curated benchmark solely contains images with only one object, which makes the experiments less convincing.
• Some experimental settings are unclear. For example, (1) How is the subset dataset for layout-conditioned scene generation evaluation selected? (2) Why do the proposed method and baselines use different diffusion schedulers for image-conditioned scene generation?
• $w$ is a critical hyperparameter for the image editing task, but its value is not provided and is not investigated in the ablation study.

3. The writing style could be improved, and some presentations are confusing.

• The symbols in some of the most critical equations, such as Equation (7) and (10), are confusing. $f^{(t-1)}$, $\tilde{f}^{(t-1)}$ and $\tilde{f}_k^{(t-1)}$ appear simultaneously. Do they have the same meaning?
• Figure 2 is misleading as it divides the generating process into the "optimization stage" and "inference stage". However, both of these stages are the inference stage for a diffusion model, and the "optimization stage" does not actually optimize network weights or input images. "Modification/modified" might be a better word than "optimization/optimized," in my opinion.

[1] Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors, ECCV'22

[2] RePaint: Inpainting using Denoising Diffusion Probabilistic Models, CVPR'22

[3] DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models, ArXiv'2307

[4] AnyDoor: Zero-shot Object-level Image Customization, ArXiv'2307

• Although the idea is quite novel, most of the machine learning techniques used in this paper can be traced back to earlier works.
• I feel that the current setting is a bit too constrained: each mask seems to be fixed in size and can only shift in image space, this is usually not the case in real world scenarios, where objects can move in three dimensions, and undergo rotation as well. It seems to me that it would be challenging to extend the current approach to these scenarios, given that the feature map is discretized in pixel space and any other transformation will result in a change of mask size/shape.
• Following the previous point, I am rather uncertain about the assumption that objects under different layout can share a single latent feature map, which is different from a (semantic) latent code. It appears to me that there is the assumption that the object appearance will stay similar when the layout changes, however, I don't think this is usually the case, as there will be changes in perspective/lighting/occlusion/etc that accompanies layout changes. All the examples shown in the paper show basically a front view of things so this is not that noticeable, but I would really want to see the method in other cases e.g. what if the room in figure 1 is wider, and moving the bed causes the side of the bed to be visible in certain layouts? What about Figure 3, where MultiDiffusion produces a backpack in a different pose (right)? Is the proposed method also able to generate anything that is not the exact from view of the bag? What if object in some "views" is occluded but not in other views? These seem to be rather important cases that the method should be able to handle.
• I probably won't have these questions if the evaluation are more comprehensive. I do find the evaluation very lacking, especially the lack of qualitative samples. I think some prelim on prior techniques can be trimmed to allow more space for visualizations.

• Missing an important related work GLIGEN [1].

  • It is the first work to bring layout-controllability to stable diffusion and should be compared as a baseline in layout-to-image generation.
  • It should be cited in Sec. 3.2.3 as the described mitigation is the same as the scheduled sampling in GLIGEN paper [1] Li, et al. "Gligen: Open-set grounded text-to-image generation." CVPR 2023.

• Given the layout-to-image setting and the illustration in Fig. 1, the paper does not quantitatively evaluate on the accuracy of the bounding box conditioning. The authors are suggested to evaluate the precision of the object location on COCO dataset following the setting in [1], as the COCO dataset is more well-established and contains complex scenes as well.

• There are no quantitative experiments / qualitative results on if the proposed approach can handle overlapping regions/masks. Suppose a person is standing in front of a car, the car bbox will be split to half by the human -- the resulting rendered result is ambiguous: is this a person standing in front of a car, or it stands between the two cars. Despite the layered latent feature maps, they are rendered to the same canvas $v_n$, and the delta noise is predicted by the u-net on $v_n$, which has the ambiguity issue.