DORSal: Diffusion for Object-centric Representations of Scenes $\textit{et al.}$

Thank you for your review and suggestions to improve our work! We are pleased that you recognize how our extensive experiments on MultishapeNet and Street View datasets support the effectiveness of our approach, and how this research can enable many applications.

In terms of the Weaknesses (W) and Questions (Q) you list, our response is as follows:

W1: The paper only shows results with low resolution.

Indeed, we agree that experiments with higher resolution would be even more impactful. However, the resolution of 128x128 we report results on is typical for relevant prior work. In fact, both the diffusion-based 3DiM baseline and the SRT-based baselines report only up to 128x128, as does the video diffusion architecture we base our decoder on.

Generally speaking, high-resolution, consistent video generation (or in our case generating multiple views of the same scene) is a challenging open problem (Hoogeboom et al., “End-to-end diffusion for high resolution images”, 2023). Fortunately, a straightforward approach to increase the resolution is via cascading: having multiple stages of diffusion, where later stages perform super-resolution to arbitrarily increase the resolution (Ho et al., “Cascaded diffusion models for high fidelity image generation”, 2022). This approach is equally applicable to DORSal, though we have not considered it in practice, as high resolution generation would require an additional standalone diffusion model and is not the main focus of our work.

W2: The proposed scene editting scheme can only support object removal and transferring.

Indeed, we acknowledge that this is a limitation of our current approach and supporting more fine-grained scene editing operations is an important direction for future work. For the rotation and translation you mentioned in particular, it is foreseeable how supervised co-training with language can provide an interface for this as in “Object 3DIT” (Mochel et al., 2023), which we cite in our work. Alternatively, the object representations themselves could be disentangled to a point, where information about the rotation or position of an object is isolated, and can thus be manipulated independently during generation. In general, fine-grained, object-level control is an active area of research, and approaches that offer such capabilities like DisCoScene (Xu et al., 2023) requires a lot more prior knowledge about the scene to be able to work.

We recognize that the current discussion of limitations is perhaps too lenient in this regard. In the revision, we will update it to comment on fine-grained control.

Q1: Could you please provide object-decomposed visualization as illustrated in the OSRT paper (Fig. 4).

Thank you for your interest. Figure 1 in the current submission contains a representative example of how OSRT assigned different parts of the scene to different slots (like Figure 4 in the OSRT paper). For completeness, we have uploaded the same visualization for the other target views as well, and for an additional scene, in the supplementary material (osrt_streetview_decomposition.pdf). It can be seen how OSRT (trained fully unsupervised here) usually offers a coarse decomposition of the background into broad segments and partially succeeds at decomposing some of the objects.

We note how OSRT was never evaluated on Street View in the first place, and is clearly pushing the limits of what it is capable of. For this reason we show interesting cases in Figure 6 and exhaustive edits including failure cases in Figure 13 in Appendix C. In Appendix C.2 we comment on how “These failure modes likely originate in part from the unsupervised nature of the OSRT base model, which sometimes assigns multiple slots to a single object, or does not decompose the scene well. Fully “imagined” objects (i.e. objects which are not visible in the input views and therefore not encoded in the Object Slots) further generally cannot be directly edited in this way.”. In that same section we also hint at possible solutions for addressing this.

Thank you for your review and suggestions to improve our work! We are excited that you agree that this work addresses an important problem and is a “nice contribution to the research of SRT”. We were also glad to learn that the paper is easy to read, well-written and very fluent and informative. Thank you also for the additional citations, which we will make sure to include.

In terms of the Weaknesses (W) and Questions (Q) you list, our response is as follows:

W1: A little reserved technical novelty.

We mostly agree with your assessment regarding technical novelty, and your summary of our work as a “Straightforward success of the proposal idea to an important problem” is well put. The focus of this work is on making progress towards NVS from few observations in real-world scenes, while also opening up (or preserving, depending on how you look at) the possibility for scene editing. We achieve this by reusing and repurposing existing techniques from the literature, i.e. object-based representations of 3D scenes and video diffusion decoders. In demonstrating the feasibility of this approach (and what’s more, how well it actually fares against the baselines), we expect follow-up work to be able to make further progress toward achieving these ambitious goals.

Because of this, we did not prioritize experiments that specifically target the design of the decoder or other architectural parts. In fact, at the pace at which video diffusion models progress, it is not unreasonable to expect a future version of DORSal to incorporate a slightly different decoder design (e.g. using latent diffusion). In terms of code, there is a publicly available implementation of OSRT (https://github.com/stelzner/osrt) and video diffusion models (https://github.com/lucidrains/video-diffusion-pytorch) already available. Hence, it is unlikely that we will release our own implementation of these. That said, we are happy to update the paper with additional details or hyperparameters if you find that anything is missing. We would also be happy to include pseudo-code in the paper if you think that would be useful. Generally, we have tried to take care in putting as much detail in the paper as possible.

W2: 3D inconsistency. SRTs do not have 3D consistency in design. Furthermore, DORSal could have worse 3D inconsistency due to its decoding process, in my understanding.

We acknowledge that DORSal is not perfectly 3D consistent for the same reasons that SRT is not. And indeed, the fact that DORSal is stochastic adds an additional significant challenge, as any new information the model adds during the generation process needs to be synchronized between all generated views. Video diffusion models offer a promising solution for this problem for short, consistent video clips (and in the future likely for much longer videos) by jointly modeling a full sequence of frames. Achieving perfect consistency across a long camera path is still highly challenging as it goes beyond the context length that can be modeled directly via attention in the video diffusion model.

To provide some further insight into the issue of 3D consistency, we have re-computed the edit-segmentation scores on a subset of the samples for DORSal, where we measure both Edit FG-ARI across all the views simultaneously and a “2D Edit FG-ARI” where we compute Edit FG-ARI for each view individually and then average the results. Note that the latter does not penalize inconsistencies between the views, hence the gap between the two scores is indicative of any inconsistencies taking place. A similar approach to evaluating 3D consistency was carried out in the OSRT paper. We obtain 0.702 Edit FG-ARI and 0.721 2D Edit FG-ARI. The small gap between these scores indicates that the segmentations obtained via this procedure are highly consistent across views.

In terms of possible applications, a more 3D consistent approach is typically preferred, and in our paper we have suggested two possible techniques for improving this. Firstly, as shown on MultiShapeNet, having a view distribution that mixes close by and far apart cameras is more desirable. Secondly, having additional stages of “frame shuffling” throughout denoising can help address this. In the limit, where each stage corresponds to a single denoising step, this becomes similar to the “stochastic conditioning” technique deployed in 3DiM for this purpose. Finally, although we haven’t shown this, it is foreseeable that a cascaded diffusion approach using additional super-resolution stages can help resolve tiny inconsistencies.

Thank you for your review and suggestions to improve our work! We appreciate that you recognize that our methodology is sound, that DORSal can obtain significantly better FID, that the renderings appear consistent across views and over time, and that the presented edits are mostly realistic.

W1: The scene editing is more a property of OSRT than the proposed method. The object-slots are pre-trained from OSRT and not refined or learned in this paper.

We agree that the editing capabilities we present here are mainly a consequence of having object slots. However, other aspects of the model play an important role too. For example, because DORSal is a probabilistic approach, it is better equipped to handle the inconsistencies that can arise due to removing or combining different slots and the uncertainty that arises from this. Indeed, compare the plane in the middle frame in the top row in Scene 2 to the same frame in the Combination scene directly to the right. Notice how DORSal generates plausible looking shadows under the plane to match the lighting when viewed from this angle as in Scene 1. This impressive feat is a direct consequence of using a diffusion-based decoder and not something that the deterministic OSRT decoder can achieve. Indeed, this observation alone already might indicate that a diffusion-based decoder might be more suitable for the editing operations presented here.

W2: One main limitation of the method is that we can not control specific object slots

Indeed, we acknowledge that this is a limitation of our current approach and supporting more fine-grained scene editing operations is an important direction for future work. For the rotation and translation you mentioned in particular, it is foreseeable how supervised co-training with language can provide an interface for this as in “Object 3DIT” (Mochel et al., 2023), which we cite in our work. Alternatively, the object representations themselves could be disentangled to a point, where information about the rotation or position of an object is isolated, and can thus be manipulated independently during generation. In general, fine-grained, object-level control is an active area of research, and approaches that offer such capabilities like DisCoScene (Xu et al., 2023) requires a lot more prior knowledge about the scene to be able to work.

We will improve our current discussion of limitations to put more emphasis on this, and highlight ways in which future research may address this.

W3: The videos in the supplementary material would have been more informative if they would have shown results for all methods compared to as well, and also the input image/images.

Thank you for pointing this out. We will add video results for baselines to the supplementary results in the updated version of our paper. In the meantime, we recommend viewing the website of the OSRT baseline (https://osrt-paper.github.io/) for representative video results.

Thank you for your review and for your constructive comments. We appreciate that you recognize the strengths of our method, in particular the substantial improvement in generation quality compared to prior work, and the novel conceptual contribution of object-level scene editing in diffusion generative models enabled by conditioning on structured, object-centric scene representations (“object slots”).

We address your comments regarding your highlighted weaknesses of the paper in the following.

W1: The model reliance on pre-trained object-centric representations instead of end-to-end training with diffusion models is a potential weakness.

We’d like to respectfully push back on this argument (in the context of this being a weakness of our paper), for the following reasons:

• Joint (end-to-end) representation learning and generative modeling with diffusion models is an active area of research and achieving competitive representation learning performance this way (as opposed to using deterministic reconstruction objectives or contrastive learning) is an open research problem. Solidly addressing this problem goes far beyond the scope of our investigation. In the future, as the community makes progress on this problem, e.g. in the setting of single-image generative models, it would be great to revisit the end-to-end setting in the context of DORSal.
• We rely on the ability to inject segmentation supervision into the OSRT base model on MSN-H for significantly improved scene editing performance. While this is straightforward for OSRT, where the decoder provides a clear (soft) segmentation prediction, it is unclear how to achieve this when training the OSRT encoder end-to-end through a DORSal diffusion decoder, which does not provide an obvious modeling path for segmentations.
• Separating the problem into a representation learning stage for conditioning information (here: object slots) is common practice in text-conditioned diffusion models. It has several benefits: 1) significantly reduced memory, compute, and infrastructure complexity; 2) modularity: our two-stage training recipe has the advantage of directly benefiting from future advances in scene representation learning methods as well as diffusion generative models.

Regarding your comment “the quality of the slot representations will become the bottleneck of improving the generation quality”: this is true irrespective of whether the model is trained end-to-end with a diffusion-based decoder or a deterministic decoder. We believe that there is no reason to assume that a diffusion-based decoder would necessarily result in better object representations compared to a deterministic decoder (which we chose). Running such an end-to-end experiment is non-trivial with the existing models used in our work since they live in different code bases and are optimized to make maximum use of available device memory (in addition to the issue of segmentation supervision injection). We leave this investigation for future work, once joint representation learning and generative modeling with diffusion models has advanced as a research field.

W2: Given that the representations are pre-trained, the model seems to be too straightforward and limited in its technical contribution.

We agree that the novelty solely in terms of architecture contribution for the individual method components is limited. In our view, the novelty and significance lies in the elegance/simplicity of the combined approach (which has not been demonstrated before), as our recipe allows for (almost) direct use of established components and thus also benefits directly from future development in these areas. Indeed, we demonstrate that conditioning a video diffusion architecture on (structured) scene representations and pose encodings allows us to overcome limitations of a prior state-of-the-art 3D generative diffusion model (3DiM) and introduce a sampling process to achieve improved consistency for sampling long camera path videos (up to 200 frames), which is an extremely difficult task.

Aside from NVS, consistent object or asset transfer between scenes in video/3D generative models is a major unsolved problem, and DORSal shows promising progress on this problem. DORSal demonstrates that this recipe can achieve new forms of control (object removal, object transfer) via simple manipulation of the conditioning information without the need to collect paired editing data or the need to perform pixel-level masking and inpainting.

We would appreciate it if you could reconsider your assessment regarding novelty (and significance to the community) to focus on aspects beyond architectural novelty, as discussed above.

I thank the authors for their response and apologies for the delay.

To clarify, my question did not pertain to the comparison between representation learned by diffusion model or OSRT, which would indeed be beyond the scope of this paper. Rather, I am concerned whether the information contained in the pre-trained, frozen representations was sufficient for the diffusion decoder to accurately render the fine details in image generation. This is questionable given the fact that the representation learning approach, OSRT, demonstrates suboptimal reconstruction quality in complex images. This could suggest that it might not capture the finer details necessary for generating new viewpoints of the same objects later when using the diffusion model.

However, after reviewing the feedback from other reviewers and the authors' responses, I do agree that it is interesting to see that the high-quality image generation can be achieved by the simple combination of existing approaches. Moreover, it is also quite surprising to see that the frozen representations could provide generations that are far beyond the detail level demonstrated by the results of the representation learning model OSRT.

Considering these points, I am raising my rating.