Compositional Scene Understanding through Inverse Generative Modeling

We thank the reviewer for the positive feedback and comments! Please see our response below about your concerns.

1. For the experiments on CelebA, I am wondering why the authors opt to use all male faces as the OOD set, which effectively removes the female/male attribute from classification. The more standard subdivision of the dataset in ID/OOD regions for compositional tasks would be to ensure that the ID set contains all values of all factors, but not all combinations, e.g., all women are only shown smiling, all men are only shown with a neutral face in the ID set. See, e.g., [1,2].

Thank you for your insightful question and the suggested references! Our primary goal is to evaluate the generalization ability of our compositional inference framework under challenging out-of-distribution (OOD) conditions by ensuring that the test data is substantially different from the training data. For example, in object discovery, the model is trained on Clevr while tested on a different dataset ClevrTex. Following the same spirit, we decided to train our model on CelebA female faces while testing it on CelebA male faces, as there are significant facial characteristic differences between the two groups. However, we agree that subdivision of the dataset in [1,2], where all attribute values are present in the training set but not all combinations, is also helpful for evaluating compositional generalization, and we will update the paper to discuss this evaluation procedure. Unfortunately, this type of combinational shift is not available to gather in a controlled manner in real-world datasets, leading us to use the CelebA dataset.

[1] Schott et al., 2022, Visual Representation Learning Does Not Generalize Strongly Within the Same Domain

[2] Wiedemer et al., 2023, Compositional Generalization from First Principles

2. What is the dataset used in Sec. 4.3 and Fig. 7/Tab. 3? Is this a dataset curated by the authors? A comparison on CelebA with pretrained models in zero-shot mode could have been equally insightful, or the authors could have opted for an established multi-label classification dataset.

Thank you for the insightful question! Yes, the dataset used in Sec. 4.3, Fig. 7, and Table 3 was curated by the authors and consists of 20 images containing a cat and a dog, 22 images containing a cat and a rabbit, and 28 images containing a dog and a rabbit. Our primary focus in these experiments was to evaluate multi-object perception performance (local factor discovery) using compositional pretrained models compared against Diffusion Classifier. We agree that testing these models on CelebA to discover global attribute factors could provide additional insights and will do so in the final version of the paper.

3. Small typo L198: ignore --> ignoring

Thank you for pointing out the typo! We will fix it in the next version.

4. How feasible would it be to extend this method to more "involved" types of compositions, such as part-whole relationships or hierarchical compositions? Could this method (in principle, given unlimited compute) be extended to infer, e.g., a complete scene-graph of an input?

Thank you for the insightful question! We believe that our method would in principle be able to scale to more “involved” types of compositions, such as inferring a complete scene-graph. In the setting of a scene graph, each of the composed factors in our systems would correspond to an “edge” or relation between two objects. We then enumerate through different edges in a graph until we get a graph whose composed set of edges leads to a good reconstruction of the scene. We will update the paper and discuss this in the future work.

We thank the reviewer for the positive feedback and comments! Please see our response below about your concerns.

1.How scalable is the compositional inverse generative modeling approach when dealing with a very large number of visual concepts or categories? Would an alternative optimization strategy reduce complexity without sacrificing accuracy? Clarify computational overhead or potential methods to reduce complexity in real-world deployments.

Thank you for the insightful question! Overall, we agree that our approach is substantially slower than existing feedforward methods, but we believe that the additional computational cost is justified by the ability of our to generalize well to unseen complex scenes. When dealing with a large number of visual concepts and categories, our naive algorithm of enumeration has exponential growth of the search space ($M^K$) for discrete values. Beyond the early stopping strategy discussed in our submission, several additional approaches can potentially significantly mitigate this computational bottleneck. One approach could be to use heuristic search algorithms on discrete values – for instance we can run beam search with a beam width of K over each attribute sequentially which can reduce time complexity from $M^K$ to $O(M*K)$, making inference more efficient for large discrete spaces. Alternatively, continuous relaxation of discrete variables with gumbel-softmax/concrete relaxation could allow the use of gradient-based search and thus avoid enumerating all configurations. Finally, since our approach allows parallel processing across the $M^K$ configurations, inference time can be drastically reduced given sufficient computational resources, potentially approaching the time required for a single configuration evaluation. We will discuss these possible extensions in the future work.

2. Consider discussing the limitations of compositional approximations explicitly in the paper.

Thank you for the valuable suggestion! We will add a limitations section in the next version of our paper and discuss this. Our compositional generative modeling approach assumes object concept independence given the input image, enabling combinatorial generalization beyond the training distribution. However, one possible limitation of this full independence approximation is that it ignores the interaction between objects, which are crucial in many real-world scenarios. As a remedy, we could potentially learn additional models that model interactions between object components which can also be composed to represent more complex scenes. We will explicitly discuss this limitation and potential remedies in the limitations section.

3. Could you further elaborate on how this compositional modeling approach would handle dynamic or interactive scenes (eg, videos)? Have you considered any temporal extensions or adaptations?

Thank you for the insightful question! Extending our approach to dynamic or interactive scenes is an exciting direction for future work that we are also currently exploring. In the dynamic setting, we can change the reconstruction objective to not be reconstructing a given image but instead to reconstruct the entire video of the target interaction. We can condition each composed generative model on different aspects of the interactions such as the identity of an individual objects or agent in the environment. The inverse generative modeling procedure can then be used to persistently discover objects in the scene, even if they are occluded at different points in time, or to infer the individual behaviors of each individual agent in the interactive environment. We will update the paper and discuss this in future work.

We thank the reviewer for the constructive feedback and comments! Please see our response below about your concerns.

1. The tested datasets contain simple concepts (limited number of objects in CLEVR, limited set of attributes in face), so although performance evaluation positively supports the proposed idea, the method in its current state still cannot be extended to more complex real world scenes.

Thank you for your insightful question! Our model is evaluated on widely adopted datasets (CLEVR and CelebA) that are commonly used for object discovery and multi-label classification. While these datasets are relatively simple, they serve as strong benchmarks for measuring effectiveness and generalization, as also acknowledged by reviewers 6fxB, Tv9z and s7kA. Nonetheless, we agree that testing our model on more complex scenes would be interesting. To take a first step towards that end, we conducted additional evaluations on ClevrTex [1], which features diverse object colors, textures, shapes, and complex backgrounds. As shown in the table below, our model outperforms all baselines in object discovery on ClevrTex, demonstrating its potential scalability to more complex scenarios.

[1] Karazija, et al., 2021 Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation.

2. When inferencing a continuous concept, how to decide how many random trials are needed in order to escape a local optima solution?

Thank you for raising this important point! We acknowledge that escaping local optima in continuous concept inference can be influenced by the number of random trials, as demonstrated by the ablation study (Table IV) of our submission. To determine the optimal number of trials, a principled approach is to monitor the reconstruction error of the input image: if increasing the number of trials no longer leads to significant reconstruction improvements, further trials become unnecessary. In practice, our experiments show that a moderate number of trials (5 or 10) is sufficient for continuous inference tasks.

3. For the tested perception tasks, how is the inference time of the proposed method compared to the related baselines ?

Thank you for suggesting the evaluation metric! We run our model and baselines on Nvidia H100 and report inference times in the table below, where “N/A” indicates that the approach is not applicable to the task. Our results show that generative approaches, including our approach and Diffusion Classifier, require more time than the discriminative approaches like ResNet-50 and Slot Attention, while our approach consumes more time than Diffusion Classifier. Specifically, our approach takes longer than Diffusion Classifier due to the need to evaluate a composition of multiple diffusion models. However, this increased inference time for our approach comes with significantly improved generalization performance. Additionally, inference time bottleneck could be further mitigated through parallel processing, which our algorithm supports, or using heuristic search algorithms such as beam search to speed up optimization, which we leave for future work. We appreciate your suggestion and will include the inference time metric in the next version of our paper and discuss how improving inference speed is an interesting direction of future work.

We thank the reviewer for the constructive feedback and comments! Please see our response below about your concerns.

1. One concern is the fairness of comparison with unsupervised Slot Attention. I recommend to compare with more advanced DINOSAUR.

Thank you for the insightful suggestion! We would like to clarify that the Slot Attention baseline we compare to in the paper has incorporated supervision on each head to the object coordinates in the scene (discussed in Appendix A.5), and thus has the same information as our approach. Furthermore, following the reviewer’s suggestion, we compared our model with DINOSAUR also with supervision on Clevr. As shown in the table below, DINOSAUR works better than Slot Attention, while our model outperforms DINOSAUR.

2. One concern is the efficiency of the algorithm. For discrete values, the size of the search space $M^K$ will increase exponentially.

Thank you for the insightful question! Overall, we agree that our approach is substantially slower than existing feedforward methods, but we believe that the additional computational cost is justified by the ability of our method to generalize well to unseen complex scenes. It’s true that our naive algorithm has exponential growth of the search space ($M^K$) for discrete values. Beyond the early stopping strategy discussed in our paper, several additional approaches can potentially significantly mitigate this computational bottleneck. One approach could be to use heuristic search algorithms on discrete values – for instance we can run beam search with a beam width of K over each attribute sequentially, which can reduce time complexity to $O(M*K)$, making inference more efficient for large discrete spaces. Alternatively, continuous relaxation of discrete variables with gumbel-softmax/concrete relaxation could allow the use of gradient-based search and thus avoid enumerating all configurations. Finally, since our approach allows parallel processing across the $M^K$ configurations, inference time can be drastically reduced given sufficient computational resources, potentially approaching the time required for a single configuration evaluation. We appreciate your suggestion, and will include comparative experiments of the overall time complexity in our paper and discuss ways to speed up inference.

3. In Algorithm 3, should we need to update the original $c_r^k$?

Yes, we should update $c_r^k$ with the gradient $\Delta c_r^k$ in line 8. We will add an explicit update step ($c_r^k ← c_r^k - \lambda\Delta c_r^k)$ to Algorithm 3 in the next version.

4. In the reference, some formats are different from others, e.g., Lines 440-441.

Thank you for pointing out the reference format issue! We will ensure reference format consistency in the next version.

5. In Table 3, why Diffusion Classifier Variant performs worse than the original Diffusion Classifier?

Diffusion Classifier is originally designed for single-label classification tasks and thus performs more poorly on the multi-object tasks we consider. To apply Diffusion Classifier on our setting, we directly condition the generative model using compound prompts (e.g., “a photo of a dog and a cat”), which we refer to as the Diffusion Classifier baseline in our experiments. We also further explore a single object variant of Diffusion Classifier for this compound setting, where we condition the generative model on individual concepts (e.g., “a photo of a dog”) and select the two classes with the highest score. Details of both methods are detailed in A.5. Since the Diffusion Classifier Variant uses individual concepts to fit multi-concept images, it has worse performance than Diffusion Classifier baseline.

6. How to implement object discovery with Slot Attention? Is it possible to infer object number and locate objects only through masks extracted by Slot Attention.

Thank you for the insightful question! To enable slot attention to predict object locations with supervision, we decode slot representations into object coordinates and enforce a coordinate prediction MSE loss by comparing the decoded object coordinates with ground truth, where we use Hungarian Algorithm to align decoded coordinates and ground truth. To infer object number, during training, we can enforce empty slots to predict a fixed coordinate [1,1] (the normalized rightmost corner of the image), while object slots predict the actual object coordinates. At inference time, if the inferred coordinates are greater than a threshold close to [1,1], we determine these slots as empty, while the rest are considered to contain objects. This approach allows Slot Attention to infer both object number and their locations.

We thank the reviewer for the constructive feedback and comments! Please see our response below about your concerns.

1. CelebA images contain only a limited number of attributes (typically 2–3). How does the proposed method perform on tasks involving a greater number of attributes?

Thank you for the insightful question! To demonstrate the effectiveness of our method on a greater number of attributes, we conducted an experiment on CelebA using 4 attributes (Black Hair, Eyeglasses, Smiling, and Wearing Hat), given our time and computational constraints. As shown in the table below, our model consistently outperforms baseline models in both in-distribution and out-of-distribution accuracy. These results demonstrate the potential of our approach to handle more than just 2–3 attributes effectively.

2. For CelebA, it may be more appropriate to use the dataset from SeqDeepFake.

Thank you for suggesting this interesting paper and dataset! After reviewing SeqDeepFake, we don’t think that this dataset is directly applicable for our binary multi-label classification task. Our method requires binary labels for a fixed set of attributes, whereas SeqDeepFake provides edited attribute sequences with varying attribute annotation across images. Nevertheless, we find that the image captioning task in SeqDeepFake is very related and interesting, and would like to explore if our model can detect edited attribute sequences in future work. We will discuss the relevance of SeqDeepFake in the related work section in the next version of our paper.

3. For the custom small-animal dataset, providing statistical characteristics such as distribution and ensuring a more consistent and diverse set of categories would make the evaluation more convincing.

Thank you for your helpful suggestion! To clarify, our custom small-animal dataset consists of 20 images containing a cat and a dog, 22 images containing a cat and a rabbit, and 28 images containing a dog and a rabbit. We will include this dataset distribution description in the next version to enhance transparency and improve the evaluation's clarity.

4. Lack sufficient discussion on the relationship between image captioning models like BLIP-2. The authors should clarify the advantages of their approach compared to existing image captioning models.

Thank you for highlighting this connection to related work! By using pretrained text-to-image generative models (e.g., Stable Diffusion), our model can tackle image captioning tasks like BLIP-2. However, our approach is applicable to a broader range of scene understanding tasks other than image captioning. For example, by conditioning on object coordinates, our approach can perform object discovery tasks and even enable generalization to more complex scenes (many more objects) than seen at training. This flexibility and generalizability distinguishes our approach from traditional image captioning models. We appreciate your suggestion and will include a more detailed discussion in the next version.

5. In line 146, it would be clearer to use the full name, 'Energy-Based Model,' instead of the abbreviation when mentioning it for the first time.

Thank you for pointing out the abbreviation issue! We will fix it in the next version.

6. In the zero-shot tasks, why does the diffusion classifier variant perform worse than the original version?

Diffusion Classifier is originally designed for single-label classification tasks and thus performs more poorly on the multi-object tasks we consider. To apply Diffusion Classifier on our setting, we directly condition the generative model using compound prompts (e.g., “a photo of a dog and a cat”), as detailed in A.5, which we refer to as the Diffusion Classifier baseline in our experiments. We also further explore a single object variant of Diffusion Classifier for this compound setting, where we condition the generative model on individual concepts (e.g., “a photo of a dog”) and select the two classes with the highest score. Details of both methods are detailed in A.5. Since the Diffusion Classifier Variant uses individual concepts to fit multi-concept images, it has worse performance than Diffusion Classifier baseline.