InCoDe: Interpretable Compressed Descriptions For Image Generation

We thank reviewer DRti for their feedback. We leverage it to present new results. Below, we address their concerns and hope to have clarified their doubts. We are happy to provide further clarifications if needed.

Evaluation

Comparison to stablished TTI [...] not sure if lines 342 to 351 refer to Table 2. Please clarify.

Table 2 is indeed comparing our method to established work for text-to-image generation using diffusion models. Lines 342 to 351 describe the baselines used in this experiments and Section 3.2 (Composable generation paragraph) describes the results. We will add a link to the table in this paragraph to make it easier for the reader.

We include a new baseline in our table. We compare to Accuracy and F1 scores of Stable Diffusion XL v1 (SD XL), one of the most advanced open models (note that DallE-3 in Fig. 2 is private and owned by OpenAI). Due to time constraints, we evaluate SD XL using 500 generated samples for both the Bedroom and Churches datasets. Our results show that even a model approximately 6.6 times larger than ours still struggles to combine multiple concepts and only slightly improves upon the results from its predecessor, SD V1. Results are the following:

Compare it to an off-the-shelf diffusion model [original DDPM paper] conditioned on facial attributes of the CelebA dataset.

For CelebA with attributes, we followed the technique suggested by the reviewer in our initial submission. Specifically, we used an implementation of the Imagen model (a DDPM sampler) and trained it from scratch, conditioning on questions and answers based on CelebA attributes. Qualitative results can be found in Figure 18.

The fine-tuning procedure shown in Figure 4 was applied only to the two LSUN datasets, which are the most challenging. While training from scratch is a feasible option, it may result in lower image quality and longer training times compared to the fine-tuning approach we propose.

While we didn't have time to include this in the rebuttal, we propose, upon request, to compare CelebA trained from scratch with CelebA trained using our fine-tuning method in the final version of this work.

Questions

$\hat{Q}(\hat{X})$ instead of $\hat{Q}(X)$?

We believe $\hat{Q}({X})$ is adequate. Our Querier Answerer takes histories S (partial query-answer sets) and predicts all answers of queries applied to the real image X. Hence, there is never a generated sample $\hat{X}$ in this branch of the training pipeline.

Enforcing constraints in Eq. 2 in practice.

These constraints are enforced by construction. The constraints indicate that 1) $q$ is the output of a querier function, modeled with our Querier and 2) the posterior is modeled by a function $f$, which corresponds to the Query Answerer.

To aid clarity, we substitute "subject to" by "where", in Equation 2 (and equivalents).

Please let us know if further clarification is needed.

Do we need a decoder if we are interested in the modified image?

Our ultimate interest is to generate images that respect the semantics dictated by queries and their answers. This indeed allows us to modify generated images by changing answers to queries. However, the Querier and Query Answerer are employed for data representation rather than generation.

The diffusion model is trained to generate images based on descriptions formed from query-answer pairs. The Querier's role is to select the most informative queries that create a concise, information-dense, query-based description. It is trained using the Decoder (Query Answerer), which predicts answers to all queries without requiring an image X as input, only histories S.

Equation 2 outlines the objective function for training the Querier, where the image generator is not involved, as its training is decoupled from that of the Querier.

What are DT-IC and TopK-IC, are they related to L342-351?

These baselines are entropy-based methods for selecting informative queries. They are described in more detail in lines 322-341. Please let us know if further clarification is needed.

What are Stab. D and Stru. D, are they related to L342-351?

Stab. D (Stable Diffusion)(SD V1 in the revised version) and Stru. D (Structured Diffusion) are text-to-image diffusion-based methods. Their description can indeed be found in lines 342-351. Again, let us know if further clarification is needed.

Thank you for responding, I will raise my score.

We thank the reviewer for their both their comments. Their constructive feedback let us to perform experiments that will indeed make this work more complete. Below we address all their concerns.

Theoretical foundations

Theoretical foundations are mainly taken from [1] and adapted.

This is true. We do not claim to have introduced Variational Information Pursuit (V-IP), which is the contribution of [1]. Instead, in our work we present an adaptation of V-IP that helps us solve the problem at hand: selecting the most informative queries to achieve succint query-answer descriptions in the context of data representation for image generation.

This paper presents algorithmic and experimental contributions, instead of theoretical ones. We propose a novel framework introducing new elements to achieve our objective, such as adapting the theoretical principles in [1] to the paradigm of generation or proposing a novel generative method.

[1] Chattopadhyay, Aditya et al. “Variational Information Pursuit for Interpretable Predictions.” International Conference on Learning Representations (2023).

Evaluation

Used models are Relatively obsolete [...] There are newer and stronger models, perhaps more capable.

We agree with the reviewer that this is a relevant comparison. Therefore, we compare Accuracy and F1 scores with Stable Diffusion XL v1 (SD XL), one of the most advanced open models (note that DallE-3 in Fig. 2 is private and owned by OpenAI). Due to time constraints, we evaluate SD XL using 500 generated samples for both the Bedroom and Churches datasets. Our results show that even a model approximately 6.6 times larger than ours still struggles to combine multiple concepts and only slightly improves upon the results from its predecessor, SD V1. Results are as follows:

BLIP does not express generated images quality [...] FID should be considered.

FID may not be an ideal quality metric for this task. Our base model and baseline, SD V1, has been trained with LAION-5B Dataset, of which LSUN data is just a small portion of approximately 0.02%.

FID tends to be more informative when comparing similar distributions, but here, we are comparing images from LSUN (a specific subset) to a model trained on a broader distribution (LAION-5B).

Therefore, we anticipate FID scores higher than usual.

InCoDe, a wrapper for SD V1, was fine-tuned with a low learning rate to preserve the prior distribution learned by SD V1. As a result, we expect the FID to be improved, but not necessarily excellent. However, this does not necessarily mean that the generated images are not realistic.

In our figures we show samples generated by different models (Figs. 1, 2, 5, 12, 13, 15, 16, 17) where good generation quality can be appreciated. We can provide more random samples of different models at the request of the reviewer.

We conducted experiments by center-cropping the 2000 test images for both Bedroom and Churches datasets, and computing FID with our base model SD V1 and InCoDe. The results show significant improvements in the Churches dataset and modest gains in the Bedroom dataset:

FID was computed with this codebase: https://github.com/mseitzer/pytorch-fid. Results will be included in the final version of the paper at the request of the reviewer.

Limitations

Method requires pre-defined set of attributes [...] less feasable in real-world datasets.

While we would argue that Bedroom or Church datasets are real-world datasets, it is true that they have less variety in their semantic elements than Imagenet, for instance.

In the case of widely variable datasets, if a user would want to capture certain tasks, this would require either larger query-sets, or richer answers to questions (non-binary). While we do not present experiments to that extent, there are no inherent limitations for our framework to accept these changes.

However, as it is true for any generative model, to capture more semantic elements we would likely need to increase the model and dataset sizes, along with longer training times.

Minor

Error in axis names Fig. 8?

This is true, we correct the figure in the revised version.

We thank the reviewer for their accurate observations. Below we address their concerns.

Limitted Flexibility

While control is improved, it restricts expressiveness.

This is accurate. This work prioritizes specialization over generalization. Fortunately, users have flexibility in designing queries, allowing them to capture virtually any desired semantics. We discuss this topic in more detail in the General Comments section, under Generalization.

We are glad to extend this discussion at the reviewer's request, and/or to include this discussion in the main paper.

Real-World Scenarios

real-world image generation tasks my introduce more variability and ambiguity [...] might be challenging [...] could limit framework's effectiveness.

While we would argue that the Bedroom and Church datasets are real-world datasets, it is true that they have less variety in their semantic elements than Imagenet, for instance.

In the case of widely variable datasets, if a user would want to capture certain tasks, it would require either larger query-sets, or richer answers to questions (non-binary). While we do not present experiments to that extent, there are no inherent limitations for our framework to accept these changes.

However, as it is true for any generative model, to capture more semantic elements we would likely need to increase the model and dataset sizes, along with longer training times.

Reliance on VQA

framework depends on accurate VQA responses [...] errors in VQA could affect performance.

We acknowledge this limitation. The effect of errors in VQA models is indeed an interesting study for future work. Fortunately, off-the-shelf VQA models often offer reasonable performance and they are the best systematic way to provide answers we have identified for textual queries. But as indicated, they may introduce error. That said, we’d like to underscore the following:

1. VQA models are primarily employed to annotate unlabeled training data. By using VQA models to answer every query for each sample in our training set, we can work with images that lack annotations, giving our framework extensive flexibility across various image distributions. However, in datasets such as CelebA, where each image is already paired with manually annotated attributes, we can bypass VQA-related errors during training by using the provided annotations.

2. At test-time, a VQA model is not strictly necessary. For any single image, query responses can be manually provided by a user, based on either (i) their semantic preferences or (ii) their visual analysis of a reference image.

3. Queries are not always textual, so models like BLIP may not be needed in every case. For example, in the expriments shown in the supplementary material (Sec. A.1.1), location-based queries are answered by referencing a specific part of the image, eliminating the need for a VQA model.

Finally, there has been interesting recent work on fine-tuning general-purpose VQA models to enable explainable decisions in image classification datasets such as ImageNet and Places365 [1]. These method could potentially be beneficial at improving the VQA capabilities of the framework introduced in this paper.

[1] Chattopadhyay, Aditya et al. “Bootstrapping Variational Information Pursuit with Large Language and Vision Models for Interpretable Image Classification.” International Conference on Learning Representations (2024).

We sincerely thank reviewer DhcA for their positive and constructive feedback on this work. Below, we address their concerns.

Related Work

Related works section is somewhat oversimplified [...] state of interpretability and controllability in image diffusion models.

We appreciate the reviewer’s suggestion and have updated our manuscript accordingly. We have added more citations and connected each control domain to its interpretability form. Please let us know if you recommend further expansion or a different direction.

Generalization

it is not clear whether InCoDe can generalize to more "free-form" types of controllability [...] users use image diffusion models to generate highly diverse images. [...] out of luck if these images do not comply with one of the pre-defined set of queries.

This is true. This work prioritizes specialization over generalization. Fortunately, users have flexibility in designing queries, allowing them to capture virtually any desired semantics.

We discuss this further in the section on Generalization of General Comments.

We are glad to extend this discussion at the reviewer's request, and/or to include this discussion in the main paper.