Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty

uncertain whether it would be a suitable fit for CVPR

This is a ICLR 2025 submission and not a CVPR submission. Our main contributions are not primarily within the computer vision space but in AI/ML. Could you please recalibrate the scores with this information?

I question the technical contribution of the paper

Please see the general reply for a summary of the technical contributions.

many previous methods have proposed self-correction techniques to improve generated images, as demonstrated in works such as [1], [2], and [3]...

The papers [1, 2, 3] focus on improving the alignment of T2I models to input prompts. Whereas this paper focuses on iterative clarification/refinement of an under specified prompt. These are two very different objectives and while both very important, this paper focuses on the latter objective.

What is the difference between the iterative procedure used in the proposed methods and that of related works? There is no direct comparison with all related works.

As mentioned above, the related works the reviewer listed and our work solve very different problems. Those related works aim to develop better methods to perform the T2I task itself, assuming the text description is detailed enough to be used to evaluate the quality of the generated images, so that they can iteratively generate better images based on the same text description. While in our work, we tackle that challenge of how to actively gather information from the user in an interactive way, when the user doesn’t write prompts that fully specify the image they would like to generate. Given that the problems are so different, the solutions are not comparable. While both methods are iterative, they are iterative in entirely different ways: we have the user in the loop. Hence, we believe it is not appropriate to compare our work with the cited papers. We assure to discuss this distinction in the related work.

It would be beneficial if the authors could compare their methods with these works and explain why the proposed method has a competitive edge.

We will add the clarification stated above to the paper along with a reference to the mentioned works, however we are unable to compare them as they do not study the same problem as we do in this paper.

If users make errors when inputting their data, is the proposed model able to detect and correct these mistakes?

We assume the reviewer is asking if the user can update the answers to previous questions in the case that the user changes their mind - ( Ie. they input “the house is red” but then decide they want to change the house to blue color) but kindly let us know if we misunderstood. The answer is yes the user can choose to directly edit the prompt, directly edit the belief graph or simply input a sentence to the agent describing their updated decision on the attributes of the image.

Regarding reasoning ability, if a user requests advanced generation requirements, such as "a robot executes a task," is the proposed method capable of handling spatially-aware generation?

We are not entirely sure what the reviewer meant by “reasoning ability” and “advanced generation requirements” in this context. We assume the reviewer is asking whether our T2I agent is able to identify and incorporate spatial relations mentioned in a prompt, a clarification question/answer or the belief graph. The answer is yes. In fact, a critical part of the belief graph is the spatial relations between entities in our design of the agent.

If the approach involves making prompt engineering more detailed than the initial prompt, it seems intuitive that the resulting image would be more accurately generated as intended. Could you elaborate on what aspects of this method are novel or distinct beyond providing more detailed prompts?

Current prompt expansion techniques typically aim to make images more aesthetically pleasing or diverse [1]. Our work can be considered as an approach for agents to actively gather information from users so that the prompt can be adaptively and sequentially expanded in a way that aligns with user intents. The followings are some aspects that we consider but current prompt expansion techniques do not:

• how to reduce the uncertainty a model has about underspecified user prompts;
• generating clarification questions / belief graphs and having users in the loop to answer clarification questions or manipulate model beliefs;
• adaptively refining the prompt based on interactions with the user to align with user intents.

Please see the general reply for the full description of novelty and contribution.

[1] Siddhartha Datta, Alexander Ku, Deepak Ramachandran, Peter Anderson. Prompt Expansion for Adaptive Text-to-Image Generation. 2023. https://arxiv.org/abs/2312.16720

How do you determine which characteristics are crucial for accurately generating each image? A clearer and more logical explanation of the criteria or process used for this selection would be helpful.

We are not entirely sure what the reviewer is asking and have two hypotheses on what the reviewer means: (1) “characteristics” refers to characteristics of the image, and the reviewer is asking which characteristics are crucial for determining if the image is accurate and why; (2) “characteristics” refers to characteristics of the thing to be clarified, and the reviewer is asking about the criteria for determining what is should to be clarified (in order to generate accurate images) and why. Please let us know if we misunderstood.

(1) which characteristics are crucial for determining if the image is accurate and why:

The important characteristics are 1) the inclusion of the entities and their attributes that the user intends to generate, 2) the image reflects the relations between entities that the user intends to generate, 3) the image is semantically consistent with the ground truth description of the image (the ground truth description describes in detail what the user intends to generate).

Logical explanation on why we believe these characteristics matters:

The crucial characteristics for accurate image generation are reflected by T2I metrics. The two well-known metrics are DSG scores and VQA scores. DSG focuses on composition (e.g., entities, attributes) and spatial relations of the generated image, whereas VQA focuses more on overall semantic understanding of the generated image in relation to the prompt. If the above characteristics are met, both scores will be high, meaning that the image accurately reflects user intents.

(2) the criteria for determining what should be clarified and why:

We detailed the criteria of what clarification questions should be asked in Section 4.1. We constructed 3 agent prototypes and each has its own question-asking criteria, which the reviewer can find in Section 4.1.2 and more details in the corresponding appendix. For example, the characteristics Ag1 uses in its question-asking criteria include (a) the uncertainty the agent has about an entity’s existence, the value of an attribute for an entity, and relation between entities, (b) the importance of the entity, attribute or relation relevant to the prompt. Both characteristics are part of the belief graph, which is generated by LLMs using in context learning.

Why use those characteristics? The belief graph directly captures the composition and spatial relations considered in DSG, since it includes possible entities, their attributes and spatial relations between entities. (a) If the agent is very uncertain about the existence of an entity, it is likely that the DSG score can increase once the user provides the information of whether the entity exists. Similarly for attributes and relations. (b) If the agent is uncertain about an attribute of an entity but the attribute is not so important, the agent probably shouldn’t ask about that entity, since the ground truth description probably does not even mention that attribute. For example, for prompt “a red hat”, the agent can be unsure about the exact shade of red, but the exact shade likely does not matter since the user might like any red color.

To make sure these characteristics are indeed crucial, we verified the performance of those agents in experiments. All agents achieve higher VQA and DSG scores than the standard T2I approach.

1. Many studies [1] focus on directly editing images...

The paper [1] and our work solve very different problems. [1] focuses on improving the alignment between generated images and input prompts via iterative self-correction. [1] assumes that the input prompt is detailed enough such that it is possible to assess the alignment and perform self correction. [1] mentions image editing as a side product, but it relies heavily on the user to know the “language” for editing (e.g., Move up “monkey #1”) which is used for self-correction. Our work, on the other hand, aims to design the overall human-AI interaction framework with T2I as a case study (see 1st paragraph of introduction). Our work focuses on proactive information gathering to clarify an underspecified prompt and belief graph for interpretability, which we believe to be generally useful for agentic behaviors during human-AI interaction.

There exists other image editing works such as Instruct-Imagen, which passively accepts new instructions from users, as opposed to actively collecting information from users. Such editing methods can be included in our human-AI interface to provide a diverse set of controls for users. But in essence, image editing and our work are two entirely orthogonal modalities of the T2I system. Image editing fundamentally aims to improve the model (such that it can accept new instructions) and we aim to provide agency for T2I. Hence we don’t think our work and image editing are comparable.

2. ...why not input a detailed prompt from the beginning? If a prompt contains sufficient information about the goal image, wouldn't it be possible to generate a good image from the start? If multi-turn interaction isn't strictly necessary because sufficiently detailed prompts still result in images different from the goal image, the proposed method lacks convincing justification for its necessity.

We agree that sometimes multi-turn interaction is not strictly necessary if the prompt has sufficient details. However, in our user study, we observed that 54.5% of human subjects found many prompt iterations to be a frequent issue (“the model does not immediately generate what I want, so I have to iteratively change my input prompt many times”). Only 5.4% found no issues with prompt iterations. Moreover, at least 90% of human subjects found our agents and their belief graphs helpful for their T2I workflow. (see both abstract, intro and section 5.4) This means our agentic framework will be useful for the majority of potential T2I users. We believe this is a strong justification for why our work is necessary.

3. Furthermore, in many cases, no matter how well the prompt is crafted, the generated image may not turn out as intended. Doesn't this method fail to address such situations?

Yes, our proposed framework does not address the prompt-image alignment problem, but studies the problem of iterative clarification/refinement of an underspecified prompt. Those two are two very important and orthogonal objectives.

The agent prototypes we described in the paper are highly modular: the agents use frozen T2I models to generate images based on the prompts that the agent updated. Therefore the quality of the final images are directly influenced by the ability of the T2I model employed. This also means that when a better off-the-shelf T2I model becomes available, it can be directly plugged into the agents and the system will achieve better performance without any additional adaptation.

Even with a highly detailed and unambiguous prompt, the system may not produce a perfect image if the underlying T2I model is inadequate. Conversely, even with a perfect T2I model, uncertainties or ambiguities in the prompt can hinder the system's performance. Many works aim to address the issue of T2I prompt to image alignment [1,2,3,4,5,6] but almost none for the second in the literature of T2I. Our work focuses on the prompt underspecification problem and aims to design agents that proactively ask users clarification questions and present beliefs to gather information about what the user wants to generate. The belief graph and the final prompt of our proposed agent could have accurate entities, attributes and relationships and in some cases T2I agents fail to follow parts of the prompts or create distorted assets in the images [4].

Table 1 and Table 2 we show evaluation metrics over T2T, T2I and I2I similarity. We can see the T2T scores as a form of system ablation, since we exclude the T2I model and only perform similarity on the caption input to the T2I model. This removes any error or prior information that the T2I model adds. In Table 1 and Table 2 per the T2T metrics, we have achieved a 92%+ similarity score, and therefore a better T2I model for the agent can immediately boost the performance of the overall agent system.

We thank the reviewer for acknowledging the novelty and simplicity of our work.

It's a nice thing that this article want to do, but the solution is so boring.

We urge the reviewer to follow Reviewer Guidelines (https://iclr.cc/Conferences/2019/Reviewer_Guidelines) when rating the paper. “Boring” is a subjective feeling, not a rubric to make recommendations on a research paper.

The solution is too simple and very uninventive… Beliefstate It's a former concept, and there's nothing inherently innovative about it.

We believe that a simple solution is a merit, not a disadvantage especially when dealing with HCI and User Interfaces, simplicity is essential to drive utility. For the novelty of this work, please see the general reply, where we also summarized the core differences between the conventional belief states and our graph-based T2I agent belief.

Put in a new concept and then come back with a self-explanatory statement.

Could you please clarify what you mean by this?

Focus on D: implementation details and show your novelty. The main text is too shallow.

Thank you for the suggestion. We will add the pseudocode for our sequential ICL approach for belief graph generation, which summarizes Appendix D and emphasizes the novelty of the belief graph generation method. We will also update the paper to clarify other aspects of the novelty of this work detailed in the general reply.

The work frames the problem as the agent updating beliefs according to a fixed world state in the user's mind. However, the user may not have a clear idea in mind (i.e., the user may not have a pre-defined world state). In contast, the user might want to get some inspirations from the system without constraints. This use case has not been considered in the system design. This limitation might affect users such as artists. This was discussed in L534-539, but no suggestions were proposed.

This is a great question. In practice, our system does work for the case where users do not have a clear image in mind. The clarification questions and belief states offer ideas on what could be possible in the image. For example, Ag2 asks “What type of berries would you like on the cake (e.g., strawberries, blueberries, raspberries, mixed)?” in Figure 4 and shows the possible entities that could appear in Figure 14 (e.g., the “people” entity not mentioned in the prompt). These aspects can give users inspiration on what to generate. Moreover, the user can directly edit the belief graph to change the existence (or attribute values) of an entity/attribute/relation mentioned in the prompt, so that the images do not have to satisfy any constraints imposed by the prompt itself.

A belief state of an autonomous agent is a distribution over “static” world states at every turn, but the world can change whenever the human user intervenes and gives the agent new observations (e.g., answers to agent questions, edits to the belief or the prompt itself). Imagine modeling the belief of a robot, the world the robot is in can change (e.g., someone pushes the robot) and the change is reflected in the observations.

The limitations on static human intention stem from theoretical considerations since we are interested in mathematically defining the objective of an intelligent agent (Section A), but the actual T2I agents we built are not limited by this assumption since these agents do not directly optimize that objective. We will make this clearer in the paper.

As for L534-539, this is discussing the limitation on distinguishing different types of uncertainty. We did give concrete suggestions: the agent can directly ask the user about their preference if the uncertainty is epistemic, and acknowledge the randomness if the uncertainty is aleatory. To predict these two types of uncertainty, one option is to use the Gaussian process probing methods developed in Wang et al., 2024b. The exact method and implementation is beyond the scope of a conference paper so we leave it as future work.

The work uses LLMs in Ag2 and Ag3 without fine-tuning. However, this work does not explore trained LLMs (VLMs) with either image data or trajectories that include asking questions. This indicates that the LLM is purely exploring in text space. The exploration might be sub-optimal since exploration on text space might be different than exploration with images in mind.

Thank you for the suggestion. We agree that using LLMs to explore in the text space can be sub-optimal. However, it is important to note that even this provides sufficient gains over baselines, demonstrating the efficacy of our system.

We agree that fine-tuning LLM/VLM on text/image trajectories that include asking questions can potentially improve the performance of the agent. A rigorous investigation is beyond the scope of this conference paper. Future work potentially includes

1. collecting data such as gold-standard trajectories or annotations on the quality of trajectories of human-agent conversations;
2. developing approaches to fine-tune the model on multi-turn trajectories of images and text.

We will add a discussion about this important point the reviewer raised.

Please understand that a conference paper has limited space. While we would like to make our work as perfect as possible, we had to prioritize the most important aspects (see the general reply) within the page constraints.

2. (continued)
   • Interpretability and controllability: The graph structure makes our agent belief more interpretable than traditional belief states, since we can visualize and progressively disclose the graph to the human user. Moreover, each node or relation in the belief graph has associated descriptions, making it easy for the user to understand and potentially edit every component of the belief graph. In our human studies, about 85% of raters found the belief graph useful. To the best of our knowledge, our work is the first to use the graph-based belief state for human-AI interaction.
3. Automated evaluation of T2I agents: We propose a novel automated evaluation approach for T2I agents using self-play. The agent interacts with a simulated user that has access to the original image and its long caption. See Section 5.1 (and C.4) for the full details of how the simulated user is constructed. This evaluation pipeline is easy to use and can help the future development of T2I agents.
4. DesignBench: We envision that a significant fraction of T2I users are artists and designers, and it is important to ensure that T2I agents are evaluated for these use cases. Hence we create DesignBench, featuring photo-realism, animation and multiple styles with short and long captions. DesignBench can be directly plugged into our automated evaluation to streamline the evaluation process.

We will update our manuscript to emphasize the above novelty, including changing the ambiguous name “belief state” to “belief graph” when we refer to T2I agent beliefs, and adding the pseudocode for our sequential ICL approach for belief graph generation.

Again, we are making major contributions to the human-AI interaction subfield of AI/ML. Improving computer vision models is NOT the focus of our work. We urge the area chair and reviewers to take this into consideration when making the accept/reject recommendation.

[1] Pasula HM, Zettlemoyer LS, Kaelbling LP. Learning symbolic models of stochastic domains. Journal of Artificial Intelligence Research. 2007 Jul 21;29:309-52.

[2] Xia V, Wang Z, Allen K, Silver T, Kaelbling LP. Learning sparse relational transition models. International Conference on Learning Representations (ICLR), 2019.

[3] https://en.wikipedia.org/wiki/Stanford_Research_Institute_Problem_Solver

[4] https://en.wikipedia.org/wiki/Planning_Domain_Definition_Language

[5] Y. Alkhazraji, M. Frorath, M. Grützner, M. Helmert, T. Liebetraut, R. Mattmüller, M. Ortlieb, J. Seipp, T. Springenberg, P. Stahl, and J. Wülfing. Pyperplan. https://doi.org/10.5281/zenodo. 3700819, 2020. URL https://doi.org/10.5281/zenodo.3700819.

[6] Caelan R. Garrett, Tomás Lozano-Pérez, Leslie P. Kaelbling. PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning, International Conference on Automated Planning and Scheduling (ICAPS), 2020.

[7] Caelan R. Garrett, et al. "Online replanning in belief space for partially observable task and motion problems." 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.

Dear reviewers and AC,

The discussion period is ending soon. Please let us know if you have any remaining comments. We really appreciate your time and effort.

-Authors