Preference Adaptive and Sequential Text-to-Image Generation

We appreciate the reviewer’s helpful feedback, as well as their recognition of the significance and originality of our work and the value of sharing our distinctive dataset. Below, we respond to the reviewer’s comments.

A: Model weights and training code release

We are currently reviewing options for releasing our user model in addition to the already open-sourced complete datasets.

B: “...proposed agent is trained with offline RL (as opposed to online) is worth highlighting more clearly..”

This is a good point. Offline RL was chosen because: 1) It allows (also) training directly on the rich human rater dataset, capturing authentic complexities. Online RL with real people is impractical, forcing reliance on simplified user models. 2) Offline RL is faster, avoiding slow environment interactions and inference with large user models/LLMs required by online RL. We will highlight this.

C: “...is the user choice model a different model?”, “the main text uses $R$ to denote user utility, but the same quantity seems to be called $s$ in the appendix”

In the paper we assume equivalence assumption between choice and preference and consistency in utility. A trained score function $s_\theta$ builds the utility function (Eq 3), scoring image slates (per prompt/user type). Applying softmax over utility scores (Eq 4) yields a user choice probability consistent with utility and the preference model. A learned temperature adds flexibility while keeping the assumptions (scaling the entire vector of utility scores by a constant $\tau_\theta$ does not alter the ranking consistency). Thus, both models share $s_\theta$ with the choice model derived from utility output. $R$ refers to the overall utility concept, while $s$ is the specific learned score function implementing it.

D: “The reported 99% confidence intervals in Figure 5 may be overly pessimistic”

Confidence intervals don't overlap vs. the base model, but some overlap exists when comparing our method with different data types. We'll run another rater study assessing full trajectory improvement (final vs. first turn), expecting clearer results as cumulative improvement is what we ultimately care about. Updates during discussion.

E: “How can accuracy on Pick-a-Pic and Spearman correlation on HPS be computed if the user type is considered”

The test sets for Pick-a-Pic and HPS are designed such that each sample contains a specific annotator's samples. To assess the posterior of the annotator over user types, we sampled a subset (of size 3) from the full set of annotated samples, then calculated the accuracy or rank of the remaining held-out samples and averaged the results over the posterior. We found this comparable to using the full set for the posterior computation. This detail was omitted due to space and it will be added in revision.

F. Questions For Authors:

1. See comment A.
2. For human data creation, we used Gemini 1.5 Flash as the candidate generator. Instead of a trained selector, a random selector proposed the slate, picking up to one prompt per category from the generator. This combination provides quality expansions and broad exploration, crucial for building an unambiguous offline latent MDP dataset.
3. Our user model consists of the utility component as well as the user choice model that translates this reward into user choices, as is standard in utility theory, choice modeling, econometrics, etc. Indeed, for a fixed choice model the user model is a generalized (user specific, set-based) preference model.
4. Our evaluation consists of 60 human raters, each experiment was evaluated over 200 trajectories. Exact metrics will be in the appendix.
5. Agent trained with sparse rewards (only at final 5th round), determined by the learned user utility function. Human rater data was also labeled using this same learned utility function post-collection.
6. Eq 2 is the generalized preference probability of preferring image set $\ell^*$ over the other sets. This is not an equality but a description of the preference probability.
7. Since the user model's goal is mimicking users for data generation, our metrics focus on performance across user types (averaged over posterior). However, we will add original baseline scores from the Pick-a-Pic/HPS papers for comparison.
8. The y-axis represents the accuracy outlined in the subtitles, while the x-axis indicates the number of user types used in the user model. The shaded area is 95% CI. We will clarify this in the paper. Appendix D.3 details cross-turn accuracy: preference accuracy between selected images at consecutive test set turns.
9. Softmax sampling: The Eq 3 aggregation function can be max, average, softmax sampling, etc. Softmax sampling means selecting an element randomly based on the softmax distribution over scores. We chose it over max to add randomness reflecting user selection variability observed during data collection and sequential T2I.
10. see comment C.

We thank the reviewer for their detailed review and hope that the following response addresses the reviewer’s concerns.

A: "The lack of experiments with other SOTA T2I generators.."

Response: To address this, we are running additional test-time experiments evaluating models our agent wasn't trained on. This includes a rater study using our method with Flux.1 and comparing our method to a GPT4.5-based agent. Results will be shared during the discussion period.

B: "Could you compare your with other preference-alignment reward models?", "lacking quantitative and objective metrics"

Response: We agree further analysis is helpful. We will add additional evaluation metrics to our experiments during the author-reviewer discussion phase (by next week) including: user model score, FID, and LPIPS.

C: “Strict word limits..”

Response: The constraint arises from the pretrained CLIP text encoders used in the user-score model, which are limited to 77 tokens. To stay below this threshold after five turns, we set the initial prompt to 12 words. Additionally, the text-to-image (T2I) model has a limited context capacity, restricting the level of detail that can be included. Despite this, the model still accommodates a variety of starting prompts that can evolve into detailed and rich expansions. Moreover, keeping initial prompts short shifts the focus toward feedback derived from user choices rather than critiques, which aligns more naturally with generalizing the standard single-turn preference setup. We intentionally minimize the influence of critiques to maintain this balance. That said, using improved encoders like LLM2CLIP is left for future exploration.

D: “What is the intention of not allowing users to see intermediate prompts?”

Response: PASTA refines prompts preserving user intent. Users choose the prompt-image pair best matching their initial prompt, preserving core elements. This method keeps the sequence of generated images closely tied to the content of the starting prompt, reducing the likelihood of producing images that stray too far from the user's starting point (e.g., beginning with a prompt about a train and ending up with an image of a dog). We experimented with displaying intermediate prompts to raters but found that presenting these prompts especially given their often complex nature placed too great a cognitive load on raters. We also believe this could compromise the quality of feedback provided by raters. Prompt refinement and expansion are natural (and commonly used) interaction modes for users engaged in iterative image generation. Surfacing the prompts as suggestions along with the images is something to be explored, but might require more "controlled" prompt expansion to elicit proper responses. We will elaborate further in our paper.

E: “What is the essence of recognizing user’s category or type in a multi-turn generation process instead of explicitly using textual interactions?”

Response: PASTA doesn't identify user types explicitly. Types only simulate consistent preferences in synthetic data creation. The agent adjusts to the user by leveraging the interaction history, adapting to their preferences. Our method offers advantages over textual instructions because users are often unaware of the prompt possibilities provided by the agent, and selecting images is significantly more user-friendly than drafting instructions. That said, PASTA can incorporate textual instructions by integrating them into the candidate generator’s prompt, allowing it to focus on specific categories or directions specified by the user. Our dataset includes rollouts (20%) with text critiques for potential model training. We kept the framework simple, leaving text integration for future work.

F: “Better / Worse / Same selection results between different turns could be influenced a lot by psychological factors”

Response: User feedback inconsistency due to psychological factors is expected. However, average human evaluations remain a valuable performance measure. Our results show the trained PASTA agent significantly outperforms a standard LLM.

G: “Is the “sample rater data” provided in the supplementary material the full version of your curated human rater data?”

Response: Our human rater data, collected with a random policy selector, consists of over 7k 5-step rollouts (>500k images), hundreds of prompts, approx. 100 raters. Simulated data: over 30k rollouts (>2.5M images). The supplementary sample is small due to size limits. A link to the full dataset will be provided in the final version of the paper.

H: “How do these prompts align with real-world usage..”

Response: Abstract prompts help discern user type preferences during interactions better than typical prompts (e.g., 'dog running'), which often have universally preferred styles and biases. Supplementary materials include full rollouts for typical T2I prompts.

We are grateful for the reviewer’s constructive feedback and for pointing out our dataset contribution as the first T2I dataset for multi-turn interaction setting. Bellow we address the concerns raised by the reviewer:

Comment 1: "This paper does not mention the scale of user annotations."

Response: Thank you for highlighting this oversight. Our human rater data consists of over 7,000 five-step rollouts (totaling more than 500,000 images), annotated by approximately 100 human raters using a random candidate selector. The simulated user data includes over 30,000 rollouts (exceeding 2.5 million images). We will add this information to the paper.

Comment 2: "The reviewers are concerned that the annotation scale might be too large, costly and not reusable."

Response: To encourage research in this novel setting, we are making all datasets used during training publicly available—we hope this will allow researchers who lack the resources to conduct such studies to benefit. Furthermore, we have proposed that exploiting trained user simulators, independent of large language models and easily trainable with the provided dataset, is an effective solution to mitigate data scarcity. Finally, we are sharing the synthetic dataset created by our user simulator to further assist researchers in addressing these challenges with greater ease.

Comment 3: “Generalizability of the method.”

Response: Thank you for raising this point. To address this we are now running additional experiments with different models during test-time (i.e., evaluating using models that our agent was not trained on). Specifically, we are conducting an additional rater study using our method with the Flux.1 T2I model. We are also running a study to compare our method to a GPT4.5-based model as an agent. We will comment here with these new results during the discussion period.

Comment 4: Analyze the synthetic data.

Response: Our approach to analyzing the synthetic data involves assessing the user model, which represents the sole distinction between the real human data and the data produced by the simulated user model. Visually, we can observe a clear distinction between user types by either scoring the top five highest-rated images in the HPS test set or by reviewing the rollouts of PASTA against the user model, both of which highlight distinct preferences across different user types. For a numerical perspective, we refer to Section 6.2, particularly Figure 3, where our user model achieves a prediction accuracy of 70% compared to real human raters, suggesting that the synthetic dataset effectively captures essential aspects of real user interactions. However, we agree with the reviewer’s perspective that further analysis is necessary. As such, we will add additional evaluation metrics to our experiments, including: user model score, FID, and LPIPS. We will comment here with these new results during the discussion period.

Comment 5: “...domain gap between single-turn and multi-turn preferences..”.

Response: Indeed, there is a distributional shift between single-turn and multi-turn preferences. However, our user model parameterization, encompassing both the user choice model and utility, relies on a single-turn score function $s_\theta$. Consequently, we can utilize single-turn datasets like Pick-a-Pic to augment the data available for training the user model. That said, due to the distributional disparity and our emphasis on the multi-turn context, these single-turn datasets are employed solely in the initial phase of the user model training to provide a stronger starting point for the subsequent phase training with the human rater multi-turn dataset, which is our primary focus. This point may not be entirely transparent in our exposition: we will elaborate further in our discussion of the user model training process in the paper.

Comment 6: “Why was the number of interaction turns set to five?”

Response: We settled on five steps to balance between providing a challenging enough interaction length and meeting the real-time expectation of users for relatively swift engagement with the agent. Exploring the (variable) optimal rollout lengths is something we have deferred to future research, given that the PASTA framework includes numerous components that warrant deeper study.

We appreciate your positive feedback on the novelty of our approach and our dataset contribution for the community. Bellow we address the concerns raised by the reviewer:

Comment 1: "...the performance gains from the user simulation require further scrutiny."

Response: The user simulator is used for two reasons: (1) generating synthetic data that is user-coherent in order to extend the real human rater dataset; and (2) labeling the human rater dataset. Our experiment with real users shows that adding additional synthetic data improves the model’s performance, indicating that our generated dataset is beneficial for training in real-world scenarios. In addition, we examine the ability of our user model to predict real users choices with 70% accuracy (Fig 3), which further indicates that our user simulator does capture some key dynamics of real user interactions. We will add a plot showing the user model score over time in order to add an additional performance evaluation metric. We will comment here with these new results during the discussion period.

Comment 2: "The dependency on particular pre-trained models(Gemini, Gemma, SDXL) might limit the generality of the results."

Response: Thank you for raising this point. In order to address it we are running additional experiments with different models during test-time (i.e., evaluating using models that our agent was not trained on). Specifically, we are running an additional rater study using our method with the Flux.1 T2I model. We will also compare our method to the use of the GPT4.5 model as an agent. We will comment here with these new results during the discussion period.