ArtWhisperer: A Dataset for Characterizing Human-AI Interactions in Artistic Creations

Questions answered:

• As mentioned in the discussion and appendix (A.14), we have demonstrated that our dataset can be used to train synthetic users by fine-tuning an LLM (we used an MT0 model here). The validation we did suggests the fine-tuned model can behave similarly to human users with respect to score trajectory / steerability. While we leave much of this to future work, it is easily conceivable that synthetic users trained on user interaction data (e.g., our dataset) can be used to fine-tune VLM models like stable diffusion to increase steerability (e.g., using our metric to assess steerability).
  • We have included more discussion on this point in the revision.
• As you point out, we were unfortunately unable to collect long term data on repeated user interactions. In the controlled user study, we did collect interactions across a number of tasks. From this data, we did not find a significant increase in steerability (after correcting for steerability differences across images). While intuitively we do believe steerability should increase over time, our dataset suggests that this increase is more subtle. We suspect additional feedback given to the user (e.g., instructions to improve / correct their submitted prompts) may more significantly increase steerability, but we did not test this hypothesis here.
• Our intuition is that model quality is a significant factor in steerability. In particular, higher quality image output increases steerability (up to a point, as image quality is not the only factor in steerability).
  • This is easily seen in the following extreme case – a user who does not look at the model output (i.e., the model output quality is low) can still steer the model through random changes to the prompt – whether the change improves or decreases the score is seemingly random, however, limiting steerability (this assumes that the model encodes the user prompt in the image, despite its low visual quality; if this does not happen, the scoring will be even more random).
• We deliberately decided to limit user input to just the prompt. This was done for a few reasons (added this additional discussion to Section 2 and the Appendix A.4 in the revision):
  1. We wanted all users who enter the same prompt for a given image to see the same output.
  2. We wanted to limit the complexity of the task for users less/unfamiliar with text-to-image models.
  3. We wanted users to generate new prompts, not simply resample new seeds until getting lucky. (While users could still employ a version of this “random resampling” strategy by making small changes to their prompts, we did not want to encourage it through a seed parameter. This random resampling strategy, while useful in practice, is not such an interesting result for research purposes as it is easy to simulate random resampling strategies without any user input.)

There are multiple uses for our dataset and data collection procedure as discussed throughout the paper and discussion sections:

1. Better understanding of user prompting strategies and diversity of user prompts
   • To our knowledge, this is the first public dataset capturing how users iterate on their prompt designs for text to image models. This dataset lets us study how people perform this iteration and how we might improve models to accommodate human usage (e.g., see Sections 2, 3 and Figures 2,4,6).
     • For example, we find using our dataset that users arrive at diverse prompts for the same image – they do not converge to the same image description (Section 3 and Figure 6).
     • As another example, we also find evidence using our dataset that there is a distinct style each user has (though the effect of style on steerability may depend more on specific image) (Section 3 and Figure 6).
2. Quantifying model alignment to users in a rigorous way through a steerability metric.
   • We use our dataset as the basis for defining a steerability metric, and propose our data collection procedure (or an analog with simulated users) can be used to assess models that require human interaction. (Section 4, Figure 7)
3. Synthetic user agents
   • As we describe in the discussion section, we believe an important use case of our dataset is to create synthetic users. We test and validate this approach where we fine-tune an MT0-large model to replicate a human prompter (using numbering from the revision: Appendix A.14, Figures 21,22,23). Creating synthetic users that employ real human prompting strategies / vocabulary has a clear benefit to model training: we can incorporate this into loss metrics to increase alignment between models and human users across an interaction.

Questions answered:

• The seed selection (and target image selection) procedure is used to find (1) a target image and (2) a user seed such that user prompting variation does not significantly alter the user generated image similarity to the target. (Minimal) additional (subjective) quality control is performed by the authors to ensure image content is reasonably aligned with the prompt.
• IRB approval is a standard process for ensuring ethical research when involving human subjects.

Thank you again for your time! We hope our responses have addressed your questions and that you would consider raising your score. Please let us know if you have additional questions and we are happy to respond further.

Questions answered:

• We have clarified the two datasets better in the revision (see Section 2.3).
• Comparison of the generated images from the target prompt with the real Wikipedia images is added in the Appendix. We also added a sentence clarifying that the metric we use is calibrated such that using the Wikipedia caption gives a perfect score for the Wikipedia image (this calibration is also done for generated AI images).
• The reason for using Wikipedia images is to collect and compare the steerability of a broad variety of images – both AI art as well as real art (e.g., paintings or sculptures) and other natural images (cities, wildlife, etc.).
• Notation has been clarified throughout the paper. In Section 2.1, $x_i$ is used for Wikipedia images and $x_{i,1}$ and $x_{i,2}$ are used for AI-generated images. In Section 2.2, we refer to images using just $x$. In Section 3, we have changed notation to use $z$ instead of $x$ to describe image/text embeddings.
• Participant demographics were intentionally not collected to avoid identifying users.
  • In the controlled user study, demographics were collected and a supplemental figure has been added. Users were sampled through Prolific to a fixed 50/50 split of male and female users. In the revision, we have included histograms of the age, sex, and ethnicity distributions in Figure 11 in the Appendix (A.6). We did not find significant differences in steerability across these demographics.
• While comparison with additional models would be nice, we believe our results are still a valuable contribution to the community.
  • The two stable diffusion models we use are in fact quite different and produce differing results given the same prompt (note, for example, we needed to re-select the user seed for the second model to ensure the model produced a similar enough image when using the target prompt; using the same seed as the original model could result in vastly different images). So, we believe the comparison we provide is interesting in itself.
  • Moreover, it also provides evidence of differences in steerability between two models and that we can measure these differences using our data collection method and proposed steerability metric.
• Steerability results between Wikipedia and AI art datasets are compared in Figure 7. Note the differences between the “AI Art” (AI Art dataset) and “Real Images” (Wikipedia dataset) indicate that real images are more steerable.
• We did not find any significant differences between paid and unpaid users.

Thank you again for your time! We hope our responses have addressed your questions and that you would consider raising your score. Please let us know if you have additional questions and we are happy to respond further.

Influence of returning users on steerability

• In the controlled user study, we did not observe a significant change in the model’s steerability (controlling for target image) over time. This suggests that user experience did not have a large impact on steerability.
• Our work provides insight into only the immediate steerability of a model, and provides evidence that absent specific feedback (e.g., instructions on how to steer a model), people do not improve steerability very quickly (within 5 examples). It might be possible that a much longer experience can improve steerability.

Knowledge of an image subject improves steerability

• This is partially due to the prompt supplied by the user (the user can provide a more specific prompt) and partially due to limitations of our image similarity metric (in some cases, the metric can be weighted to the subject matter, and referencing the correct subject in the prompt can lead to a higher steerability score). A figure (Figure 18 in the Appendix) was added to the revision. It shows a significant difference when the user is familiar with the subject of the target image (defined as the user referencing the key subject in at least one of the prompts they submit).
• People were instructed to and did attempt to maximize the synthetic score. We did not collect separate comments from users.
• As discussed in Appendix A.4, users in the controlled user study were given a bonus for achieving higher scores.
  • Users generally took the task seriously. A prefiltering step was used that removed data where a user did not achieve beyond a certain minimum score (these were identified as cases where a user was simply exploring and not trying to complete the task)
• As variance of image output (for a given target prompt) increases, so does the steerability score (larger steerability score corresponds to more difficulty in steering the model). In other words, if repeatedly sampling a model with the same prompt can produce a wide range of outputs (high variance), then the model is likely less steerable for the content in those generated images. We added a figure (Figure 17 in the Appendix) and a brief discussion on this point in the Appendix of the revision.
• The variance largely depends on the image – for some images it is quite large while for others it can be quite small. This is the reason for our procedure to select the target image to show users and the fixed seed we provide for users. The target image and seed are selected to minimize the difference between the target image and other images generated using different seeds with the same target prompt.
  • The intuition here (why $t_{i_1^*}$ is more representative) is the following. $t_{i_1^*}$ is the target image closest to the center (using median) of the set of candidate target images (using a fixed target prompt). While the model may produce a wide range of images for the given target prompt, we pick the target image that is the “average” one (using median instead of mean to reduce the effect of outliers). By picking this “average” image, we also hope to reduce the difference of images generated using similar prompts. We have clarified this in the revision.

Thank you again for your time! We hope our responses have addressed your questions and that you would consider raising your score. Please let us know if you have additional questions and we are happy to respond further.