Measuring And Improving Engagement of Text-to-Image Generation Models

We sincerely thank the reviewer for the insightful review and feedback. We try to address the reviewer's questions and comments in detail below:

---

The paper lacks discussion of some relevant works, other works on using human feedback to improve image generation models should be considered

---

We have tried to discuss works related to human feedback in the current draft (we cover that again below). Further, as suggested by the reviewer, we will expand the discussion of works related to utilising human feedback to train image generation models in the next paper draft. We discuss the different works below:

To align image generation with human preferences, reinforcement learning with human feedback (RLHF) has been utilised for text-to-image generation models [1, 2]. In these approaches, the latent image generation part of the diffusion model (either UNet or a Transformer) is trained using a reward model in the case of DDPO or using user preferences directly in the case of DPO. Both these approaches involve collecting human preference datasets.

Some more recent works are discussed below (as suggested by the reviewer):

Liang et al. [3] introduce a dataset RichHF-18K and a multimodal transformer model (RAHF) that aims to predict detailed human feedback, such as identifying implausible image regions and misaligned text. They collected rich human feedback over 18,000 images from various users. Then they tune the RAHF model to predict implausibility scores, heatmaps of artifact locations, and text-image misalignment. The predicted feedback is utilized to enhance image generation quality through targeted finetuning.

Du et al. [4] curate a large-scale dataset consisting of over one million generated advertising images, each annotated with human feedback regarding their suitability for advertising purposes. This dataset is created using a diverse selection of products from JD.com, featuring generated advertising images alongside corresponding product images with transparent backgrounds and meticulously crafted prompts by professional designers. Additionally, the paper introduces a multimodal model trained on this dataset to simulate human feedback of advertising suitability and automate image evaluation. This model utilizes an efficient recurrent generation process to produce high-quality advertisement images.

Clark et al. [5] propose a method for gradient-based reward finetuning based on differentiation in the diffusion sampling process. This approach, Direct Reward Fine-Tuning (DRaFT) allows for optimising diffusion models by incorporating differentiable rewards based on human feedback. They demonstrate the application of this method on a variety of reward functions, such as aesthetic score [6], PickScore [7] and Human Preference Score [8].

Liang et al. [3], Du et al. [4], Clark et al. [5] use simulated human feedback to refine the image generation process for specific image quality dimensions of realism [3,4,5], aesthetics [4,5], visual appeal [5] or advertising suitability [4].

Further, several human preference datasets for text-to-image generation have been collected in literature. These include Pick-a-Pic dataset [7], dataset generated from the Stable Foundation Discord channel [8], ImageReward dataset [9] etc. The human preferences in these datasets have been collected by explicitly asking humans to state their choices. These datasets are often accompanied with their own metrics for human preference alignment such as PickScore [7], Human Preference Score [8], ImageReward score [9] etc. While the authors of these research works have shown that these metrics are better aligned with human preferences when compared with conventional metrics such as CLIP score, BLIP score, Aesthetic Score, etc, however, they still need not align with viewer engagement of images. Hence, in our work, we use implicit data about human preferences derived from engagement metrics such as likes. Thus, we explore holistic and user-centric metrics of engagement that measure dimensions beyond traditional image quality. Our contribution is unique in the sense that we prioritize viewer engagement as the core feedback, allowing for engagement-optimized image generation that directly aligns with how users interact with visual content online. While traditional metrics like aesthetics, FID, ImageReward, image realism help in making the images look real, aesthetic and structurally correct, optimizing and measuring viewer engagement of images has potential applications in personalization, efficient A/B testing, recommender systems, and even public awareness and information campaigns.

References (continued)

---

Why was SD 1.4 chosen for train-time optimization, as it is one of the weakest models?

---

Thank you for suggesting this experiment. In the paper, we performed train-time optimisations on SD 1.4 due to resource constraints. SD 1.4 being of relatively smaller size enabled us to conduct thorough experiments in various training configurations. As kindly suggested by the reviewer, during the rebuttal phase, we conducted experiments to apply train-time optimisation over more recent and strong text-to-image models, like Pixart-Sigma and SDXL (512 resolution). Specifically, we performed preferred finetuning on high engagement images (Section 4.2). Results reveal the impact of improvement in image engagement after training.

.

Does KPI refer to the normalized likes of an image? So the regression value of KPI is just used in training phase?

---

Yes, KPI refers to the account-normalised likes of the image which reflects its engagement level. It serves as ground truth for training EngageNet for the image engagement prediction (regression) task. EngageNet’s predicted scores serve as a metric to judge the engagement level of images generated by various text-to-image models in the Engagement Arena. For maintaining uniformity and improving clarity, we would rename KPI to engagement in the updated draft of the paper.

---

latency of the proposed components, such as the Retrieval module

---

For the retrieval module, we use FAISS framework to index the DistilBERT vector embeddings of captions belonging to images in the high engagement data subset of the EngagingImageNet train data. Next, for every image caption in the low performance subset of the test data, we retrieve the semantically most similar caption from the corpus of high-performing images.

Once the vector index is created, average retrieval time per query = 0.1852 seconds / query (Averaged across ~2K queries). Memory consumption: 1.1 GB GPU RAM; Hardware: Nvidia A100-SXM4

---

The paper would benefit from a more detailed explanation of the DDPO method utilized, including the hyper-parameters and loss terms

---

Here, we provide details of the DDPO method adopted for aligning Stable Diffusion with engagement. Some of these are discussed are discussed in Section 4.3 and Appendix G.1 of the paper. We will expand and highlight the details better in the next draft version.

The denoising process in diffusion models is a multi-step recursive process with a pre-specified finite number of steps. In DDPO [1], this denoising process is viewed as a finite horizon Markov Decision Process (MDP), where the state comprises the current context, the number of steps left in the process, and the current denoised image. The action to be taken is to predict the next image using this state.
The image forming the initial state is sampled from a standard normal distribution. Mathematically, a finite horizon MDP is defined as a tuple $\{T, S, A, P, R\}$, where these components are defined as:

1. $T$ is the horizon or the number of steps in the MDP.

2. $S$ is the state space. Here it comprises three components: the context $c$ (i.e., the text prompt used for conditioning the diffusion model), the current number of steps left in the denoising process $t$, and the current denoised image representation (a given vector encoding of the image) $x_t$.
   The initial or starting state has the context $c_0$ given as input, the number of steps left at the beginning $t_0 = T$, and the initial image representation is sampled from a normal distribution of appropriate dimension, $x_0 \sim \mathcal{N}(0, I)$.

3. $A$ is the action space, and here it is the space comprising all image representations $x$. If $x$ is a $d$-dimensional vector, then $A = \mathbb{R}^d$.

4. $P : S \times A \rightarrow \Delta(S)$ is the transition function. Here, we specify $P$ separately for each of the three components of the state as: $$

   P_c = \delta(c_t), \quad P_t = \delta(t - 1), \quad P_x = \delta(a_t)

$$ , where the current state is $(c_t, t, x_t)$, the current action $a_t = x_{t-1}$, and $\delta(\cdot)$ is the Dirac delta distribution.
5. $R : S \times A \rightarrow \mathbb{R}$ is the reward function that takes a state and action as input and returns a scalar reward.
We generate this scalar reward signal using EngageNet. EngageNet is used to estimate the user engagement of images generated by Stable Diffusion. The reward signal is then leveraged to guide Stable Diffusion to generate higher-engagement images, as illustrated in Figure 13a.

As the agent acts in the MDP, it produces trajectories, which are sequences of states and actions:
$\tau = (s_0, a_0, s_1, a_1, \dots, s_T, a_T).$ The reinforcement learning (RL) objective is for the agent to maximize the expected cumulative reward over trajectories sampled from its policy. Assuming a fixed sampler, the diffusion model generates a sample distribution $p_\theta(x_0 \mid c)$. The denoising diffusion RL objective is to maximize a reward signal $r$ which is defined based on the generated samples and their corresponding contexts:

$J_\text{DDRL}(\theta) = \mathbb{E}_{c \sim p(c), \, x_0 \sim p\theta(x_0 \mid c)} \big[r(x_0, c)\big]$

for some context distribution $p(c)$.

As suggested by the reviewer, we will include additional details about the hyperparameters and loss terms of the DDPO method in the next draft of the paper.

Hyperparameters:

• batch size = 8
• epochs = 100
• number of samples per epoch = 64
• learning rate = 5e-6
• weight decay = 1e-4
• clip ratio = 1e-4
• optimizer = AdamW

References:
[1] Black, K., Janner, M., Du, Y., Kostrikov, I., & Levine, S. Training Diffusion Models with Reinforcement Learning. In The Twelfth International Conference on Learning Representations.

---

lacks a user study or online experiment

---

We have initiated the effort to conduct a user study. We have started the process of Ethics Review Board approval. We aim to incorporate the results of the user study by the camera-ready version deadline.

---

discuss a broader range of use cases and scenarios, in-depth discussion on potential use cases

---

One of the key contributions of this paper is the training of EngageNet, an engagement-aware vision-language model (VLM) model for predicting viewer engagement over images. This provides a pathway for empowering a gamut of applications. We will add future work and applications in the next paper draft.

In the paper, we demonstrate two applications of the EngageNet model – (1) proposing Engagement Arena, an automated arena to benchmark the engagement of text-to-image models using EngageNet's predicted engagement scores, and (2) Leveraging EngageNet-based reward scores to optimize text-to-image generation for engagement.

Here, we also discuss broader applications of this work beyond the ones already demonstrated in the paper:

• Personalization, Campaign Optimization, and Recommender Systems: EngageNet model can aid marketers and advertisers by selecting/prescribing images among a pool of candidate assets that maximize viewer engagement, making it a vital tool for campaign optimization. EngageNet can be integrated with multimodal recommendation systems to improve the relevance and appeal of suggested images shown to users.

• Health and Public Awareness Campaigns: Generating engaging and compelling images for public awareness campaigns, such as health education, can amplify their effectiveness. There are many research works in the public health and medicine literature showing the positive outcomes of health interventions like engaging advertisements [1, 2].

• Infinite Personalization: Today, an ad delivery engine has to choose amongst a fixed set of candidates while delivering an ad to an incoming visitor. Thanks to innovations in generative AI models like LLMs and Diffusion models, we can imagine generating many variants of an advertisement. For each incoming visitor, we can use Engagenet to score the variants, thus enabling infinite personalization, where every individual can receive bespoke content, maximizing relevance and engagement.

• Efficient A/B Testing: A/B testing is a cornerstone of optimizing marketing campaigns. However, due to the complexity and resource-intensive nature of A/B tests, few marketers are actually able to use them. Among those who do engage in A/B testing, the data-intensive requirements necessary to reach statistical significance often limit them to testing only a few variants at a time. Thus, marketers can employ EngageNet to shortlist and prioritize images with high engagement potential, leading to more efficient and data-driven A/B testing.

Future Directions

Building on the foundational work of this paper, future research could explore the following areas:

• Cross-platform engagement optimization: Extending the framework to incorporate diverse engagement metrics across platforms such as Instagram, YouTube, etc., capturing a broader spectrum of user preferences.
• Explaining content factors leading to higher engagement: Understanding why certain visuals generate higher engagement is crucial for both creators and researchers. Future work could involve enhancing EngageNet and Engagement Arena to capture subtle engagement dimensions such as emotional resonance, visual complexity, and long-term memorability.
• Prescribing Content Strategies to Creators: Beyond explaining engagement factors, future iterations of EngageNet could actively prescribe actionable strategies to creators. For instance, it could suggest specific attributes (e.g., color schemes, composition, or visual themes) that are likely to increase engagement for a particular audience or context, thus empowering creators to produce high-performing content efficiently.
• Video Engagement Optimization: Extending the proposed framework to handle video content could provide insights into optimizing video advertisements, storytelling, and social media clips for engagement. It involves considering dynamic elements like pacing, transitions, and viewer retention metrics, which play a critical role in video content performance.
• Fashion industry: The fashion industry could benefit from integrating engagement-aware content generation into virtual try-on systems. Models could generate visuals tailored to consumer preferences, encouraging them to explore and interact with virtual products. This could also enhance online shopping experiences by presenting customers with styles predicted to resonate with their personal tastes.
• Enhanced Recommender Systems: Engagement-aware images have the potential to enhance recommendation models significantly. By integrating these visuals into recommender systems across domains like retail, food delivery, and streaming services, businesses can present more appealing and relevant suggestions to users, thereby boosting engagement and conversions.

References (continued...)

We are thankful to the reviewer for their encouraging and constructive feedback. Below, we address the reviewer’s questions and comments in detail:

---

Definition of viewer engagement

---

In the context of this paper, viewer engagement refers to the measurable interactions and that a viewer exhibits towards an image, on a given digital platform. These interactions reflect the degree to which the image content resonates with the viewer and elicits actionable behavior. Precisely, in the EngagingImageNet dataset, we consider the account-normalised likes received on a social media post as the metric for viewer engagement.

---

Detailed statistics of the dataset

---

We discuss dataset details and statistics in the following sections of the paper:

• Section 2: It lists the number of tweets extracted, number of media tweets, number of accounts, time period of collection, counts, average engagement and engagement thresholds for bucketisation of tweets into high, medium, low categories.
• Table 1: It presents a comparison of EngagingImageNet data size and associated metadata with other datasets containing image preferences
• Appendix D: It discusses the filtering steps used to prepare EngageNet training data, thresholds for filtering, tweet length, time period, number of accounts and dataset size after filtering,
• Appendix E: Table 6 presents average number of objects contained, average aesthetic score and average CLIP score of high and low engagement images in the EngagingImageNet dataset. Figure 6 shows that high and low engagement images have a similar distribution of aesthetic scores, further providing evidence that existing text-to-image evaluation metrics like aesthetic score may not be aligned with actual image engagement. Figure 7 displays plots showing variation of number of tweets and likes with time for a few companies in the EngagingImageNet, indicating that engagement may follow seasonal patterns for some companies. We have also added Figure 8, which features a sunburst diagram illustrating the distribution of tweet topics within the EngagingImageNet dataset, showcasing its wide variety of diverse topics.

If you believe any additional analyses could strengthen the paper, we would be happy to incorporate the same into the next draft.

---

how is the temporal information being utilized?

---

• We provide temporal information (posting date) as input to the EngageNet model as our experiments indicate that temporal information also plays a key role in the engagement of an image.
• Impact of temporal information on Pearson Correlation: In Section 3.2 (lines 270-275), we show results of the ablation study to assess the impact of temporal information. If posting date is not provided as input to the EngageNet model, the correlation of EngageNet’s predictions with actual likes drops (0.62 to 0.53).
• Seasonal Patterns of Engagement: Figure 7 displays plots showing variation of number of tweets and likes with time for a few companies in the EngagingImageNet dataset, which reveal seasonal patterns in social media engagement, further corroborating the significance of temporal information.

---

The paper does not clarify whether any collected data contain identifiable information.

---

We have discussed PII in Appendix H. We will highlight this better. Concretely, we have implemented measures to remove personally identifiable information (PII). Our dataset is compiled without collecting any personal data, focusing solely on public, non-individualized information. All references to specific users or personal identifiers have been removed. Additionally, we collect only aggregate metrics (e.g., overall user interaction data) to measure engagement trends without compromising individual privacy. Further, we plan to open-source EngagingImageNet in phases starting with smaller releases, engaging with the community continuously such that any issues if present are brought to notice and resolved faster.

We sincerely thank the reviewer for their thoughtful and encouraging feedback. Below, we address the reviewer’s questions and comments in detail:

---

bias toward corporate content, more diverse data sources would provide a broader foundation

---

Can EngageNet be adapted to predict other engagement metrics ?

---

As suggested by the reviewer, we perform experiments to check EngageNet’s generalization to other social media platforms, engagement metrics and data sources. Specifically, we use the FlickrUser dataset [1] which contains ~500K images from ~2.5K users of the popular photo-sharing platform Flickr. For each image, it contains engagement metrics such as “number of favorites”, “number of views”, etc.

Reasons for picking this dataset:

• To check engagement prediction on non-corporate content.
• To check generalization capability across others metrics and social media platforms

Out of a total dataset of 500K images, we use only 15K images for finetuning and we use a test set of size 5K.

• Firstly, as a baseline, we finetune the pretrained LLaVA-1.5 Vision-Language Model (VLM) to predict the “normalised favorites” (i.e. “number of favorites” / “number of views”) as the engagement score of an image, given contextual information about the image (such as upload date, description, tags, etc.). This model achieves a correlation of only 0.23 with the actual “normalised favorites” of the test set images.
• Next, we finetune our EngageNet model, an engagement-aware VLM that inherently understands image aspects that contribute to engagement, for the same task as above. We find that our finetuned EngageNet model achieves a strong correlation of 0.53 with the actual “normalised favorites” of images, despite giving minimal training data.
• This stark performance gap between the two models can be attributed to EngageNet’s prior learnings about image engagement, which enables it to exhibit strong generalization capability across different engagement metrics, social media platforms and even non-corporate content.

References:
[1] Abdullahu, F., & Grabner, H. (2025). Commonly Interesting Images. In European Conference on Computer Vision (pp. 180-198). Springer, Cham.

References:

• [1] Wakefield, M. A., Loken, B., & Hornik, R. C. (2010). Use of mass media campaigns to change health behaviour. Lancet (London, England), 376(9748), 1261–1271. https://doi.org/10.1016/S0140-6736(10)60809-4
• [2] Kite, J., Chan, L., MacKay, K., Corbett, L., Reyes-Marcelino, G., Nguyen, B., ... & Freeman, B. (2023). A model of social media effects in public health communication campaigns: systematic review. Journal of Medical Internet Research, 25, e46345.

What are the further directions in this research topic? Will the EngageNet also be appliable to other research domains, such as multimodal item recommendation?

One of the key contributions of this paper is the training of EngageNet, an engagement-aware vision-language model (VLM) model for predicting viewer engagement over images. This provides a pathway for empowering a gamut of applications. We will add future work and applications in the next paper draft.

In the paper, we demonstrate two applications of the EngageNet model – (1) proposing Engagement Arena, an automated arena to benchmark the engagement of text-to-image models using EngageNet's predicted engagement scores, and (2) Leveraging EngageNet-based reward scores to optimize text-to-image generation for engagement.

Here, we also discuss broader applications of this work beyond the ones already demonstrated in the paper:

• Personalization, Campaign Optimization, and Recommender Systems: EngageNet model can aid marketers and advertisers by selecting/prescribing images among a pool of candidate assets that maximize viewer engagement, making it a vital tool for campaign optimization. EngageNet can be integrated with multimodal recommendation systems to improve the relevance and appeal of suggested images shown to users.

• Health and Public Awareness Campaigns: Generating engaging and compelling images for public awareness campaigns, such as health education, can amplify their effectiveness. There are many research works in the public health and medicine literature showing the positive outcomes of health interventions like engaging advertisements [1, 2].

• Infinite Personalization: Today, an ad delivery engine has to choose amongst a fixed set of candidates while delivering an ad to an incoming visitor. Thanks to innovations in generative AI models like LLMs and Diffusion models, we can imagine generating many variants of an advertisement. For each incoming visitor, we can use Engagenet to score the variants, thus enabling infinite personalization, where every individual can receive bespoke content, maximizing relevance and engagement.

• Efficient A/B Testing: A/B testing is a cornerstone of optimizing marketing campaigns. However, due to the complexity and resource-intensive nature of A/B tests, few marketers are actually able to use them. Among those who do engage in A/B testing, the data-intensive requirements necessary to reach statistical significance often limit them to testing only a few variants at a time. Thus, marketers can employ EngageNet to shortlist and prioritize images with high engagement potential, leading to more efficient and data-driven A/B testing.

Future Directions

Building on the foundational work of this paper, future research could explore the following areas:

• Cross-platform engagement optimization: Extending the framework to incorporate diverse engagement metrics across platforms such as Instagram, YouTube, etc., capturing a broader spectrum of user preferences.
• Explaining content factors leading to higher engagement: Understanding why certain visuals generate higher engagement is crucial for both creators and researchers. Future work could involve enhancing EngageNet and Engagement Arena to capture subtle engagement dimensions such as emotional resonance, visual complexity, and long-term memorability.
• Prescribing Content Strategies to Creators: Beyond explaining engagement factors, future iterations of EngageNet could actively prescribe actionable strategies to creators. For instance, it could suggest specific attributes (e.g., color schemes, composition, or visual themes) that are likely to increase engagement for a particular audience or context, thus empowering creators to produce high-performing content efficiently.
• Video Engagement Optimization: Extending the proposed framework to handle video content could provide insights into optimizing video advertisements, storytelling, and social media clips for engagement. It involves considering dynamic elements like pacing, transitions, and viewer retention metrics, which play a critical role in video content performance.
• Fashion industry: The fashion industry could benefit from integrating engagement-aware content generation into virtual try-on systems. Models could generate visuals tailored to consumer preferences, encouraging them to explore and interact with virtual products. This could also enhance online shopping experiences by presenting customers with styles predicted to resonate with their personal tastes.
• Enhanced Recommender Systems: Engagement-aware images have the potential to enhance recommendation models significantly. By integrating these visuals into recommender systems across domains like retail, food delivery, and streaming services, businesses can present more appealing and relevant suggestions to users, thereby boosting engagement and conversions.

We sincerely thank the reviewer for their encouraging review and feedback. We try to address the reviewer's questions below:

---

Given the situation that many labs may lack computational resources, how to make it possible for these researchers to follow your work?

---

Thank you for raising this important point regarding accessibility for researchers with limited computational resources. We recognize the importance of ensuring that our work can be built upon by a broad community, including labs and student researchers with constrained computational resources.

• While the initial dataset collection and benchmarking phases required substantial computational resources, the subsequent steps, including engagement-driven retrieval, finetuning, and DDPO (for engagement-aligned image generation), can be conducted on relatively modest infrastructure. Therefore, by open-sourcing EngagementArena, EngagingImageNet, and EngageNet, we encourage the research community (especially student researchers) to build upon the released resources and submit their innovations in better engagement-driven retrieval, finetuning and engagement-aligned image generation mechanisms.

• We aim to make EngagementArena as an open source platforms like LMSys and labinthewild.org where students and researchers can contribute and run experiments on public infra.

  • LMSys is supported by public donations (https://lmsys.org/donations/) from organizations like Huggingface, MBZUAI, Kaggle, NVIDIA, etc. Similarly, a few organizations have privately reached out to kindly lend their resources to make a similar open-source infrastructure. We plan to release this on acceptance.

• Metrics more aligned with engagement than the proposed one: Just as the image aesthetic score prediction landscape evolved from initial papers to more refined and accurate models, researchers can use the EngagingImageNet dataset and build upon the EngageNet model to develop better metrics for engagement. This iterative improvement process will refine the understanding of what drives viewer engagement and lead to more aligned and nuanced models over time.