Social Reward: Evaluating and Enhancing Generative AI through Million-User Feedback from an Online Creative Community

The response to the weakness #2:
We agree with your suggestion of comparing the model performance trained with other contrastive learning based losses. In fact, we were running experiments with different losses and observed limited impact of loss on model accuracy.
For instance, model trained with more conventional Contrastive loss produced 67.1 % in pairwise accuracy metric on test set. Suggested by you InfoNCE yielded a higher 68.5% on the test set. But since the best performance was achieved by Triplet loss (69.7%) it was the one included in the paper.
Model performance comparison with different loss functions will be enclosed to the paper submission as an appendix.

Please accept our heartfelt appreciation for your diligent and insightful review of our manuscript. Your detailed observations and suggestions have shed light on several aspects of our paper that require further attention and refinement, and we believe they will significantly enhance the overall quality of the manuscript.
With respect to the reasoning behind prioritizing remixes as user feedback metric (weakness #1) there are several considerations:

• Since our platform revolves around creative editing and sharing images, users are naturally motivated to express themselves visually through remixes. This can be illustrated by remixes being made by our user base 25 times more than like and 40 times more than comment. Thus remix provides a much richer source of user feedback.
• Remixes inherently involve a more active form of engagement compared to more passive actions like views, comments or likes. Users not only appreciate the image but actively participate in the creative process by remixing it, showcasing a deeper level of involvement and investment in the content thus adding a layer of quality to the engagement metric.
• Directly using views as a metric may not accurately reflect user engagement because views only measure passive consumption, providing no indication of user interaction or appreciation. It doesn't distinguish between users who briefly glance at an image and those who actively engage with it.

The response to the weakness #3:
In order to address you question better please find below a detailed description of PA Image-Social dataset collection procedure. This description will also be added to the Appendix part of our manuscript.

• We start from discovering the most popular synthetic images in Platform A in terms of accrued remixes within a 10 month time frame. All images are sorted by remix number in descending order.
• Top 1% of the images are selected as positive images (in paper this approach is called “content signal”, section 3.2). Additionally, in case if the image had been remixed by an influencer user it is also added to the set of positive images (in paper this approach is called “creator signal”, section 3.2).
• Then we retrieve prompts associated with collected positive images.
• For the obtained prompts, we retrieve all the images that received at least one remix and construct the distribution of view count before the first remix.
• We cut that distribution by 99% percentile and use the obtained view count threshold in the next steps.
• For the same prompts, we retrieve all the images that received exactly zero remixes (there are large amounts of “zero remix” images due to “Pareto like” nature of content diffusion in social platforms).
• For every prompt, we filter out “zero remix” images that are below view count threshold. In such a way, only images for which there is enough evidence of negative user feedback will be labeled as such.
• During data collection procedure some cases which cause inconsistencies are dropped, for example, we drop prompts for which there are no images other than the ones labeled as popular or there are no images that received zero remix etc.
• Additionally, we employ in-house mature content detection models to filter out images that fall within the NSFW category.

We want to extend our gratitude for the time and effort you have invested in reviewing our work. Your in-depth analysis of our manuscript shows genuine interest in our research and serves as a great source of inspiration for future creative endeavors.

• You correctly pointed out that the purpose of this work isn't the introduction of the novel RLHF framework, but establishing and exploring new dimension in human preference reward modeling.
• With respect to the data privacy question Platform A is a company that complies with GDPR, CCPA, and other data protection legislation, and is collecting, storing, and processing users' data in accordance with the consent received. The Platform also allows the users to opt-out of certain processing purposes, as requested.

The response to the weakness #4:
Thank you for your very valid remark regarding the longitudinal design of data collection.
Although our study may not establish causation or the direction of effects in the same way a classic longitudinal design would, the extended data collection period can still provide valuable insights into how variables may influence each other over time. Due to this consideration of the importance of having a representative dataset in temporal dimension, in the scope of this work we collected data from a prolonged period of 10 months in the scope.
We plan to dive into a longitudinal study for user preference shift and how that might lead to our score model updating over time, as the scope of our next work.

The response to the weakness #3:
While social bias presents an important concern in terms of collecting candid and reliable responses in self-report surveys, we believe its impact within the scope of our study is limited. We leverage organic user behavior as a source of our data. Users are not asked to participate in the survey, solely implicit anonymous feedback in the form of users organically choosing images for editing purposes is used.
However, we fully share the concern of accounting for biases in data collection procedure from such noisy source as social network and specifically address such biases as Content Exposure Time Bias, User Follower Base Bias and Prompt Bias (section 3.2 Data Collection).

The response to the weakness #2:
For more visuals of generated image comparison between baseline and fine-tuned models please refer to figure 3 of supplementary material appendix file named “SocialReward_Appendix_ICLR_2024_Conference_Submissions.pdf”.
We believe numbers reported in Table 6 showcase significant performance gain of fine tuned model over the baseline. The percentage values in the table signify the percentage of times an image generated by a fine-tuned model is favored in terms of specific metric over the one generated by baseline model for the given prompt. We can observe that with exception of CLIP alignment, all scores are favoring fine tuned versions in at least 63% of cases and up to 75% of cases.
With respect to the consistent patterns observed in text-to-image models fine-tuned using SocialReward here are some of them:

1. Improved quality:

• Overall quality/fidelity of generated images visibly improved which can be noticed in decreased number of artifacts, distorted objects and shapes, more minimalistic and higher aesthetic quality of generated images

2. Complete Display of Main Objects:

• The model consistently generates images where the main object is fully displayed, minimizing instances of partial or cropped representations. This characteristic ensures that the central focus of the image is readily available for creative manipulation.

3. Preference for Minimal Text:

• There is a notable tendency for the model to avoid generating unnecessary text in the images, emphasizing a visual-first approach and compensating for the weakness of current models to generate text without artifacts.

4. Reduced Collages and Merged Images:

• There is a decrease in the generation of complex image collages or the merging of multiple images. This design choice provides users with a clean canvas, facilitating easier integration of additional elements during the creative editing phase.

5. Clarity in Backgrounds, Especially in Natural Scenes:

• The model consistently produces images with clear and distinct backgrounds, particularly noticeable in natural scenery. The absence of additional objects in the background enhances the usability of the generated images for creative projects, offering a blank slate for further customization.

We would like to express our appreciation for your thoughtful review, which can be instrumental in improving the quality of our work. In response to your feedback, we would like to address the specific points you raised.
The response to the weakness #1:
Under the relatively small scale of collected user preferences in Pick-a-Pic dataset we mainly meant comparison with our PA Image-Social dataset. For instance, we have about 2.5 times more images (656K vs 1.7 M), about 5 times more image pairs (615 K vs 3M) and about 2.5 more prompts (38.5 K vs 104 K). Please refer to Table 1. But most importantly is 230 times difference in the number of users involved (6.4 K vs 1.5 M). This number is critical in the light of our research goal, evaluating the performance of text-to-image models within the context of social network popularity for creative editing, because of two reasons:

• It is necessary for the collected human preference to be from a large group of users to be deemed representative of community scale feedback
• The authors of Pick-a-Pick built a web application, where users can generate images and express their preferences on them. Each prompt-image pair is getting feedback from a single user (the one who generated the image) in contrast to ”collective feedback” (which implies multiple users’ independent feedback on the same image), that is the principle upon which PA Image-Social dataset is built.

We would like to express our sincere gratitude for your thoughtful and constructive review of our paper submission. We appreciate the time and effort you have invested in reviewing our work. With respect to the question you raised, firstly, let us note that our paper indeed can greatly benefit from incorporating more information regarding PA Image-Social dataset. On the second note, here are the details you requested:

• Popular (deemed as "positive") images constitute 8% of all images in PA Image-Social dataset. The main reason behind the smaller number of “positive” images in PA Image-Social dataset is typical “Pareto like” pattern observed in majority social networks where minority of the content gathers majority of community engagement.
• There are 174 influencer users (users are categorized as influential in our platform if they have a certain amount of followers and are frequently posting new content) whose remixing of generated images contributed to the dataset collection.
• Engagement levels of individual users, in our case, expectedly follow power law distribution logic (pattern which usually describes user activity in online social platforms). For instance, the least active 50% of our users generate about 30% of remixes. Whereas the most active 10% generate about 40% of remixes.
• We plan to include this information in the paper’s Appendix section.