Measuring And Improving Persuasiveness Of Large Language Models

Thank you for your thoughtful review and suggestions on how we can improve the paper. They are very helpful for improving the overall work.

if the semantic meaning is consistently preserved between paired messages. for the generation tasks, it remains unclear if the generated outputs consistently adhere to the specified conditions.

The question about transsuaded text having similar meaning pertains to the degree to which AI can operate on messages without the sender’s supervision. This could be considered as “autonomy” of AI agent ([3] Hancock et al). We define nine types of transsuasion (Table-1) varying the autonomy of the AI agent. E.g. autonomy can be restricted to changing just the image (VisOnly), or just paraphrasing text (Parap).

In the table below, we show the similarity metrics between transsuaded tweets obtained or generated from (1) ground truth (GT) (2) GPT-4o (3) Ours(13B) (4) Vicuna(13B). We use the following metrics to test semantic similarity between transsuasion pairs of the ground truth and generated samples:

1. NER Match: The number of NERs that are same on both sides divided by total NERs
2. Factuality match: We prompt GPT-4o to assess if both tweets do not differ factually, we also ask it to give a confidence rating on a scale of 5 tell if it is sure about both the facts being different and only consider the pairs where the confidence is at least 4.
3. MetricsMatch: The proportion of pairs of tweets that follow the constraints we created in Table 1

The tables show that not only are the input pairs largely similar semantically, but also that the output pairs are more controlled. Thanks for the suggestion we have added this as table 15 in the appendix.

Win rate across topics, A deeper analysis into which domains the model excels in and where it falls short

We extracted topics from the username and Twitter bio using BERTopic. These topics were clustered and assigned a name by GPT-4o-mini and visualized using a sunburst graph (Figure 5 in the new version of the paper). Similarly, we also analyzed the topic distribution of tweets (Figure 6 in the new version of the paper).

Win rate across topics: In Figure 8, we present our highest and lowest average win rates against GPT-3.5, GPT-4, and GPT-4o across various topics. Our performance reaches its peak, with win rates close to 80%, on topics such as retail product innovation, e-commerce sales, and new launches. Conversely, on topics where we have the lowest win rates—such as market analysis, platform growth, and fashion in tech—GPT-4o performs comparably or slightly better.

characteristics of model-generated messages and the factors that improve their persuasiveness. Are there be any simple factors like the length of a message and the existence of hashtags or images

What makes a tweet more persuasive? We observe that the correlation between tweet length and persuasiveness is negligible (0.04, p-value = 4.88e-65), and linguistic features such as emojis, sentiment, and hashtags have minimal influence (see Figure 7 in the new version). However, we do observe certain brand specific insights in model generated tweets. To extract these brand specific insights we first cluster the embedded (using RoBERTa) tweets of a brand and summarize the insights from a cluster using gpt-4o-mini. We have added them below (also in appendix section B.5)

• Bulgari
  • Transsuaded tweets evoke strong emotional engagement and vivid imagery
    • Example: https://x.com/Bulgariofficial/status/1856736301657235947
  • Transsuaded tweets emphasize products rather than events.
    • Example: https://x.com/Bulgariofficial/status/1843573678736584907
  • Transsuaded tweets showcase a unique and innovative design element.
    • Example: https://x.com/Bulgariofficial/status/1846936730471129102

Similarly, we have extracted rules for some more brands and industries below

• Starbucks
  • Transsuaded tweets emphasize a seasonal theme or promotion.
  • Transsuaded tweets convey a personal experience or sentiment.
• Nike
  • Transsuaded tweets indicate urgency or time sensitivity
  • Transsuaded tweets include a specific date and time.
  • Transsuaded tweets emphasize a specific cultural or historical significance.
• Airbnb
  • Transsuaded tweets evoke a nature-centric experience.
  • Transsuaded tweets emphasize a specific location or city.
  • Transsuaded tweets emphasize local economic impact.

discussion on the experiment results

We acknowledge the need to provide deeper insights into the experimental results. Below, we address specific aspects of our work. We have also added these to the paper (Appendix Section-F, we will move it to the main paper in the camera-ready).

Scaling Trends for LLM Persuasion: Several studies [1,2] highlight an increasing correlation between model scale and persuasiveness. However, our fine-tuned 13B models (CS+BS+TS) with an ELO of 1304 outperforms significantly larger models on generative transsuasion, such as LLaMA-3-70B (1187) and GPT-4 (1213), GPT-4o (1251). Additionally, our 7B model (1099) surpasses GPT-3.5 (1092) in performance (refer to table 3 for detailed evaluations). These results indicate that persuasive capability is not solely determined by model scale. Instead, targeted training of smaller language models, when combined with scaling, can yield competitive or superior outcomes. From Table-3, we also see that GPT-4o beats GPT-4 by 38 elo and GPT-3.5 160 elo points on PersuasionArena, which correspond to a win rates 55.45% and 71% according to Bayes-Elo Calculation. This is in line with the OpenAI GPT-o1 model card [2], where the win rate of the latter is 78.1% on the ChangeMyView benchmark (compared to 71% on our arena).

Training regimes for transsuasion models: We ablate our experiments across various training regimes (finetuning (IFT), DPO), task combinations (BS+CS, BS+CS+TS), and the inclusion of self-generated explanations (Ours-Instruct). Our findings reveal that multi-task IFT (BS+CS+TS) achieves the best ELO (1304) on generative transsuasion. In contrast, DPO trained on TS samples performs slightly lower (1283) overall but demonstrates a marginal advantage of in similarity on NER Match(+2.8%), MetricsMatch (+2.5%), and FactualityMatch(+0.7%) (refer to Table 15). Training exclusively on BS and CS yields the highest performance for behavior simulation (62.2%). While the addition of self-generated explanations does not significantly affect BS, CS, or TS individually, it notably enhances performance on downstream tasks including Humans as judges of persuasion(+5.5%), marketing blogs simulation (+6%), and Audience specific transcreation (+3%) (refer to Tables 7, 9, and 10 respectively).

Transfer of Transsuasion to Other Tasks: To check the transfer of persuasion capabilities measured over Twitter to other domains, we test all models on 4 benchmarks: Humans as Judge (our study), Humans as Judge (Anthropic Persuasion study), Marketing blogs dwell time and views prediction, Audience specific transcreation. This also allows us to test the transfer power of Twitter-finetuned 13B model whose persuasion capabilities were developed over Twitter to other channels and domains.

1. Marketing blogs dwell time and views prediction: We improve 19% and 22% compared to base model on dwell time and views prediction respectively (refer to table 9)
2. Audience specific transcreation: We improve the performance ~2x compared to the base model on targeting (refer to Table 10)
3. Humans as judges (our study): we improve by 15%, 20% and 50% on on Upvote/Downvote classification, Reasoning classification, and Feedback perplexity compared to the base model (refer to table 7)
4. Humans as judge (Anthropic study): Our rank correlation is 0.47, ~6.5x more compared to the base model (0.07) (refer to table 8) These show that models trained on the task of transsuasion also transfer to completely unseen domains, channels, behaviors, and tasks. These findings demonstrate the robustness of our framework and the broader applicability of PersuasionArena and Bench across varied tasks involving persuasion.

Further to measure the robustness of PersuasionArena we collect the rankings from all metrics (8) of the (4) transfer benchmarks above, next we calculate the mean kendall tau coefficient between PersuasionArena these 8 rankings, this gives us a mean Kendall Tau coefficient of 0.69 which is considered strong. The consistent rankings of PersuasionArena and unseen domains further validate its importance.

What makes content more persuasive: We observe that the correlation between tweet length and persuasiveness is negligible (0.04, p-value = 4.88e-65), and linguistic features such as emojis, sentiment, and hashtags have minimal influence (see Figure 7 in the new version). However, we do observe certain brand specific insights in model generated tweets. To extract these brand specific insights we first cluster the embedded (using RoBERTa) tweets of a brand and summarize the insights from a cluster using gpt-4o-mini. We have added them in appendix section B.5

[1] Durmus, E., Lovitt, L., Tamkin, A., Ritchie, S., Clark, J., & Ganguli, D. (2024, April 9). Measuring the Persuasiveness of Language Models. https://www.anthropic.com/news/measuring-model-persuasiveness.

[2] Gpt-o1 system card." https://openai.com/index/openai-o1-system-card/

[3] Hancock et al, AI-Mediated Communication, JCMC, 2019

Thank you for the follow-up question.

The text of the trassuaded tweet depends on the degree of autonomy given to the LLM. The truthfulness is controlled by the following levers: (semantic similarity, edit similarity, image similarity, link match, NER match, and fact match). We have covered these aspects in Table 1 and in the table in our first response. Qualitatively, take the example of this transsuasion pair:

• Tweet: Among experience hosts, 50% are women, and 55% of home hosts are women. <HYPERLINK> <HYPERLINK> Image: an Infographic with two women in a kitchen laughing and talking.
• Transsuaded Tweet: Women make up a larger majority of home hosts in some countries, like New Zealand (70%), South Africa (63%) and the Philippines (61%). Santa Barbara has the most women experience hosts (65%). <HYPERLINK> Image: an Infographic telling percentage of experience hosts in Santa Barbara are women
• Hyperlink: https://news.airbnb.com/women-hosts-have-earned-nearly-20-billion-on-airbnb/

Here, the human expert picked up a fact different from the one pointed out in the original tweet (from the indicated URL) to make the tweet more persuasive. Similarly, an LLM has to be given some autonomy to change. It can be by picking up a different statement from a set of statements (as in the example above), or just changing the wording and not the core content (Case: Ref), or not changing the tweet text at all, and just changing the image (VisOnly). While an LLM under test for its persuasive capability can change the facts as well to make the content more persuasive, this problem is controlled to a large extent by the way we construct the dataset for transsuasion. As we show in the first reply, GPT-4o (an independent judge) agrees 88.3% of the time that the facts in the transsuasion pair are the same with high confidence. To further enhance the pipeline, future work can look into joining our work, especially PersuasionBench and Arena, with extensive NLP literature on fact-verification.

Starbucks:

1. Transsuaded tweets emphasize a seasonal theme or promotion.

• Username: “StarbucksIndia”
• T1 date: 27th October 2021
• Tweet: “Relish the sweetness in every sip of the season’s special at your nearest Starbucks store.”
• T2 Date: 28th October 2021
• Transsuaded Tweet: “Celebrate #AStarbucksDiwali at your nearest Starbucks store & make this Diwali even more memorable by relishing our season's special with your friends and family at your nearest Starbucks store.”
• Explanation: “Diwali” is a major festival in India that occurred between 2 November and 6 November in 2021, and happens between October and November every year. The transsuaded tweet uses this fact because of the date in context.

2. Transsuaded tweets convey a personal experience or sentiment.

• Username: “StarbucksNews”
• Tweet: “Schultz: Shanghai is the city with the most Starbucks locations in the world <HYPERLINK>”
• Transsuaded Tweet: “This city has the most @Starbucks locations in the world, your own Shanghai! - Schultz <HYPERLINK>”
• Explanation: The transsuaded tweet refers to Shanghai as “your own” conveying a more personal experience as opposed to just stating it as a fact.

Airbnb

1. Transsuaded tweets emphasize a nature-centric experience

• Username: Airbnb Australia
• Tweet: 📍Tasmania, A 25 minute drive to the stunning Bay of Fires, this two-bedroom home comes complete with a playground and BBQ pit for toasting marshmallows, for the perfect family stay <HYPERLINK> <HYPERLINK>
• Transsuaded Tweet: 📍Tasmania, This minimalist timber barn sits on three acres at the foothills of the Dial Ranges. Ideal for switching off and slowing down, take your pick of beautiful local climbs and finish the day with a bath under the stars. <HYPERLINK> <HYPERLINK>
• Explanation: The transsuaded tweet portrays a nature-centric experience with “local climbs, bath under the stars”, while both are used to promote tourism in Tasmania, Australia.

2. Transsuaded tweets include a specific location or city

• Username: Airbnb
• Tweet: Among experience hosts, 50% are women, and 55% of home hosts are women. <HYPERLINK> <HYPERLINK> Image: an Infographic with two women in a kitchen laughing and talking.
• Transsuaded Tweet: Women make up a larger majority of home hosts in some countries, like New Zealand (70%), South Africa (63%) and the Philippines (61%). Santa Barbara has the most women experience hosts (65%). <HYPERLINK> Image: an Infographic telling percentage of experience hosts in Santa Barbara are women
• Hyperlink: https://news.airbnb.com/women-hosts-have-earned-nearly-20-billion-on-airbnb/
• Explanation The transsuaded tweet is more engaging due to the use of specific location mentions (both in text and image).

Nike

1. Transsuaded tweets include a specific cultural or historical significance.

• Username: Nike
• Tweet: The KOBE XI ‘What The’ combines the highlights into one memorable shoe. <HYPERLINK> <HYPERLINK>
• Transsuaded Tweet: Legendary shoes for a legendary career. Make your own Kobe XI #MambaDay iD tomorrow. <HYPERLINK> <HYPERLINK>
• Explanation: The transsuaded tweet leverages the historical significance of Kobe Bryant’s (a prominent Basketball figure) career rather than emphasizing the shoe itself.

2. Transsuaded tweets include a specific date and time.

• Username: nikebasketball
• Tweet: Iconic blue for an iconic run. Customize the #KYRIE2. #BringYourGame @NIKEiD <HYPERLINK> <HYPERLINK>
• Transsuaded Tweet: 20 years that fueled the Love & Hate. #KYRIE2, Customize your own @NIKEiD. <HYPERLINK> <HYPERLINK>
• Explanation: The transsuaded tweet utilizes specific time period “20 years” which is a common theme of successful tweets for Nike.

Thank you again for your valuable feedback, which has improved the qualitative strength and readability of our paper! We hope our answers have satisfied your concerns, and we would love to answer any remaining questions. 🙂

After the reviewer's question and our reply yesterday, we conducted more analysis to analyze the truthfulness of the models trained on the transsuasion dataset. To understand this better, we computed the performance of trained models with comparable baselines on the TruthfulQA dataset.

TruthfulQA [1] is a standard benchmark included in the widely used LLM benchmark MMLU [2] to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. Questions are crafted so that some humans would answer falsely due to a false belief or misconception. To perform well, models must avoid generating false answers learned from imitating human texts [1].

This analysis specifically highlights the robustness of trained models with the base LLM and allows us to address potential concerns regarding ethical issues and truthfulness. We compare our trained models with their base LLM (lmsys/vicuna-13b-v1.5), Llama-2-chat, and OpenLLaMA [3]—on the MMLU benchmark's TruthfulQA task [1], a well-established benchmark for this purpose. Additionally, we provide scores for the 7B variants of OpenLLaMA, LLaMA-2-chat, and Vicuna to improve the interpretability of the metrics in terms of sensitivity. The corresponding results are presented in the table below, with bar plots provided in Figure 12 of the appendix.

TruthfulQA measures the performance on the following metrics:

• MC1 (Single-true): Given a question and 4-5 answer choices, select the only correct answer. The model's selection is the answer choice to which it assigns the highest log-probability of completion following the question, independent of the other answer choices. The score is the simple accuracy across all questions.
• MC2 (Multi-true): Given a question and multiple true / false reference answers, the score is the normalized total probability assigned to the set of true answers.

All evaluations were done 0-shot, through Harness LM-eval [4]

The evaluation confirms that our fine-tuned models maintain truthfulness levels very close to the base model, Vicuna-13b-v1.5, with minimal performance degradation after fine-tuning. Further, our models consistently perform better or comparable to other comparable 13B models, such as LLaMA-2-13B and OpenLLaMA-13B, across both MC1 and MC2 metrics. Importantly, our models preserve the truthfulness of the base LLM while enhancing persuasiveness, validating our approach of using semantically similar training data to maintain factual consistency and truthfulness without compromising on persuasion. We have included this analysis along with the results mentioned in the previous reply in Appendix Section-C.3 of the updated version.

We hope that we have been able to answer the reviewer's questions and would request the reviewer to increase the score if they agree that the responses have been helpful in addressing the concerns.

References:

[1] Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.

TruthfulQA: Measuring How Models Mimic Human Falsehoods (Lin et al., ACL 2022)

[2] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

[3] Geng, X., & Liu, H. (2023, May). OpenLLaMA: An open reproduction of LLaMA. Retrieved from https://github.com/openlm-research/open_llama

[4] Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., ... & Zou, A. (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. arXiv preprint arXiv:2405.14782.

I believe it is still Twitter policy that only Twitter ID can be shared publicly. However, given the exorbitant amount they currently charge for API access to hydrate the tweets, how do you plan on releasing the data you collect?

We plan to open-source the collected datasets, source code, and trained models. Our aim is to build PersuasionArena and PersuasionBench, inspired by LMSys. LMSys is supported through public donations (https://lmsys.org/donations/) from organizations such as Hugging Face, MBZUAI, Kaggle, NVIDIA, and others. Similarly, a few organizations have already reached out privately, offering resources to support the development of open-source infrastructure for PersuasionBenchand PersuasionArena. To benchmark their models on LMSys, model developers submit either model parameters or APIs. Following a similar approach, we plan to run benchmarks and arenas where model developers can continuously evaluate their models. These resources will be released upon acceptance. Regarding the release of actual Twitter data, you are correct—Twitter IDs can be publicly shared. We will therefore release the tweet IDs so developers with Twitter API access can begin working on the benchmarks. For those without access, the last time we reviewed Twitter's Developer Terms and Conditions (in January 2023), it was permissible to share datasets with fewer than 1 million tweets. Since then, the terms have been updated. This 1-million constraint still covers the majority of our samples (>90% of transsuasion data). We plan to confirm with the Twitter API team whether this constraint remains valid.

Line 296-298, “…measures a model’s capabilities to simulate behavior over (future) time unseen during the training.” But is this future time within LLM’s pretraining knowledge cutoff or not?

LLaMA-2’s cutoff date is earlier than our time split, and we found no tweets in the SFT dataset of Vicuna-1.5 What is the oracle model in Table 2? The Oracle is a Vicuna-1.5-13B (same as the base LLM for Transsuasion) finetuned on Train+Test data, the details are outlined in lines [342-353] of the current version, We will fix the typo in line [346] of the current version. The oracle is trained on the train and test data with the best training regime (for the comparative transsuasion tasks) to measure the persuasiveness of the transsuaded tweet.

Why do you use Vicuna 1.5 13B to finetune?

At the time we were conducting the experiments, Vicuna 1.5 13B was the most capable model (in the 13B parameters scale), and had a knowledge cutoff before our time-split.

Even though you claim that the time gap between paired posts is not that important (e.g. Table 6), but that table itself seems to show statistically significant correlation for some accounts between like difference and time difference?

Yes, we also agree that there is a statistically significant relationship, but it is very small (-0.079 to 0.087), we have rephrased it to “very small” to make it clearer.

Line 480-482 “LLaMA-3-70B, while being significantly smaller than GPT-3.5 and GPT-4, has a higher persuasiveness.” – I am afraid I am not following where you get this conclusion?

Thank you for pointing out, GPT-4 is a typo there, we have fixed it in the current version at line [518-519]

Around line 847-849, “their KPIs. Table 4 shows the results of this study. We observe that despite being experts in marketing, the budget allocation by these marketers had almost no correlation with any of their key performance indicators.” -> I am not sure if 0.207 is “no correlation”?

Thank you for pointing out, we have corrected it to “low or no correlation” in table 4.

Line 1431-1437: I think T1 and T2 here are actually semantically quite different? Could you mention how many pairs are like this where one version of the post actually contains much more information than the other one? This undermines your data collect regarding “matching” posts, right?

The example you have mentioned belongs to the Highlight case (the hyperlink mentioned is same and LLM’s are given the context of the webpage, for details please refer to Hilight in Section B.2 of the appendix) both tweets utilize different information from the webpage, which require a lower level of cosine semantic similarity in text (>0.6)

Line 1545-1555: Do you think it would be better if you define what low/medium/high actually means for the LLM in the prompt? I think perhaps there’s a potential of underestimating 0-shot model performance due to this?

Thank you for pointing this out, the listing that you have mentioned was used to train the model, we have added the listings for 0-shot evaluation in Listing 25, we tried several strategies including rating out of 3, top, bottom and middle, but this worked the best in our empirical evaluation.

Conceptual Framework, The paper's framing of persuasion

We appreciate the reviewer's concern and would like to address the distinction between persuasion and social media engagement in the context of our work.While it's true that our primary dataset leverages social media interactions, we position our work within the domain of persuasion due to its broader transferability and application beyond engagement metrics.

To check the transfer of persuasion capabilities measured over Twitter to other domains, we test all models on 4 benchmarks in zero and few shot: Humans as Judge (our study), Humans as Judge (Anthropic Persuasion study), Marketing blogs dwell time and views prediction, Audience specific transcreation. This also allows us to test the transfer power of Twitter-finetuned 13B model whose persuasion capabilities were developed over Twitter to other channels and domains. Marketing blogs dwell time and views prediction: We improve 19% and 22% compared to base model on dwell time and views prediction respectively (refer to table 9) Audience specific transcreation: We improve the performance ~2x compared to the base model on targeting (refer to Table 10) Humans as judges (our study): we improve by 15%, 20% and 50% on on Upvote/Downvote classification, Reasoning classification, and Feedback perplexity compared to the base model (refer to table 7) Humans as judge (Anthropic study): Our rank correlation is 0.47, ~6.5x more compared to the base model (0.07) (refer to table 8) Further to measure the robustness of PersuasionArena we collect the rankings from all metrics (8) of the (4) transfer benchmarks above, next we calculate the mean kendall tau coefficient between PersuasionArena these 8 rankings, this gives us a mean Kendall Tau coefficient of 0.69 which is considered strong. The consistent rankings of PersuasionArena in zero/few shot and unseen domains further validate its importance. These show that models trained on the task of transsuasion also transfer to completely unseen domains, channels, behaviors, and tasks. These findings demonstrate the robustness of our framework and the broader applicability of PersuasionArena and Bench across varied tasks involving persuasion. The term "persuasion" captures the transfer capabilities demonstrated by our framework, distinguishing it from just social media engagement prediction. The consistent generalization of PersuasionArena across varied studies [1,2,3] which frame this problem statement as “Persuasion” validates the broader implications of our work.

Literature Coverage

Literature Coverage: Prior research in argument quality highlights features like reason-giving, logical coherence, originality, relevance, interactivity, logos, ethos, and pathos as key to persuasive arguments [4,5,6]. For instance, arguments that clearly explain reasons or use credible evidence, such as statistics or sources, are often rated as more convincing. Logical coherence, achieved through clear and relevant progression, also plays a vital role. Annotators tend to prefer succinct and focused arguments, especially when comparing pairs, as these maintain clarity and avoid off-topic points.

Our work on transsuasion aims to improve measure and improve persuasion in language, while keeping the semantic meaning intact. We will revise our related work section to clearly position transsuasion within this and explain how it complements and extends these foundational contributions. Thank you for helping us better articulate our work.

In their research, Hackenburg et al. (2024) identify a log scaling trend for political persuasion with large language models, showing that persuasiveness gains diminish as model size increases. This study provides evidence that beyond task completion capabilities, scaling model size may have limited impact on enhancing persuasive effectiveness. Thank you for the valuable literature, we have added them in lines 195 and 457 of the current version.

Structural and Presentation Issues

it would be great if you could have a separate section on related works, rather than embedding it in the intro.

Thanks for your suggestion we have added a related work section in the appendix (section G) we will separate and add these in the camera ready version of the paper.

it would be great to include a figure of the tasks you are actually measuring (e.g. BS, CS-CT).

We have included the task figure(Figure 11 in the new version) in the Appendix.

there is a lack of qualitative result. While the numbers certainty indicates your fine-tuned model drastically lead to higher score in evaluation metrics, could you show paired examples, perhaps in the appendix, as to how exactly the model behave, before vs after fine-tuning, to let the reader contextualize the change in numbers?

We agree with your suggestion, we have improved the presentation and added examples for transsuasion from the untrained model in Section D. Notable trends like “usage of urgency, complex design elements” are missed which we have validated through our analysis in response to reviewer Xibh, What makes content persuasive

I am of the opinion that the things starting line 352 (“Extent of Transfer of Persuasive Skills) are more like robustness checks and are not directly relevant to the main benchmark? Perhaps this would then give you enough space to take a deeper dive in the main results, which is somewhat rushed?

The goal of our work is to build a comprehensive framework to measure the persuasiveness of LLMs, motivated by benchmarks measuring other natural language capabilities: BigBench, HELM, MMLU, MSR-VTT, CodeXGLUE, and MATH. We believe that it is very important to measure persuasiveness on a breadth of tasks covering different aspects including predictive and generative capabilities, which is why we created a meticulous benchmark to cover the generalization across 6 (BS, CS, TS-CT, TS-GT, TC, and Hum-Per) tasks.

I think perhaps “observational data” would be a better term to use than “natural experiment”?

Natural Experiments and Observational Studies are used interchangeably in literature [6,7], we understand the confusion and will clarify this in the camera ready version of the paper.

Please move the ethics section to the main paper (which gives you 1 page that doesn’t take up page limit). The ethics of this study is way too important to be somewhere in the appendix

Thank you for the valuable suggestion. We agree with this, and will shift the Ethics section to the main paper in the camera ready version.

Questions

Is there any risk of data contamination? I.e. could the LLMs have seen sense tweets as well as their engagement numbers already?

The simulation results for all zero/few shot elements in Persuasion Table 2, with nearly random performance largely indicates that this should not be the case. Further our time split consists of samples from the latest months, which is outside the knowledge cutoff window for LLaMA-2 and we found no such examples in Vicuna SFT datasets either.

Conceptually, where do you think the performance ceiling is? How far can we get in maximizing the persuasiveness of a message?

Thank you for posing such an intriguing question. Based on our benchmarks, the persuasiveness of large language models (LLMs) generally falls short of the best human writers. However, in specific domains and topics, their performance can sometimes surpass even top human writers. This advantage stems from LLMs' ability to aggregate and learn from a vast and diverse pool of data, far exceeding the exposure any single human writer could achieve. As the scale of available data grows, we anticipate that these models' performance will continue to improve. However, it's important to note that societal norms and perceptions of persuasiveness evolve over time. For example, messaging that resonated strongly during the COVID-19 era might not have the same impact today. Similarly, trends often reemerge—consider how retro aesthetics or communication styles periodically make a comeback. This fluidity means that no single model can remain the ultimate persuader indefinitely. Instead, it’s likely to resemble a "police-and-robber" dynamic, with models striving to keep pace with societal shifts while simultaneously influencing those shifts by introducing new trends. Ultimately, this underscores the importance of ongoing benchmarking and continuous training to ensure that models remain effective and relevant over time.

clarity on timing. It may be that certain times are just more active. Twitter has a cyclic activity pattern that peaks at specific times. This likely has a confounding effect on tweets that appear even hours apart. How did the authors control for these confounds?

Cyclic engagement To analyze the relationship between tweet success ($Y$) and temporal features ($X_1$: hour of the day, $X_2$: day of the week), while accounting for the brand ($Z$). We began with paired tweets ($Tweet_X, Timestamp_X$) and ($Tweet_Y, Timestamp_Y$), where $Tweet_Y$ is always more persuasive (e.g., higher likes) than $Tweet_X$. Each pair was converted into independent samples: $Tweet_X$ was labeled $Y = 0$, and $Tweet_Y$ was labeled $Y = 1$, retaining their corresponding temporal features and brand identifiers. We stratified the data by $Z$ (brand) to account for brand-specific differences in tweet performance, ensuring that temporal effects were evaluated independently of brand.

For each brand, we performed a chi-square test of independence between $X_1/X_2$ and $Y$, calculating individual p-values and effect sizes (Cramér’s V). Brands with fewer than 500 samples were excluded to avoid unreliable results (the coverage still remains > 95%). Finally, we combined p-values using Fisher’s method and calculated a weighted average of Cramér's V across brands to quantify the overall effect. The results showed a statistically significant relationship between temporal features ($X_1/X_2$) and tweet success ($Y$), with a combined p-value of $p < 0.05$. However, the weighted average of Cramér's V indicated that the effect size was small to very small (e.g., $0.05 <= Cramér's V < 0.1$). This suggests that while the time of day and day of the week have a measurable effect on tweet performance, the magnitude of the effect is limited.

Specifically, the weighted average of Cramér's V was $0.06$ for $X_1$ (hour of the day) and $0.09$ for $X_2$ (day of the week), suggesting a limited magnitude of influence. Despite the small effect sizes, the relationship is statistically significant, with a combined p-value of $1.345 * 10^{-5}$. This indicates that the time of day and day of the week have a measurable but negligible effect on tweet performance as shown in other studies [1,2].

Our original hypothesis was that these effects would be reduced because we only consider brand accounts, which to some extent optimize their timings already. We find that the p-values from the chi-square tests are negatively correlated (-0.31, p=0.003) with the number of followers a brand has, this indicates that more successful brands observe lesser effect of timing (hours of day and days of week) on their engagement. Thanks for the suggestion, we will add this analysis to the paper.

1. Rea, Louis M., and Richard A. Parker. Designing and conducting survey research: A comprehensive guide. John Wiley & Sons, 2014.
2. Gignac, Gilles E., and Eva T. Szodorai. "Effect size guidelines for individual differences researchers." Personality and individual differences 102 (2016): 74-78.
3. Hancock et al, AI-Mediated Communication, JCMC, 2019

My final concern has to do with general clarity: I found it a bit challenging to navigate the tasks outlined in the paper. I generally feel like there was more emphasis on creating many tasks instead of focusing on a few very important ones. More generally, I worry that focusing on many tasks detracts from the contributions of the dataset. For example: clustering the tweets, running some kind of topic analysis, or highlighting the types of domains the dataset consists (in the main paper) would provide more value than an additional task.

The goal of our work is to build a comprehensive framework to measure the persuasiveness of LLMs, motivated by benchmarks measuring other natural language capabilities: BigBench, HELM, MMLU, MSR-VTT, CodeXGLUE, and MATH. We believe that it is very important to measure persuasiveness on a breadth of tasks covering different aspects, including predictive and generative capabilities, which is why we created a meticulous benchmark to cover the generalization across 6 (BS, CS, TS-CT, TS-GT, TC, and Hum-Per) tasks.

However, we agree with the suggestion of adding dataset analysis; therefore, we have included additional details, such as dataset statistics, industry distribution of brands, and topic analysis of tweets from each industry. To understand the industry distribution of brands in our dataset, we extracted topics from their Twitter bio using BERTopic. These topics were clustered and assigned a name by GPT-4o-mini and visualized using a sunburst graph (Figure 5 in the new version of the paper). Similarly, we also analyzed the topic distribution of tweets (Figure 6 in the new version of the paper). If there is any additional analysis that will highlight our contributions better, please let us know.

A final question (minor question): the setup of T1, T2 pairs reminds of an RLHF-esque setup. Why didn’t the authors pursue a “reward model” paradigm?

Reward model paradigm: Thanks for the suggestion, we have added details for our ablation using DPO in PersuasionArena (extended in Table 3). The DPO model gets an ELO of 1283 compared to the ELO of SFT, which is 1304. While DPO shows lesser ELO than the SFTed model, it shows higher performance on similarity metrics: semantic similarity (87.2%), factuality match (94.3%), and NER match (94.9%). Our primary motivation for doing SFT as opposed to DPO is that models from prior works [7] trained to generate performant content have benefitted from multi-task training (Content Simulation, Comparative Transsuasion) (Please refer to appendix, table 8 in [7])

[4] Xie, Chengxing, et al. "Can Large Language Model Agents Simulate Human Trust Behaviors?." (Neurips 2024)

[5] Aher, Gati V., Rosa I. Arriaga, and Adam Tauman Kalai. "Using large language models to simulate multiple humans and replicate human subject studies." International Conference on Machine Learning. PMLR, 2023.

[6] Chen, Weize, et al. "Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors." The Twelfth International Conference on Learning Representations. 2023.

[7] Harini, S. I., et al. "Long-Term Ad Memorability: Understanding & Generating Memorable Ads." (WACV)

We sincerely thank the reviewer for their feedback. Below, we try to address the reviewer's questions and concerns.

Using an entirely different brand is probably more likely to have a big effect on persuasiveness

We only consider tweet pairs from the same “brand”, more specifically, the sub-brand (e.g. AdobePhotoshop, AdobeLightroom, GoogleChrome, GoogleAI) or twitter username (mentioned in Line [235]). Other than having the same brand, the samples should be less than 45 days apart and are semantically and lexically close to each other.

Small changes in the input can yield significantly different meanings that cosine similarity or edit distance may not capture

“minor tweet edits” often have larger semantic variation

“iPhone” or “Andriod”

While it is true that tweets from individual users can vary in almost all aspects of the message (e.g. Android and Iphone), considering that problem itself, we keep our focus on enterprise accounts. The products and services of such accounts remain relatively consistent over time. This consistency allows marketers to experiment with various messaging strategies, resulting in differential audience engagement rates. Further, since the products stay the same, it limits the autonomy of the marketer and consequently the transsuasion data and the models trained on it. The question about transsuaded text having similar meaning pertains to the degree to which AI can operate on messages without the sender’s supervision. This could be considered as “autonomy” of AI agent ([3] Hancock et al). We define nine types of transsuasion (Table-1) varying the autonomy of the AI agent. E.g. autonomy can be restricted to changing just the image (VisOnly), or just paraphrasing text (Parap).

To verify this quantitatively, we evaluated the generations on the task “NER Match”, which covers Organizations and Products. We see that our model correctly uses 92% of the NERs, similar to the few shot base model. Results for this are given below.

In the table below (also added as Table 15 in the Appendix), we show the similarity metrics between transsuaded tweets obtained or generated from (1) ground truth GT (2) GPT-4o (3) Ours(13B) (4) Vicuna(13B). We use the following metrics to test semantic similarity between transsuasion pairs of the ground truth and generated samples:

1. NER Match: The number of named entities that match on both sides divided by total named entities
2. Factuality match: We prompt GPT-4o to assess if both tweets do not differ factually, we also ask it to give a confidence rating on a scale of 5 tell if it is sure about both the facts being different and only consider the pairs where the confidence is at least 4.
3. MetricsMatch: The proportion of pairs of tweets that follow all the constraints for that task (edit similarity, text similarity, and image similarity) as defined in Table 1

• PARAP: Text Edit Similarity > 0.6, Text Cosine Similarity > 0.6
• REF: Text Cosine Similarity > 0.8
• VisOnly: Image Cosine similarity > 0.7

The tables show that not only are the input pairs largely similar semantically, but also that the output pairs are more controlled.

I see that the authors did some time analysis in Table 5; and that some of the p-values are indeed < 0.05. How exactly was this analysis done? I'm trying to interpret the correlation coefficient.

Table 5: In Table 5, we calculate and show the correlation between (1) the difference between the time of posting of two tweets and (2) the difference between the likes of these tweets. A negative correlation would imply that earlier or older tweets get more likes. We find that all significant correlations (across all tasks) are negligible (-0.05 to -0.006). This implies that there is very little impact of the time difference in the difference of likes between the pairs of tweets (across all tasks).

Dear Reviewer UfKL,

We trust our response has addressed your concerns and provided the necessary clarity. We kindly wish to note that other reviewers, such as EsZt, have increased their scores after their concerns were addressed, and reviewer Xibh has promised to increase their score following their last comment. We sincerely believe their helpful engagement has increased the paper's quality. If there’s anything further we can do to enhance the paper and ensure it meets the acceptance criteria, please don’t hesitate to let us know.

Thank you!

I've gone ahead and re-read the paper, along with the reviews from the other authors. In light of additional concerns, I'm going to decrease my score.

First, I am quite worried about the confounds in this work. I think there's plenty of prior work that argues for controls that this paper does not engage with. It's really hard to control for these exogenous factors---and I worry that the generative models are optimizing for things that are out of the writers control. I also agree with EsZt re: the conceptual framing; I really think this paper would be better positioned as an engagement prediction paper. This way, the authors can engage more with work in that area. I've cited two papers in my response that might point authors in the right direction---happy to cite more.

Second, the oracle: why is the oracle trained on the test set too? Doesn't this reward overfitting to train? I mean, a model that does really well on the train set and then okay on the test set will be penalized more than a model that does just well on both (by the oracle). I also find it odd that for many of the human-eval tables, GPT-4o performs better than the fine-tuned models.

Finally, clarity---this is also a major one: I think this paper could really use more focus! The results and discussion (Sec 5) lists many tables without engaging with what the results mean---readers are forced to go to the Appendix, and put together a narrative. I honestly think removing some tasks, or even splitting up the benchmark and tasks into two different works, would help significantly.

I want to end by saying this work is really cool! I think there's a lot of promise, but I'm leaning more towards a reject.

Thanks for the comprehensive response! I am leaning towards keeping my score-

Regarding content confounds:

First, even within the same brand or sub-category may often have tweets that focus on a specific feature. So for photoshop, my tweet might end up focus on two different features within Photoshop, confounding persuasiveness. Do the authors have an anonymous dataset link I can look through? I'd love to take a look at some samples myself.

Regarding topline generations:

To analyze this trend further, we extracted the brands where the trained model has the maximum improvement over the human topline.

Can I see concrete examples of what generated tweets outperformed the topline? And what the actual topline tweets are? Again, I'm a little suspicious that the confounding variable here is something that is out of the author's control---and not the actual framing that the author has control over.

Regarding Timing

I am quite surprised that timing has a very small effect on persuasiveness! Time of day definitely has an impact on the number of users active on a platform---and even the content they engage with [1, 2]. Can you fit a simpler model here? Take a linear regression model, normalize the likes within a specific account, and map the time of day to the likes? Is the R^2 really 0? This alone could be a big social science result (if true??) I'm happy to try to run this analysis myself, given the dataset.

[1] Ozum et al. Tweets we like aren’t alike: Time of day affects engagement with vice and virtue tweets

[2] Piccardi et al. Curious Rhythms: Temporal Regularities of Wikipedia Consumption

Regarding Clarity

Finally, I don't think adding more results here really helped with clarity. I agree with EsZt here- I can't really recommend acceptance since this work was generally really hard for me to navigate personally.

Regarding Timing: timing has a very small effect on persuasiveness! Time of day definitely has an impact on the number of users active on a platform. Can you fit a simpler model here? Take a linear regression model.

1. You are right, the timing of posts having a small effect on persuasiveness sounds counterintuitive. The Time of day does have an impact on the number of users active on a platform and the content they engage with [1,2]. We also show in the Chi square tests and Cramer's V analysis that there’s a significant impact of time of posting p=1.345 * 10^-5, but the effect is small in magnitude (<0.1).
2. The distinction between what we have done and what has been shown in literature comes from the various filters we have applied to construct the dataset. We illustrate the effect of each filter in the analysis below:

We have applied the following filters (covered in Sections 2 and B.6):

• Business or Enterprise Accounts: We collected 10k usernames from Google searches on Wikipedia entries that were marked as “business” or “enterprise”, we also identified 317 usernames that were found to be mislabeled as business or enterprise, based on Deberta Zero-Shot classification and manual filtering on top of it.
• <10 posts/day: to filter accounts that do event based marketing, or whose tweets are only seen for shorter periods of time.
• Non-news, forecast, time dependent tweets: We tag username descriptions from a few-shot (human-labeled) Deberta-v3 model, to filter out event based marketing channels, and accounts whose pairs showed significant difference (in content and likes) due to the timestamps. These reduce the number of tweets from 79M to 8.9M.
• Manual Filtering: Finally, amongst the remaining 2457 accounts, we manually filtered out 112 accounts after clustering the tweets from each of these, resulting in 2345 accounts and 4M unique tweets
• Semantic Similarity: They should have high semantically similarity, the filters are given in Table 1. They should have a difference in likes of at least 20 percentile of the likes in their bimonthly period.

To analyze the effect of each filter in diminishing the effect of time of posting in determining the engagement of the post, for each filter, for each username, we train a linear model on hour_of_day, day_of_week (X) and predict the monthly normalized likes (Y). We report the mean and standard deviation r^2 scores across all usernames.

As seen from this table, the r^2 scores keep decreasing as we apply our filtering steps, thus showing that while in the original data, there is a significant and appreciable impact of time on engagement, in the filtered data, there is a very minor impact of time.

To analyze the individual impact of each filter, we also perform this analysis in a different way where for each filter, we take all unfiltered samples and calculate the r^2 score for a classifier trained only on that data. Here are the results of that experiment:

For your ready perusal, we have attached the code and anonymous data to the supplementary. We have anonymized the tweet_ids, usernames as uids, and have given the timestamps as date_x and date_y for tweet with id_x and id_y. Tweets labeled as (Y=1) appeared on the right-hand side of a transsuasion sample (id_y > id_x). We have also attached a notebook where we run the binary classification using LogisticRegression. Thank you for engaging with us and asking us questions that can be scientifically answered. We would love to engage more with the reviewer to help clarify better.