Optimizing Interpersonal Communication by Simulating Audiences with Large Language Models

Thank you Reviewer hVxs for your detailed review. We are elated that you find our framework steps intuitive and well-grounded, and we appreciate your attention towards the high quality of our human data and the promise that our results show.

| The paper could benefit from better defining e.g. advice and breaking down what constitutes good advice or any possible distinctions between how LLMs seem to generate responses conditioned on advice vs. how humans may take the advice. Thank you for the suggestion. We have reworked the section where we describe the framework, including the section about exploring advice (3.1), to be more grounded in psychology literature that people recall useful advice or prior experiences when considering their next action. We have also better motivated the incorporation of advice into creating the message candidates (section 3.2), and we believe that the “inner voice” method from Park et al. (2023), combined with our framing of “remembering a piece of advice”, may be similar to how humans would typically interpret advice as well. However, there is still much work to be done investigating similarities and differences of SoTA LLMs and human behavior. | The experimental procedure could be better summarized in the main paper body - the detail in A.4 is in stark contrast to the brief explanation on page 6 of the paper itself. This is a good point - we have added a paragraph providing details on the rating interface that participants were given in Section 5.1. We also clarify the descriptions for the candidates and baselines that the humans are rating by providing examples in Appendix A.4 and the EGS prompt pipelines in Appendices A.3-A.7. Unfortunately, we have not had the luxury of space to move much more of the experiment description into the main text, but if you have any particular details you would like us to include, please let us know.

| LLMs are highly sensitive to prompt engineering, wording, and order. This paper would benefit from a deeper discussion of how the prompts were developed, any success or antipatterns noticed, and stability of results WRT prompts.

Thank you for this suggestion, we completely agree that the sensitivity to prompts is a concern with LLMs. First, our framework implementation randomly assigns candidates to be shown first or second in the pairwise comparisons. To complement this, we perform analysis on the order of the examples shown in the pairwise comparions (Appendix B.11) and demonstrate that whether a candidate shows up first or second does not bias whether it is favored. Second, when we constructed the prompts templates, we designed them to be very simple to minimizes possible biases towards or irregularities in performance across candidates. In all steps of the pipeline, we only prompt the LLM with the scenario description, the action it is to take, and the role it is supposed to simulate (if applicable). The only modification we make is to ask the model to generate reasoning before the final output following Wei et al. (2022), which is for performance as well as interpretability and sanity checks. We also add the full list of prompts in the order they are used, as well as examples, in Appendices A.3-A.7. Third, during development, we noticed only a few antipatterns w.r.t to the prompts: LLM outputs not obeying specific parsing rules (we cover this briefly in Appendix A.6), poorer agreement/performance on unorthodox advice (Appendices B.1 and B.3), and worse performance when trying to combine three (or presumably more) advice (Appendix B.9). Fourth, we notice a few successes in prompting as well: Prompting for unorthodox advice generates creative and unexpected outcomes that improve the ratings of downstream candidates (Appendix B.8); allowing for the conditioning on more than a single piece of advice improves the top-performing candidates but not as much in general (Appendix B.10); searching for conceptual advice instead of encouragement-type advice or irrelevant advice leads to significantly better performance (Appendix B.7). Fifth, we also highlight a list of general observations we make qualitatively, with specific examples included, in the new qualitative analysis (Section 4). The section contains six high level observations, two about Explore, two about Generate, and two about Simulate. Sixth and last, we conduct a preliminary analysis into the stability of results w.r.t. prompts in the SHP evaluation (Appendix C.2). Here, we replace each sentence in the redditor simulation prompt with a variant and find that performance is roughly equal or better than the original.

Thank you reviewer Dz3q for your review, we appreciate you for going through the paper in such detail. We are excited that you share our vision that studying the application of LLMs to interpersonal communication is an important research direction.

| discussions about some of the recent advances in prompt engineering, such as self-consistency

Thank you for your suggestion, we completely agree! In addition to self-consistency being reliant on the assumption of an underlying fixed answer, we performed a new ablation on unorthodox advice that suggests that the self-consistency method’s motivating intuition that better answers = higher probability does not hold in the interpersonal communication case. We have added a discussion of prompt engineering past work the extended related work (Appendix D), and we point to it in Section 2. We add a justification of the baseline selection in Appendix A.7.1, and point to it in Section 5.1. Details on the new ablation for unorthodox advice are in Appendix B.8.

| I think it will also strengthen the motivation to study/focus on the interpersonal communication application, rather than focusing on general prompting methods.

We have edited various parts of the paper to reflect this suggestion: In the prompt engineering paragraph of the extended past work (Appendix D), we specifically differentiate our work from other prompting methods by highlighting that it is focused on the interpersonal communication application. We add a paragraph on episodic future thinking from psychology to the past work section (Section 2), drawing connections between our work and existing literature on the human process of simulating future outcomes. We also connect our framework back to this concept in the discussion (Section 7.4). We add various psychology and computer science literature to ground the claims in our introduction and methods sections (Sections 1 and 3). Please let us know if you feel we can do more in any other areas.

| The stakeholder idea is interesting. It demonstrates how focusing on the interpersonal communication task is meaningful. However, it makes me wonder if it is meaningful to consider stakeholders for the scenarios after the first two.

We completely agree - multiple stakeholders/audiences is a feature of EGS that can have variable impact depending on the problem context. As our target is to assist users in all communication scenarios, we believe multiple stakeholders is an important piece of the framework. In cases where only one audience is needed, the LLM does not generate extraneous audiences, and nothing is lost.

| it would be better to have the generated stakeholders and their generated weights for all scenarios, perhaps in the appendix.

We have added a comprehensive list of the stakeholders of each scenario, their weights, and their full descriptions in Appendix A.5. Please feel free to take a look!

| I wonder if we can improve the framework by simulating the stakeholders before the generate step. If we condition on each stakeholder and then generate candidates, will it generate better candidates compared to the case without conditioning on the stakeholders?

This is an interesting suggestion! We conducted a quick experiment to see how candidates generated based on both stakeholders and advice, compared to those generated with just advice. Specifically, for the plane crash scenario, we compare the best candidate against six new candidates. These contain three candidates conditioned on the best candidate’s advice set and the “general public” stakeholder, and three candidates conditioned on the same advice pair but on the “victims’ families” stakeholder. Across three simulated comparisons per pair and audience profile for 36 comparisons total, we find that none of the new candidates are favored compared to the original best candidate, suggesting that conditioning on specific audiences does not perform substantially better than conditioning only on advice. We add a detailed description of this experiment, including prompts + design choices, the new generated candidates, and the pairwise comparison results in Appendix B.12. It is worth noting that these comparisons were done by GPT-4, and even though we have shown that agreement between humans and GPT-4 in the scenario were high, it is still possible that humans would rate these new candidates differently. Additionally, your suggestion also inspires the more general question of: Which steps of EGS or interpersonal communication in general benefit most from considering audience reactions? We think is a very interesting future direction to study.

| How the prompts are designed for the CoT baseline?

The prompt is designed to be very simple to minimizes possible biases towards any specific candidates. Specifically, we only prompt the LLM with the scenario description, the action, and the communicator’s role. We also ask the model to generate reasoning before the final output, following Wei et al. (2022).

The prompts themselves are structured as follows: “[Scenario description] What would you [insert action] if you were [insert communicator]? Please reason about the situation before providing your answer. Provide your answer in this format: Reasoning: [text] Answer: [text]”

An example for the product launch scenario: “{Scenario description} What would you say if you were the CEO? Please reason about the situation before providing your answer. Provide your answer in this format: Reasoning: [text] Answer: [text]”

We have also added a full list of prompts in Appendix A.7.2.

| Experimental results demonstration:

a. For Table 2, the authors choose “>0.6” as a high agreement. However, it would be better if a baseline is included to clarify how a “high agreement” is decided.

We agree with the reviewer that “high” agreement is not well defined, especially when we defined our own agreement metric. We replace all instances of “high agreement” with “significant agreement”, and back up this claim with two new agreement analyses comparing the mean scores of winners and losers of pairwise comparisons in the Simulate step.

One is a multi-level model where the scenario is treated as a random effect, and we find a significant fixed effect for the winner vs. loser of the simulated audience pairwise judgements (see Section 5.3). The other finds that in individual scenarios, the winners also obtain significantly higher scores in a paired samples t-test across five scenarios and a subset of a sixth scenario (see Table 3, which was previously Table 2). Both of these analyses use a metric complimentary to the previous metric in the original, which we justify in the paragraph before the third metric (our original percentage agreement) is introduced in Section 5.3.

b. Similarly in Table 3, it would be better to mark the significance between comparisons.

We have added statistical significance testing comparing EGS with both baselines using bootstrapping with 10,000 samples to the result. Averaging over the eight scenarios, we find that EGS significantly outperforms both GPT-4 zero shot and CoT baselines (see Table 2, which was previously Table 3).

c. In Table 5, what do the results represent? I think it is the accuracy of selecting the correct upvote, but it is not stated in the paper.

Yes, this is correct! We have added the relevant details in the first sentence of the second paragraph of Section 6.

We thank Reviewer GN7N for their clear and thoughtful review. We are encouraged that the reviewer found that using LLMs to serve different roles to be a promising research direction, and our experiments to be comprehensive.

| how the eight scenarios are chosen based on the 10 fundamental processes of interpersonal communication, and why it is representative of all possible social tasks

Generalizability and representativeness was something that we carefully considered throughout our project, especially since the domain of general interpersonal communication is impressively large. We would like to share a list of steps that we have taken to try to make our evaluation (including the scenarios) as cohesive as possible, which is available in the first point of the general comment. Though we do not believe nor claim that our experiments are fully representative of all social tasks, we believe that we have taken significant steps in trying to ensure the generality of our analysis, while the former is much more difficult to achieve.

| The initial generation results bottleneck the final generation. It would be interesting if it could aggregate the valid points in both candidates and then decide the final generation.

Your suggestion to use the pairwise comparisons to create new candidates is very interesting! We conducted some preliminary investigations that we report below, and we have added a more detailed description to Appendix B.14, and add a shoutout in the main paper in Section 3.3.

Method: After the simulate step, we take the top k candidates and their pairwise comparisons, and ask GPT-4 to generate a new candidate that performs better based on the comparisons. For more details including the specific prompts, please see Appendix B.14. We also mention this briefly in the main paper as an alternative when introducing the aggregation of the simulated comparisons.

As this method would require inserting all relevant pairwise comparisons into the new candidate generation prompt, it is better to use a smaller k to be able to check manually that the comparisons are actually taken into consideration. In our preliminary investigation, we use k=2 as a simple example.

Results: We find that the new candidates try to incorporate information across both candidates, and are thus much longer than their original counterparts. For example, in the white lie during date scenario, the two original candidates are:

"Wow, I must say that style isn't easy to pull off, but you certainly can! You just light up in whatever you wear and that joy is contagious. It's not everyone who can wear a jacket like that and make it work. You make it feel like more than just a jacket, you bring it to life!"

"Wow, I love how that jacket brings out the sparkle in your eyes, and I can see how happy it makes you feel. It certainly adds a unique flair to your whole look."

And the new candidate:

"Wow, that jacket sure is unique! And trust me, it's not everyone who can wear something like that and bring it to life. The way it brings out the sparkle in your eyes is really captivating. Above everything, what I love the most is seeing how happy it makes you feel. Your joy really is contagious, and it lights up your whole look!"

Examples are in Appendix B.14.

Speculatively, this often makes their utterances too lengthy for the scenario. Future investigations can be done to see if these outcomes are indeed less preferred by human raters, while also exploring how LLMs can better combine message candidates in a concise manner.

More generally, there are many options after the simulate step to combine scores into a final candidate message. EGS can be viewed as a search + filter framework, and the best-performing advice and/or candidates can be interpreted and aggregated by the user as they see fit. We also add a brief evaluation on various alternative aggregation methods motivated by cognitive science literature (Gates et al., 2020), including maximin, maxsum, and inequality aversion (see Appendix B.13).

| In step 2, when saying “iterates over subsets of advice”, how these subsets are formed?

Initially, subsets of advice were formed by taking the powerset of the set of advice. To restrict the advice space to a reasonable and efficient size, we limited advice sets to the power set of cardinality 2. This decision is backed up by a prelimiary study suggesting that > 2 pieces of advice was less effective, which we have included briefly in Section 5.1 and in more detail in Appendix B.9.

We thank Reviewer V2zC for acknowledging the difficulty of the domain of problems we propose a solution to. We agree with you that generalizability is a key challenge in our work, and we would be glad to share a list of steps we have taken throughout the project to address this, available in the first point of the general comment.

| One possible benchmark could be the negotiation conversations to check what percentage of the time the proposed approach will be able to win the negotiation

The bargaining metric you propose – evaluating the percentage of the time the approach wins the negotiation – is quite similar to what we measure when we ask the crowdworkers to evaluate our generated candidates & baselines. Specifically, we ask, “How likely would you be willing to negotiate the price of the vase with Jill?”, and the scale (0) Definitely not willing ... (5) Neutral ... (10) Definitely willing (see Section 5.1, second-to-last paragraph). Here, the metric is softer than whether the candidate would win the negotiation, but it serves a similar purpose. The other 7 scenarios also each have a unique metric just like this one, which can be found in the bulleted list in Appendix A.8.

| on SHP dataset, it would be nice to have more fine-grained study on the type of the domain/user preferences and personalities and their connection with the framework's performance

While it is difficult to exactly identify the preferences and personalities of individual Redditors, we can use each subreddit rules and guidelines to better understand the preferences of Redditors who browse specific domains. As discussed at the end of Section 6, we found that some domains such as asksocialscience dictate that comments “must be supported by citations”. This preference is reflected in the performance of the framework with the Funny setting being significantly worse than that of the Default setting. For the askculinary domain, the subreddit rule states that posts and comments should be “specific” and “detailed”, but there are no rules to force all comments to be strictly informative. Consequently, the Funny setting performed better on this domain.

To further understand how the domain/user preferences are connected with the framework’s performance, we have also added a more fine-grained study on the SHP dataset. Specifically, we perturb the Funny Redditor prompt to reflect different user preferences and personalities, which results in a total of 6 different Redditor profiles. We conducted this experiment on the askculinary domain, and the new prompts can be found in Table 18.

We found that two out of the four new profiles performed identically to the original Funny prompt while the other two saw significant improvements. The highest performing one asks the LLM to prefer “niche” topics, so the performance increase could be attributed to the “specific” and “detailed” rules of the askculinary subreddit. More analysis can be found in Appendix C. We thus found that the performance of the framework depends on the preferences of the domain and the alignment between the used prompt and user preferences.

| it is important to investigate the outcome of their generations in different cultures or on people with different personalities.

We share your concern about our framework being affected by underlying social norms captured by the LLM. We agree that thorough investigation is necessary with respect to people of different cultures and personalities. As EGS is model-agnostic, this becomes a large undertaking outside the scope of this paper, since such a study would need to identify and define new metrics and evaluation protocols, given the EGS framework. We plan to follow-up with a more comprehensive effort in future work. In the meantime, we have added this shared concern in the discussion (see Section 7.3, last paragraph).

It is worth noting that one benefit of EGS is its interpretability and ease of selecting alternatives (see Section 7.1). Thus, it makes it easy to analyze through EGS for any potential biases that the underlying LLM may contain. Furthermore, it also makes it easy to find alternatives in case the user is not satisfied with the recommendation.

| In table 2, how do we assure that the outcome of EGS is significantly better than the other baselines? have you done any significance testing?

Yes! We have added statistical significance testing to the result, where we compare EGS with both baselines using bootstrapping with 10000 samples each. Averaging over the eight scenarios, we find that EGS significantly outperforms both GPT-4 zero shot and CoT baselines (see Table 2).