Learning Personalized Story Evaluation

Q1. The description of the paper is too fragmented and less logical.

A1. We have reorganized the paper to make it clearer. We refer the reviewer to the general response for more detailed information.

---

Q2. Flowchart & a framework diagram

A2. Thanks for the valuable suggestions! We add a flowchart for the data construction in Figure 2 and demonstrate our PerSE framework in Appendix Figure 11.

---

Q3. need further clarity on the practical implications of the research work as well as specific application scenarios. A3. In this paper, we mainly focus on

1. introduce two personalized story evaluation datasets. By elaborating on the data construction, these two datasets are less affected by the contamination issue of LLM-based evaluation models. They can be used as better benchmarks for both training and testing of the personalized story evaluation models. The pipeline can be easily adapted to other tasks.

2. propose PerSE to analyze the unseen reviewers’ preferences from their prior reviewers and predict specific reviews from their views. By using a small amount of data as the instructions, we enhance LLMs’ capabilities of modeling the reviewer’s preference. The experimental results show that PerSE can predict personalized reviews for new users with only 3 prior reviews.

Thanks to this capability, PerSE can be used in the recommendation system to select the most suitable story for a new reviewer without finetuning the preference data of this reviewer. For example, given several stories and three reviews from a new user, we can use PerSE to rank them and provide suggestions for this user. PerSE can further tailor story generation models to align with the unique needs and preferences of different users by using the score of PerSE as the reward.

Q1. Ambiguity in Metric Basis: clarify what specific metrics this "prediction accuracy" is based on. Is it linked to the previously mentioned Pearson correlation, or are there other relevant metrics involved?

A1. The prediction accuracy is the accuracy of binary classification, which is calculated by $accu = \frac{\text{the number of true prediction}}{\text{the total example size}}$. We view the comparison of each aspect as a binary classification between the given two plots. We also clarify this point in the revised manuscript.

---

Q2. Lack of clarity in model attribution in contribution statements. "Current LLMs" refer to models like ChatGPT-4, and it should distinguish that the pronoun "they" in "with instruction-tuning on several thousands of data, they can" pertains to the proposed method based on LLaMA-2.

A2. Here the “current LLMs” refer to the direct usage of LLMs as the evaluation model, and “they can” indicates the instruction-turned LLMs. Both of them can refer to LLaMA-based or GPT-4-based models. The main difference here is before and after instruction-turning. We have revised the contribution part.

---

Q3. Limited Diversity in Baseline Models: For not including a wider range of commonly used methods in the field, such as matrix factorization or other score prediction techniques, as part of the baseline models for comparison?

A2. We add matrix factorization baseline[1] on Per-MPST for the main Table 2 (previous Table 3) as below. However, on Per-Doc, both plot pairs and the annotators of the test set have no overlapping with the training set, so the matrix factorization cannot apply to this setting. MF baseline has a similar performance to the simple baseline, which averages the score in the preference.

---

Q4. Do you think it would be fairer to apply fine-tuning to chatGPT-4 via OpenAI's API to ensure a level playing field?

A4. Yes, we agree it would be a fairer comparison, and we expect that using PerSE with a more capable backbone model than LLaMA (e.g., ChatGPT) will naturally yield better performance downstream because the method we proposed in PerSE is orthogonal to the backbone. We included ChatGPT not as necessarily a fair comparison but just as a point of reference to put PerSE's performance in context.

[1] Matrix factorization techniques for recommender systems

Q1. how exactly an anonymous plot is summarized?

A1. We use oasst-30b[1] to summarize the plots by prompting it to give a summary of the main idea of the anonymous plot. We add a flowchart for data construction in Figure 2 and provide the detailed prompts we use for anonymization and summarization in Appendix Figure 9. We also refine the description in Section 3 to make it clearer.

---

Q2. It'd be great if much larger scale datasets could be used.

A2. One of our motivations for re-proposing Per-MPST and Per-DOC is the difficulty of collecting personalized evaluation data. Because of this difficulty, the fact that PerSE still works well with our current smaller dataset sizes is also a strength of our method - it doesn't need the larger datasets to work fine. Moreover, our Per-MPST has ~10k data for training and ~1k for testing, which is comparable to other personalized text classification and generation datasets[2].

---

Q3. What is the significance of the comparison results between PERSEcomp-13b and PERSEcomp-7b in Table 4? Providing t-test results can be useful.

A3. We perform the t-test between PERSEcomp-13b and PERSEcomp-7b for Table 4 with scipy.stats.ttest_ind and report the p-value here. All p values are smaller than 0.05, indicating that these two models significantly differ from each other.

---

[1] OpenAssistant Conversations--Democratizing Large Language Model Alignment.
[2] LaMP: When Large Language Models Meet Personalization

Q1. What happens if there isn't training data for PerSE in a domain?

A1. In principle, PerSE is not specifically trained to evaluate stories because the main capability PerSE gains from finetuning is to analyze the reviewers’ preferences from the prior reviews and use these preferences to evaluate new cases. The story generation is a typical generation task where personalized evaluation is important. If there is no personalized training data for evaluation, we can directly prompt our personalized fine-tuned PerSE (as what we did on GPT-4 and LLaMA baselines) to evaluate. If there is a small amount of annotated personalized data in this domain, then we can further fine-tune PerSE on this domain to improve the performance.

---

Q2. How biased is PerSE toward LLaMa generations?

A2. Based on the description in the official repo [1], the story pairs in PerDOC are generated by a LLaMA-based fine-tuned model (oasst-30b). To investigate the potential bias in the story generation model, we randomly select one of the paired plots and ask GPT-4 to rewrite it. We randomly chose 50 examples for each aspect in Per-DOC. After rewriting one of the paired plots, we asked PerSE-7b and PerSE-13b to compare them again. We calculate the percentage that PerSE mistakenly selects the LLaMA’s output (the original plot) as the better one, and the total percentage that PerSE prefers LLaMA’s output. The results are shown below. PerSE-7b is a little biased on the LLaMA-based stories, while PerSE-13b has no obvious bias. It indicates that with finetuning our PerSE is less biased towards LLaMA-based generation.

---

Q3. What is the rationale for expecting that movie review data of MPST is not a part of pretraining data for LLaMA? Is this likely to lead PerSE models to be biased similarly to GPT4 (albeit to a lesser extent due to personalization)?

A3. As we further discussed in previous Appendix A.1.2, LLaMA models have a similar bias as the GPT-4 (We moved this figure to the main body and it is Figure 3 now). This is why we use anonymization and summarization to alleviate the influence of memorization in our data construction. Therefore, our test set is less affected by the known bias and is a better benchmark to evaluate LLM’s capability of evaluation. Moreover, it ensures that our PerSE is tuned on a less biased training set.

---

Q4. Citation / link for the MPST dataset

A4.We fixed this in the revision.

---

Q5. Rewrite Section 3

A5. We have rewritten this section in our revision and see the general response for more details.

---

Q6. Discuss or experimentally demonstrate the extent to which PerSE is likely to prefer generations from a LLaMA-based text generation model.

A6. See A2 above.

---

Q7. Sec 5.1: How are the k examples from a reviewer's historical reviews used for training and evaluation selected? Is this a random set of k examples for the reviewer or is something else done?

A7. As described in Section 3, we randomly sample k reviews. It is used to simulate the inference time where the new reviewers randomly provide several reviews as their preference.

---

Q8. Discuss the implications of training PerSE on generated stories and its ability to judge generations from other systems.

A8. Here we would like to highlight that PerSE can evaluate stories generated by any system and the tuning phase will not introduce extra bias to the system it was trained on. There are several reasons:

1. As we show in A2, although PerSE is tuned on the stories generated by the LLaMA-based model, it does not show obvious preference over the LLaMA’s generations.
2. In our training data, all stories are generated by the same system (DOC framework with oasst-30b). Some generated stories are preferred and some are rejected. It is less likely for PerSE to learn a bias towards this system (for example, always give a better score on this system).
3. Even if we have multiple systems in our training data, our PerSE will not learn the bias either. One reason is that during the finetuning there is no system label to indicate where the stories are from, so PerSE cannot get such information and learn the bias. The other reason is that in the personalized setting, for the same premise with the same story pair, the preferred one could change due to the different reviewers’ preferences. So it is less likely for PerSE to learn a clear bias of always preferring one system.
4. Training on human ratings of the generated system outcome is a commonly used method in learned evaluation metrics and has shown great success and generalization ability in many metrics, such as BLEURT[2] and COMET[3].

Q9. In a future use of PerSE do you envision that researchers will train a personalized evaluation model on the ratings from a small set of annotators?

A9. As we clarify in General Response, the small set of annotators and their preferences are organized as the instructions for instruction tuning [4]. The success of instruction-tuned LLMs (such as RLHF in GPT-4) shows that with a little data, LLMs can improve the capability of following the instructions. So in future use, researchers can use the training strategy of PerSE in other domains and improve the capability of evaluating from a personalized view.

---

Q10. Why does it make sense to use a model pre-trained on one set of users to evaluate text generated for a (hypothetical) other set of users?

A10. Two settings (predicting for existing reviewers / predicting for unseen reviewers) have their own use cases. For the first setting, the model already remembers the preferences of these reviewers and directly predicts for them. Under the second setting, the model can be easily adapted to new reviewers without further training. Our PerSE focuses on the second setting. It does not try to remember the preferences of these annotators. Instead, it predicts the preferences of new reviewers that are unseen during training. In our test set, all reviewers are different from those in the training set. The experimental results show that our PerSE can be used for new annotators with new preferences.

---

Q11. The bias present in GPT4 for known plots immediately motivates a personalized evaluation - please consider rethinking/rewriting this motivation. For example, is there a simple bias correction one could apply to ratings produced from GPT4 so that known plots receive smaller scores?

A11. To create a dataset for personalized story evaluation, we need to alleviate the influence of memorization to ensure the performance of different evaluation models is not affected by the known bias. We have revised this section to make it clearer and we refer you to the general response and the revision manuscript for detailed information. We propose two techniques: anonymization and summarization to reduce the known bias and use them during our dataset construction. Figure 3 shows the effectiveness of our strategies.

---

[1] https://github.com/facebookresearch/doc-storygen-v2.
[2] BLEURT: Learning Robust Metrics for Text Generation.
[3] COMET: A Neural Framework for MT Evaluation.
[4] The Flan Collection: Designing Data and Methods for Effective Instruction Tuning.

Thanks again for the insightful feedback on our work!

We've carefully worked to address your comments/questions and revised our manuscript based on the valuable suggestions.

We would like to kindly remind you that the end of the discussion phase is coming in two days and We are happy to discuss any further concerns.