Personalized Language Generation via Bayesian Metric Augmented Retrieval

Dear reviewer,

We appreciate that you review our paper and provide insightful comments. We would like to address your concerns as follows:

Did you perform statistical tests on Table 1 and 2? If so, please provide details.

We propose to look at the paired difference of the performance metrics: for each subject, we compute our method's HR@3, Rouge1 and the competing method’s HR@3, Rouge1. We propose to test the hypotheses:

Null hypothesis: Ours HR@3 and Rouge1 score equal competing method’s HR@3 and Rouge1

Alternative hypothesis: Ours HR@3 and Rouge1 are greater than competing method’s HR@3 and Rouge1.

In order to test the above hypothesis, we use a one-sided Wilcoxon signed-rank test to compare the paired performance metric values. The p-value of the test between our method and the baselines is reported in the following table.

Suppose we choose the significant level at 0.05. The table indicates that our method significantly outperforms baselines across all datasets.

Dear Reviewer, thank you for your feedback on our submission. Regarding the missing qualitative evaluation, we would like to provide an illustrative example from the Law dataset. Consider the query 'Explain Criminal Law to me', posed by a user whose profession is a Dentist. Our system knows nothing of this user but a few of his liked or disliked documents (a total of 18 documents in our experiment). Our system uniquely produces documents with analogies pertinent to medical or health-related fields, such as patients or treatment or therapy. This illustrates how our method tailors content to user-specific contexts, even with limited information about the user.

Our method:

Criminal law refers to the body of laws that define and regulate behaviors that are considered to be crimes. You may encounter the concept of criminal law in your practice when dealing with **patients** who have engaged in illegal activities. Your expertise can be crucial in providing insights into their mental capacity, potential mitigating factors, or the need for **treatment or therapy**. 

Euclidean:

Criminal law is a branch of law that deals with offenses committed against society as a whole, rather than individual disputes. This is similar to how physics principles, like the laws of motion, help maintain order and predict the movement of objects in the universe.

BM25:

Criminal Law is like a set of guidelines that defines what actions are considered against the law and can result in punishment. Just like there are certain rules and principles you follow while creating your beautiful artwork, there are rules in society that everyone must abide by. You may find Criminal Law relevant to your work in developing AI systems that interact with society.

MIPS:

Criminal law is a branch of law that deals with offenses committed against society as a whole, rather than individual disputes. By incorporating the principles of criminal law into your storytelling, you can create suspenseful plots, explore moral dilemmas, and delve into the psychological motivations behind criminal behavior. 

Dear reviewer NT9n,

Here we show another example in the Politics Dataset, whose user identity and concept are more widely known. In this example, we observe that only our method generates content that is aligned with the common objective of an online content creator, which is to build trust and loyalty or appeal to your target audience

True user identity: Online Content Creator

Query: "Explain women's reproductive rights"

Our method:

Women's reproductive rights include access to contraception, family planning, and maternity care. They also include the freedom to make choices about when to have children and how many. Some places have restrictions on access to reproductive healthcare, limiting women's choices and jeopardizing their well-being. By highlighting women's rights, you can **appeal to your target audience of women** who want to make decisions that align with their personal goals and aspirations. In summary, understanding and respecting women's reproductive rights can **build trust and loyalty**.

Euclidean:

Reproductive rights refer to a set of freedoms that women have regarding their reproductive health and choices. The lack of access to contraception and abortion services has historically limited women's ability to control their fertility and plan their families. By supporting women's reproductive rights, we promote equality and empower women to have control over their own lives. By studying the struggles and triumphs of the past, historians can provide valuable insights into the evolution of women’s reproductive rights.

BM25:

Women have the right to make decisions about their own reproductive health and rights. You can help by examining the impact of different levels of reproductive rights. For example, you can examine the effect of different types of birth control on the rate of unwanted pregnancies. You could also ask questions about the effectiveness of different countries' reproductive rights laws, such as those that allow women to choose not to give birth to their children in the first place. 

MIPS:

Women's reproductive rights are crucial because they ensure that women have autonomy over their bodies. The lack of access to contraception and abortion services has historically limited women's ability to control their fertility and plan their families. By studying the struggles and triumphs of the past, historians can provide valuable insights into the evolution of women’s reproductive rights. By recognizing and respecting women's reproductive rights, we empower them to take control of their lives, pursue their agricultural endeavors, and contribute to the sustainable development of our communities. 

Dear reviewer,

We appreciate that you review our paper and provide insightful comments. We would like to address your concerns as follows:

Am I understanding correctly that the 5000 large document collection contains the 50 concept definitions rephrased in 100 different ways?

Yes, it is correct. As mentioned in Section 6.1, we synthesize 3 datasets (Political, Legal and Technology) with that setup.

Am I understanding correctly that the proposed approach never uses the pre-trained summarization model in learning user preferences?

Yes, it is correct. However, we did use a pretrained summarization model to shorten documents for practical purposes.

Is the user feedback for Phase 2 obtained from labelled data? Eg if the document corresponds to the correct concept+user it is a positive document and negative if not?

A user likes a document if that document lies in the top 5 documents that are closest to the user’s true document in terms of Euclidean distance; A user dislikes a document otherwise.

Please clarify in greater detail what data is used for training and evaluation. "For each user type, we systematically exclude all documents associated with that user type from the database, treating them as reference documents" -- its not clear what this means? Is this implying that no documents corresponding to the user type are available for training?

Yes, the document corresponding to the user type will not be used for training but only used as a reference document to compute the top-5 documents that will be liked by the user.

Sec 6.1: What reference documents are used for summarization evaluation? Eg in computing metrics like rouge.

The document corresponding to the user type will be used as a reference document to compute rouge scores for the generated answer.

If the experimental setup contains as input a query + positive document for user type - It would be illustrative to have a simple personalized ranking baseline that computes a score based on a simple linear interpolation concept-document score and a concept-user_type score. Perhaps a user_type could be represented with the average of document embeddings for that user and the weight for the two components tuned on a dev set. Alternatively, please consider formulating an appropriate, simpler personalized ranking baseline and comparing to it in addition to the ones in the paper.

We consider fitting user preference feedback to a simple logistic regression binary classifier. Please refer to the table below for results on all three datasets.

Also consider discussing differences to the task/approach of: https://arxiv.org/abs/2211.09260

The work in Task-aware Retrieval with Instructions by Asai et al. (2022) adapts its search results using an additional search task instruction $t$ in natural language, along with the user's query $q$ (see Section 3, 'Task Formulation'). Our work differs from that of Asai et al. (2022) in that we rely on the users' ratings (like/dislike) on a selection of summarized documents, instead of using the task instruction $t$.

The other issue is the dataset: it's entirely synthetic. There is no evidence that the proposed model would be useful at reflecting real users preferences.

We do not fully understand the concern here.

Our ultimate goal is to generate high-quality text (meaning that the generated text can be aligned better with the user’s preference). We do not have to reflect 100 percent correctly the user’s preferences in order to generate high-quality text. Our model is a model, it is an abstraction of the real world, it may be wrong, but it is useful.

There are a plethora of examples that follow the same argument, let us describe a few:

Discrete choice: The field of discrete choice builds abstract models that can help to predict the user’s choice behaviors (whether the person buys a certain product or not). For sure, it is impossible to reflect accurately the users preferences, nevertheless, discrete choice models can have reasonably good performance in predicting the user’s buy/not buy decision. Discrete choice models are also used a lot in marketing to personalized advertisement. Unfortunately, discrete choice model also require a large number of samples and/or contextual information to perform well, and does not fit into our setting. Portfolio allocation: It is also impossible to learn the risk appetite of the investors. Nevertheless, it does not stop us from building proxies (different utilities functions), that can help us optimize the portfolio allocation.

Regarding the synthetic data concern: there are a growing literature on using GPT to generate datasets, for both training and testing. We list a few here:

ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (Hartvigsen et al., ACL 2022)

A Synthetic Data Generation Framework for Grounded Dialogues (Bao et al., ACL 2023)

Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., ACL 2023)

Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text (Dou et al., ACL 2022)

Dear reviewer,

We appreciate that you review our paper and provide insightful comments. We would like to address your concerns as follows:

What happens when you fit a simple linear model to users original preferences and update in an online fashion?

As mentioned in Section 6.2, we only ask users to rate a total of 18 shortened documents (N_c = 3, C1 = 3, C2 = 3). This number of samples is much too low for fitting a simple logistic regression binary classifier (with two classes, like or dislike) and will lead to overfitting. We, anyways, have implemented the baseline that you proposed, in which we learn a logistic regression model to predict whether the user likes or dislikes each document. We conduct the comparison on the Politics, Law and Technology dataset. The table below presents the performance between two models. We can clearly observe that our proposed method dominates the simple logistic regression baseline.

Why are we modeling the user preference as a Wishart probability distribution?

Because a popular approach for document retrieval is the nearest neighbor approach. The standard, off-the-shelf implementation uses an Euclidean distance. Thus, a reasonable approach to cater the user preference in the retrieval procedure is to alternate the distance. We thus need to parametrize the distance so that we can perform the learning process. Mahalanobis distance is the most natural way to generalize the Euclidean distance, and a Mahalanobis distance is parametrized by a positive semidefinite matrix (this is the matrix $A$ in our paper). To invoke a Bayesian method, we need to impose a prior and a posterior on the parameter space, and the parameter space in our case is the set of all a positive semidefinite matrices. And the Wishart distribution is, again, the most natural distribution supported on the positive semidefinite matrix space. We hope that this line of argument clarifies your concern.

Why do we need a determinantal point process? How would that compare to random sampling?

Detrimental Point Process is needed because when a user arrives, there is no user preference information that we can use to personalize the retrieval process. This is the cold-start setting, and diversifying the retrieval via DPP is a well-known and powerful technique in this cold-start situation. Many papers have demonstrated that DPP delivers better performance than random sampling, including: [1] Wang et al. (2020) Personalized Re-ranking for Improving Diversity in Live Recommender Systems, arXiv:2004.06390 [2] Liu et al. (2022) Determinantal Point Process Likelihoods for Sequential Recommendation, arXiv:2204.11562 [3] Wilhelm et al. (2018) Practical Diversified Recommendations on YouTube with Determinantal Point Processes. When we design our framework, we have already optimized the choice of the component. That is why we use a determinantal point process. We will add this argument to the revision.

Accidentally the review appearing here is not for this paper but for an other paper assigned to me. The review fro this paper "Personalized Language Generation via Bayesian Metric Augmented Retrieval" is also linked to an other paper.