Preference Learning of Latent Decision Utilities with a Human-like Model of Preferential Choice

Thank you for your thoughtful review. We address specific questions below:

Limited Applicability: The models require data that includes the orderings of each feature […] and the true utilities

This is incorrect.The data we used was straightforward choice data consisting of the options presented to the user and the choices they made. While our cognitive model does assume that choices are informed by the ordering and utility observations you mention, these observations happen “inside the choice maker’s head” and are thus latent to an outside observer (see eq. (3)). We will update the paper to be more explicit about this. We have also included graphical models (Figure 1b and 1c in the rebuttal PDF document attached to the general comment) which clarify which variables are observed and which are not to both the choice maker and an outside observer.

The experimental results primarily focus on negative log-likelihoods (NLLs). […] Comparing other metrics (e.g., accuracy) would better demonstrate the contribution of this work.

The evaluation of our model and the baselines on choice data sets follows prior work [18]. All models we consider predict a probability for each choice (they are probabilistic classifiers), rather than a single deterministic choice. This is important as human choices do not appear to be deterministic. Thus, our main concern is that these models should be well-calibrated. This is not something we can check with a measure like accuracy. Rather, it requires a proper scoring rule, like the NLL we used. In our other experiments - which were more representative of a typical preference learning setting - we did use other metrics to show significant practical improvement. For example, we measured how consistent a ranking implied by the inferred utility is with ranking collected from users in section 4.2.1.

the paper does not provide standard deviations for the results to indicate robustness, even though the results are statistically significant.

We agree that this would be a good supplement to the statistical significance tests already carried out in the paper. We will add to the appendix a table listing the mean and standard deviations of the averaged negative log-likelihoods obtained over the test folds in the choice data experiments (section 4.2). A separate set of utility and choice model parameters was inferred for each test fold, so this should give readers a good idea of the robustness of the models. We have added this table to the PDF attached to the global response (Table 1).

[regarding L561: doing gradient descent multiple times] It would be helpful to show the mean and standard deviations over several random seeds instead of just the best result to demonstrate training robustness.

We should stress that we used gradient descent multiple times to infer the latent choice model and utility parameters from human data, not to train the CRCS model itself. Even on the largest choice dataset, each inference run took only a few minutes. As requested, we will add a table to the appendices showing mean and standard deviations of the average NLL obtained over several independent gradient descent runs. We have added this table to the PDF attached to the global response (Table 2).

How can the true utility or its generation process be observed or estimated in real-world scenarios?

This question gets to the heart of what makes human modeling difficult. As we cannot look inside people’s heads, it is impossible to observe humans’ true utilities or their true choice making process. Modeling lust follow the standard scientific process of proposing a new model based on some theory or assumptions of how this process works, and then evaluating that model against competing baselines. A choice model (including the underlying utility function) can be evaluated in two ways: i) we can check if the model correctly predicts people’s choices (experiments in section 4.2 and 4.3) and ii) we can check if the inferred utility is consistent with other observations that are derived from that same utility function (e.g., the evaluation of the implied rankings we performed in section 4.2.1).

How should the prior distributions in Table 3 be selected or estimated? Are there general methods for estimating these prior distributions from a given dataset?

The priors for the choice model and utility parameters have very little effect on the model itself. However, as they are used when training the model, it is important that they cover any parameter values that we may wish to use in practice. We followed standard Bayesian practice for selecting them. As we had little prior knowledge of which parameter values were most likely in practice (beyond some insights from Howes et al. [17] regarding which parameter values actually affected the model’s output), we used flat priors that were as broad as sensible (see last paragraph of appendix A.1).

What are the architectures of neural networks $\hat{q}$ and $\hat{u}$.

We will add these details to the appendix. We have provided an overview of the architecture of both networks in the general response and a visualization in Figure 1a in the accompanying PDF.

Hello reviewer hXMa: The authors have responded to your comments. I would expect you to respond in kind.

Thank you for the review of our paper and thoughtful feedback. We address your main questions and feedback below:

I find this work to be a nice example of cross-disciplinary ML project: model from psychology, use of variational methods to make it computationally feasible, diverse set of experiments.

Thank you for the positive feedback!

What is the number of parameters for each of the models? [what are] the network architectures for $\hat{u}$ and $\hat{q}$.

We will include details regarding network architectures and sizes to the appendix. We have provided a summary of these details in the general response. In the PDF attached to the general response, you can find the number of parameters for each model in Table 3, and a visualization of the network architecture in Figure 1a.

Have you thought [of] applying the model to LLMs for preference learning?

This is a very good point. LLMs have had a tremendous impact on preference learning research and properly aligning them to human preferences is an important societal problem. Our results suggest that modeling context effects is important when learning from human preferences, and so we do consider applying our model to LLMs important future work. Right now our model assumes that options are represented by features. Thus, given some featurization of, say, LLM text outputs for a given prompt, our model could be directly applied. However, this would ignore the reading and interpreting of these texts that human evaluators have to do. We think additionally modeling those tasks in the choice model, by extending the foundation provided by the current model, could be even more transformational. Based on what we have written here, we will add a more detailed discussion on the implications of our work to LLM fine-tuning to the related work section.

the meaning of CRCS acronym is not introduced until line 125.

Thank you for pointing this out. We will introduce the acronym earlier.

a graphical model describing all the variables in the model can be useful

Agreed! We have added two graphical models in the PDF attached to the global response. One shows the problem from the point of view of the user making a choice (Figure 1b in the attached PDF) and the other shows it from the point of view of an outside observer trying to infer the user’s utility function (Figure 1c in the attached PDF). We will add these figures to the appendix of the paper.

line 153: when reading this paragraph for the first time, I found it strange that the parameters are observed (known). It then became clear that they are observed only by the user that is being modelled.

Thank you for pointing this out. As another reviewer also got confused by this, we will include the graphical models and also update the writing to clarify that these observations indeed happen in the user's head and are not observable from the outside.

Hello reviewer cVZM: The authors have responded to your comments. I would expect you to respond in kind.

Thank you for your review of our paper and your feedback. We have addressed the main questions and concerns you have raised below.

the policy network takes in $x, w, \tilde{u}$ and $\tilde{o}$, and a sufficiently flexible surrogate should be able to learn arbitrary functions of x and w already. Why is the explicit feature engineering needed? [what is meant by] "built in" context effects in CRCS and "learned" ones in [LC]-CRCS.

Thank you for pointing out that this was unclear. To clarify, the observations $\tilde{u}$ and $\tilde{o}$ are not engineered features, but rather a core part of our cognitive model. As explained in section 3.1, the cognitive model theorizes that context effects come from the fact that humans make choices that maximize option utilities estimated from noisy observations [17], specifically noisy observations of each option’s utility ($\tilde{u}$) and noisy feature value comparisons across options ($\tilde{o}$). It has been empirically validated that maximizing these utility estimates given these observations leads to the context effects exhibited by humans. Our tractable surrogate for this model, CRCS, inherits these same context effects, which is why we describe them as “built in”. However, we found that this model did not reproduce all context effects present in the considered choice data. LC-CRCS addresses this by introducing a learnable component that is able to learn context effects not yet captured by CRCS itself. The advantage of having a model where these context effects are “built in” is that this creates inductive biases that lead to better utility function inference and choice prediction, as we show in the experiments.

This also makes me wonder: what's the network architecture used and other training hyperparameters (optimizer, learning rates, etc)?

Although these details are available in the code we submitted in the supplement, we do agree that it would be easier for readers if they were available as part of the appendices as well. We will add details on the network architecture and sizes to the Appendix. We have provided a summary in the general response.

In general I would expect a sufficiently flexible surrogate to also achieve better than ~92% agreement with the original model.

We used the most stringent possible definition of agreement, namely that both models had to agree on the full ordering of the three option utilities. This is clearly a high bar. Moreover, the original model we compared to uses a Monte-Carlo estimator that estimates the expected values of the option utilities from a large but finite number of samples. Thus, the quantity we compare to is not some ground truth reference but rather another (noisy) estimate. As any error could be due to either the proposed surrogate or the original model being wrong, even a perfect surrogate would likely not reach 100% agreement.

I'm also puzzled about the claim of insufficient option data to [train] the utility network -- shouldn't it be possible to simulate arbitrary amounts of observations[...]? I can see why sampling only from the training data would create this restriction, but I don't understand why this choice was made.

Unfortunately, in this specific instance, there was insufficient detail available regarding how the original user study, reported in [37], was done; no details were available on car and participant features that were used to construct the original option sets. We were therefore unable to generate option sets from a distribution that aligned well with the original experimental condition, forcing us to sample them from the training data only. This was not an issue for the other choice tasks, where the details of the user studies were more fully documented and we were able to simulate an arbitrary number of choices. Appendix A.1 discusses these issues in more depth.

Reporting summed rather than averaged NLLs (with standard errors) seems like an odd choice -- why was this done?

We reported summed NLLs in the same way as was done in prior choice modeling work [18]. However, we agree that additionally reporting standard deviations or errors would add value to the paper, especially - as reviewer hXMa explained - to help readers understand the robustness of the models to variations in the training data. We will add to the appendix a table listing the mean and standard deviations of the averaged NLLs obtained over the test folds in the choice data experiments (section 4.2). You can find this information in table 1 in the PDF attached to the global response.

Can the authors comment about the choice of Wilcoxon signed-rank test for some hypothesis tests and t-tests for others?

In most of our experiments we compared different model scores (e.g. NLL) on the same static data, and thus we used paired two-sample tests to compare the scores between models. As normality generally did not hold, we used Wilcoxon signed-rank tests for this. For experiment 4.3, where we used active learning, the evaluation data differed between the models as each would have selected different points to train on, thus we used an unpaired two-sample test, in this case an independent t-test. We have noticed now that we misreported the hypothesis tests used in sections 4.2.1 and 4.3 – these should be switched. We used an independent t-test in 4.3 and a Wilcoxon signed rank test in 4.2.1. We will fix this in the paper and will explain briefly why each hypothesis test was chosen.

Why are baselines not present in the regret panels of figure 1?

Comparing to baselines was not the focus of these experiments. These are synthetic tests where we measured parameter recovery and practicality of CRCS in real-world experiments. No human data was used, and instead choices were generated from our own CRCS model. Clearly, our own model will fit best to choices generated by it, so any comparison to the baselines seemed to us as though it would be disingenuous, in this instance.

Hello reviewer CwtL: The authors have responded to your comments. I would expect you to respond in kind.

Thank you for your comment. We apologize for misunderstanding your original question regarding LC-CRCS.

Fundamentally, CRCS as we defined it does not learn context effects from (human) data. Instead, context effects emerge given an empirically validated theory (choices maximise expected utility given noisy observations; section 3.1). This theory can be simulated and thus be amortised into \hat{q}. $\hat{q}$ can therefore be trained without any access to human data. There are a number of free variables in the model (the choice model and utility parameters) which are inferred from human data in our experiments, but these determine the utility function and various noise levels within the choice process, and are thus not directly responsible for the context effects. Although not learning context effects from human data has clear advantages, it also means that the model will not generate context effects not explained by the theory. This is why we introduced the additional linear mapping $g(w,x)$ in LC-CRCS. This can be learnt from human data and can therefore fit to context effects not yet explained by the theory or generated by CRCS.

Thank you for the review and feedback on our paper. We address specific questions below:

While it was easy to follow the theory, I found it a little difficult to deeply understand the application section. A little more focus on methodology on at least one experiment might have been more helpful. Specifically, it was a little tricky to cleanly connect the theory to the application for me.

Thank you for this feedback. We will update the paper to more explicitly connect the data available in sections 4.2 and 4.3 to the observed and unobserved variables discussed in section 3: In 4.2 we are given a large dataset of tuples $(x^{l},y^{l})$ where each $x^{l}$ is a set of options presented to a user and each corresponding $y^{l}$ is the observed choice. Given this data we now look to infer the unobserved utility parameters $w$ and choice model parameters (e.g. $\varepsilon, \boldsymbol{\tau}, \sigma^2_{calc}$ in the case of CRCS) that best explain this data. The difference in 4.3 is that the dataset is not given all at once, rather we have access to the option sets $x^{l}$ and can actively select which $y^{l})$ we want to observe.

The contribution is somewhat incremental [...] Since the Bayesian model already exists, the main contribution is estimating the two intractable quantities using NNs, [...] It’s useful to have such models, but I am not sure if they generate any foundational insights.

We disagree that the contribution is incremental. The foundational insight is that a general theory of human cognition - namely computational rationality - offers a powerful and general inductive bias for machine learning. We study how to mitigate the effect context effects - a human bias widely studied in cognitive science - have on utility inference from preferences. Whereas prior work has attempted to account for these effects by learning them from data without any such inductive biases [18,35], we follow a model-based approach which leverages an existing cognitive model introduced by Howes et al. [17]. This model, which is an instantiation of computational rationality for decision making, is empirically validated to reproduce known context effects. This provides us with strong inductive biases that significantly improve utility function inference and choice prediction compared to prior work, showing that our model-based approach works for the important problem of learning from preferences. But the cognitive science literature suggests that deriving inductive biases from computational rationality can generalize to any setting in which people make decisions [A,B,C]. This is in our view a foundational contribution to both cognitive science and machine learning. We will discuss this contribution more explicitly in the introduction and discussion of the paper.

We also count a number of other tangible contributions. We have introduced a new choice model, CRCS, which improves utility function inference in preference learning compared to prior work. This required making the model introduced by Howes et al. [17] tractable using surrogates. We have also provided further empirical evidence for this specific model. This is significant because SOTA cognitive models such as [17, 38] are seldom tested on large choice, mainly because inference is generally intractable. Lastly, we have shown that there are context effects that the original Howes et al. [17] model does not capture, and have introduced LC-CRCS which introduced a learnable mechanism that can fit to these additional context effects.

[A] Richard L. L., Howes A., and Singh S. "Computational rationality: Linking mechanism and behavior through bounded utility maximization." Topics in cognitive science 6.2 (2014): 279-311.

[B] Lieder F., and Griffiths TL. "Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources." Behavioral and brain sciences 43 (2020): e1.

[C] Oulasvirta A., Jokinen JPP., and Howes A. "Computational rationality as a theory of interaction." Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022.

[The paper] might be of interest to a smaller subset of the community [...] might be better suited for area-specific workshop.

We disagree. Learning from preferences is an increasingly important approach in human-in-the-loop machine learning for inferring unknown utility functions. The modeling of those preferences (choices) is therefore relevant to a wide part of the ML community. We see this reflected in the fact that earlier work on choice modeling, which we improve on, has been published at top-level ML conferences such as ICML [35] and KDD [18]. Furthermore, our work fits within a wider body of recent work published at NeurIPS on human behavior modeling [D,E,F,G].

[D] Belousov B., et al. "Catching heuristics are optimal control policies." Advances in neural information processing systems 29 (2016).

[E] Binz M., and Schulz E. "Modeling human exploration through resource-rational reinforcement learning." Advances in neural information processing systems 35 (2022).

[F] Chandra, K., et al. "Inferring the future by imagining the past." Advances in Neural Information Processing Systems 36 (2023).

[G] Teng T., Kevin L., and Hang Z. "Bounded rationality in structured density estimation." Advances in Neural Information Processing Systems 36 (2024).

For synthetic data, if the generative model is closer to the inference model, it’s not very surprising that the latter does better than another model unaware of the generative process. This weakens the evidence on synthetic data a little.

We do not agree that this weakens the evidence. It is clear that aligning a system to a user’s utility will make the system more useful. However, how to achieve such alignment in practice is not trivial. The simulated use cases we present point to the fact that preference learning with our model is a practical solution for this.

Hello reviewer rsF2: The authors have responded to your comments. I would expect you to respond in kind.