Mixed Likelihood Variational Gaussian Processes

We thank the reviewer for their constructive comments. Below we provide our response.

In said section, authors explain the advantage their method has over these baselines, and their explanation is reasonable, however, the paper could be made much stronger by empirically showing these advantages by adding them to the comparison in the experiments.

We have compared mixed likelihood training with heterogeneous multi-output GPs on the robot gait optimization data. The heterogeneous multi-output GPs use two latent GPs, and are trained using two likelihoods: a Bernoulli likelihood to fit the preference observations and a Likert scale likelihood to fit the confidence ratings. The LMC coefficients in the heterogeneous multi-output GPs are learned by maximizing the ELBO on the training data.

We note that heterogeneous GPs have lower Brier scores and higher F1 scores compared to standard variational GPs, especially when more training data is available. However, mixed likelihood trained GPs with a single shared latent consistently outperform heterogeneous GPs (with two latents). Moreover, learning two latent GPs and LMC coefficients tends to overfit, especially when the number of training data points is limited; see the F1 scores when $n = 25$ in the right panel.

https://imgur.com/a/4kubKSe

I do not like the following sentence in the abstract: "However, GPs modelling human responses typically ignore auxiliary information, including a priori domain expertise ...". One could argue the choice of GP prior (kernel and mean) is precisely supposed to capture a priori domain expertise. I believe abstract could be improved by first mentioning the model and then describing the gap it addresses in existing research, e.g. "Having different likelihood functions gives much more flexibility in encoding soft constraints and auxiliary information than the standard GP model" or something along those lines

We acknowledge that some wording in parts of the abstract is inaccurate, and we will modify the abstract accordingly.

I believe there are much earlier references for the Kriging equations than Gramacy, 2020. For example the famous Gaussian Process book by C.Rasmussen and C.Williams 2006.

Gramacy (2020) is a book that provides an overview of GPs and their applications. We will add a citation to Williams and Rasmussen (2006) here. (We did cite the book by Williams and Rasmussen (2006), but at a different location.)

In line 73, I believe the equation of kernel matrix is wrong.

Yes, this is a typo. The kernel should be evaluated on the data points, not the function values.

The ordering of subsections in section 4 is a bit weird. Section 4.1 introduces the concrete problem, then Sections 4.2 and 4.3 talk about more general concepts and then Sections 4.4 and 4.5 focus again on the concrete problems. The logical flow here is a bit unclear. I think ordering the sections as 4.2, 4.3, 4.1, 4.4, 4.5 would make more sense and would be easier to follow (start with abstract concept -> go to concrete examples)

We put Section 4.1 as the first subsection because it motivates the problem of Bernoulli level set estimation, which admittedly many readers may be unfamiliar with. Diving directly into the abstract concepts in Section 4.2 may leave readers unmotivated. Though, we do acknowledge that the ordering could be very well non-optimal. This section was a bit hard to write as we aim to compress a lot of information into it. Nevertheless, we will reassess and polish this section.

On page 3, section 4.2, I believe in the equation, $\Phi(\cdot)$ is not defined. It is defined later as a Gaussian CDF, but it should be defined here, which is where it is used first.

Thank you for pointing out this. We will add the definition at its first appearance.

Why is the focus specifically on human feedback data? I believe the proposed method can be employed in other scenarios as well, where auxiliary information or prior knowledge is available. This is by no means a criticism, merely a question.

Yes, we are also interested in finding other applications of the methods. We focus on human feedback data primarily because we have access to these types of data.

Are there any plans to release the code?

Yes, we plan to release the code upon acceptance.

As far as I understand, authors compiled an entirely novel dataset for the purpose of section 4.5. Are there any plans to release that dataset upon acceptance?

We plan to release all data used in this paper. The only exception is the haptic preference data in Section 5.3, because this is not collected by us and is out of our hands.

We thank the reviewer for their constructive comments. Below we provide our response.

Could other types of scores/scales play similar role as the Likert scale? If so, did you consider any other in particular? or is there a particular reason behind the choice of the Likert scale? If other scales are not applicable, can you clarify why?

Yes, there are other types of scales available. For example, we could use a slider scale, where the confidence rating is collected by asking subjects to drag a slider along a continuous scale from 1 to 10. The slider scale rating is continuous but non-Gaussian, as the output range is between 1 and 10. Hence, it requires designing a likelihood for the slider scale. In this paper, we choose the Likert scale due to its simplicity.

Would your method scale with high dimensional input (eg raw sensor data from a real robot)? If not, which adjustments would be needed?

Whether the model applies to raw sensor data in robotics depends on whether GPs are suitable for these types of data, which is orthogonal to our mixed likelihood training methodology. This could be an interesting direction to explore in the future.

We thank the reviewer for their constructive comments. Below we provide our response.

One question I have is about the motivation behind the Likert scale likelihood. Someone would expect the use of ordinal regression likelihood for the ratings. However, the authors claim that ordinal likelihoods "cannot be mixed directly with preference observations because they treat the latent function as an “ordinal” strength ranging over entire real numbers". Can you clarity further this point?

The ordinal likelihood assumes the probability of observing a Likert scale rating $y = i$ is
$

\Pr(y = i \mid f) = \Phi(c_i - f) - \Phi(c_{i-1} - f) \; \text{for} \; i \geq 2, \quad \text{and} \quad \Pr(y = 1 \mid f) = \Phi(c_1 - f) \; \text{for} \; i = 1,
$

where $c_i$ are cut points. Note that the latent $f$ has to have a range $\mathbb{R}$. In particular, $f$ needs to be able to go to $-\infty$. Otherwise, $\Pr(y = 1 \mid f) $ is always lower bounded by a constant, in which case the likelihood is unable to output all categorical distributions.

In preference learning, however, we want to predict the Likert scale ratings based on the preference strength, i.e., the absolute value of the latent function difference $\lvert f(x_1) - f(x_2) \rvert$. The intuition is that larger preference strength correlates with higher confidence ratings. The preference strength is a non-negative number and is not compatible with the ordinal likelihood.

If we cannot use ordinal likelihoods, then this could mean that in general mixing different likelihoods is not so easy because modeling different outputs may require modeling different "scales" of the shared GP function. However, i can imagine that there must be some principled ways to resolve this, such as by introducing some learnable scaling parameter per likelihood.

Yes, we need to be careful what likelihoods can be mixed together. At the end of the day, we need to ensure all likelihoods are able to share the same latent function. We did try to use the ordinal likelihood by applying the following transformation on the latent
$

h(\mathbf{x}_1, \mathbf{x}_2) = \log\big(\lvert f(\mathbf{x}_1) - f(\mathbf{x}_2)\rvert + \epsilon\big),
$

where $\epsilon > 0$ is a small positive constant. Now $h(\mathbf{x}_1, \mathbf{x}_2)$ has a range $\mathbb{R}$, and then we put an ordinal likelihood on $h$. However, this trick does not improve the performance. We believe it fails due to model misspecification. Even though the log transformation fixes the obvious range issue, it imposes a strong assumption that the ordinal strength (how likely the subject chooses a high confidence rating) grows logarithmically with respect to the preference strength. It is possible that there might be other non-linear transformations that could make the ordinal likelihood work, e.g., adding additional tunable parameters to the above log transformation. But the transformation cannot be a simple (linear) scaling.

We thank the reviewer for their constructive comments. Below we provide our response.

The paper introduces new concepts like the Likert likelihood without a detailed theoretical discussion on its limitations or potential drawbacks, which could leave readers with questions on its applicability in various contexts.

One limitation is that the Likert scale likelihood introduces additional hyperparameters (the cut points). Thus, this may increase the risk of overfitting in the case of limited data. A concrete example is subject #2 in Section 5.2, where the mixed likelihood-trained model fails to improve the performance. This is because subject #2 is predominantly confident and has a sharply different confidence distribution from other subjects; see Figure 9 in the appendix. In particular, they reported no ratings $\leq 3$ at all. As a result, the Likert scale likelihood struggles to estimate the cut points.

Handling of Noisy or Inconsistent Feedback: How does your model handle situations where human feedback is noisy or inconsistent? Are there any mechanisms in place to detect or mitigate such issues, and how does this affect the model's overall performance in such cases?

1. The Bernoulli likelihood (and the preference likelihood) models the response as a Bernoulli distribution, which handles noise through the stochasticity in the likelihood by design.
2. The Likert scale likelihood Eq (2) also handles noise by stochasticity, as the discrete distribution Eq (2) assigns nonzero probabilities to all possible options.
3. In addition, the categorical distribution Eq (2) outputted by the Likert scale likelihood is damped with a uniform distribution using a lapse rate for enhanced robustness to outliers; see Line 666 in the appendix.
4. By their nature, all real-world data used in this paper are inherently noisy, as human responses are often inconsistent. The fact that our models improve the performance across different tasks already demonstrates their abilities to handle noise.

Generalizability to Other Feedback Types: Have you considered testing the proposed framework on other types of human feedback (e.g., response times, eye-tracking data, or even more complex preference signals)?

We have implemented a response time likelihood and tested it on the robot gait data released by Shvartsman et al. (2024). The likelihood is based on the assumption that the response time $t$ follows a log normal distribution. We observe that our response time likelihood does improve the performance (lower Brier scores and higher F1 scores). However, it was not fully developed before the submission deadline and thus we did not include it in the paper.

https://imgur.com/a/qezbyeD

Shvartsman, M., Letham, B., Bakshy, E., & Keeley, S. L. (2024). Response time improves gaussian process models for perception and preferences. In The 40th Conference on Uncertainty in Artificial Intelligence.