Preferential Normalizing Flows

In Proposition 2.1, it is not very clear what is the relationship between W and the limit $\lim_{\beta \rightarrow 0} p(\mathcal{D}) = 1$.

As the “noise level” $\beta$ goes zero, there is no noise in RUM (i.e. $W=0$ with high probability), which implies that the expert always chooses $\mathbf{x}$ which maximizes their utility / belief density, that is $argmax_{\mathbf{x}} p_{\star}(\mathbf{x})$. Only observations consistent with argmax have non-zero probability and hence the probability of the only possible data collection is one. Since argmax is invariant to monotonic transformations, one cannot identify the target belief density, or it is unidentifiable at least up to a monotonic transformation.

The symbol $\mathcal{D}$ is used in two places, i.e., Proposition 2.1 and Section 4.1

We modified the notation in Proposition 2.1 from $\mathcal{D}$ to $\mathcal{D}_{\textrm{pref}}$.

In Assumption 3, what is $Exp(s)$

Exponential distribution with a rate parameter s>0. We will clarify this in the final version.

In Proposition 3.5, is $s()$ a function or a constant?

It is a constant, the rate parameter of the exponential distribution. The equation reads as $s$ times $f(\mathbf{x}) - f(\mathbf{x}_j)$, the parenthesis being for the difference and not indicating a function.

In Eq.4, Is the numerator $\exp{(sf(x)\lambda(x))}$?

No, the equation in the paper is correct, i.e. the integrand in the numerator is $\exp{(sf(\mathbf{x}))\lambda(\mathbf{x})}$. Note that $f(\mathbf{x}) := \log p_{\star}(\mathbf{x})$, whereas $\lambda(x)$ is directly a density.

The symbol $\mathbb{x}$ is used in several places. Sometimes, it represents a random variable, sometimes, it represents a sample.

We used $X$ for a random variable and $\mathbf{X}$ for a set of samples (design matrix). To avoid confusion, we changed our notation so that the random variable is now denoted by $\mathbb{X}$.

In Eq. 6, is $f_i = f(x_i)$?

Yes. We will clarify this in the final version.

...Based on Eq. 1, the $f$ is a continuous function. Based on Eq. 6, $p(f) \propto \prod \exp{f_i}$. Therefore, why do we need normalizing flows to represent $f$...

The critical requirement is that the result has to be a density. More precisely, $\exp(f)$ needs to normalised over the argument x, i.e. $\int \exp(f(\mathbf{x}))d\mathbf{x} = 1$. Normalising flows provide a natural tool for representing densities, whereas for an arbitrary network we would need explicit normalization that is not computationally feasible.

Further, Eq. (6) relates to the functional prior $p(f)$, which defines a probability density over the function values.

What’s the meaning of Theorem 3.6. I may have missed something, but I did not see any discussion or analysis on this theorem.

Theorem 3.6. provides the formula for the k-wise winner distribution in the limit of $k$ (number of alternatives) approaches infinity. We use the limit result to construct the functional prior, which helps to solve the collapsing and diverging mass problem. Even though we eventually use only small $k$, we can use the limit distribution as a prior, because we can temper the finite-k distribution to resemble the limit (Figure 2).

The paper mentions that one challenge of this task is samples presented for the expert might be drawn from an independent and known distribution. How this challenge is addressed by the proposed method?

The whole method is designed specifically to address this challenge. If the samples were drawn from the target itself then standard flow learning would be sufficient as a solution, whereas learning the flow from preferential responses for samples drawn from any other distribution requires the full machinery we introduce. In other words, the challenge is addressed by the combination of the RUM model for preferential data and interpretation of the k-wise distribution as tilted version of the belief as described in Section 3, as well as the functional prior presented in Section 4. Further, the new ablation study (Figure R1) provides some insights on how the problem becomes more challenging when sampling distribution for the candidates is very different from the target density. Yet, the proposed method is able infer the target density, although the estimate is not accurate as it would be when the sampling distribution is closer to the target.

Are there any existing methods that can be used as baselines?

Majority of prior elicitation works assume fixed prior family (e.g. a Gaussian) and would not be fair baselines; they can be made arbitrarily bad by making a poor choice of the distribution. Furthermore, there are no specific methods that learn from the preferential comparisons and hence conducting such a comparison would require deriving details for a new method anyway. While there are some methods that can estimate flexible densities (the GP-based methods we cited in Introduction), they all require completely different kind of input information and hence cannot be compared against in our setting. We learn from preferential comparisons, whereas e.g. Oakley & O'Hagan (2007) learn from percentiles.

Even though there are no natural baselines, we hope that the additional experiments relating to sensitivity of the method in terms of the key parameters ($k$, $n$, $\lambda(x)$) address in part the same request.

Do we have any quantitative metrics to evaluate the proposed method?

Since the problem is a density estimation problem (from a preference data), a natural metric is the distance between the estimated density and the target density. The proposed method produces relatively low (Wasserstein and the MMTV) distances between the estimated flow density and the target density in the experiments.

References

Oakley, J. E., & O'Hagan, A. (2007). Uncertainty in prior elicitations: a nonparametric approach. Biometrika

...Because of the underlying data model for preference data (eq. 3), I suspect that the method here is actually optimizing a lower bound on the log likelihood... Specifically, their objective may be similar to the Max surjective flow described in Nielsen et. al. [2020, https://arxiv.org/abs/2007.02731]...

There are some similarities between the Max surjection transformation and the preferential likelihood given by (noiseless) RUM, which is essentially an argmax operator. However, the main difference is that the Max surjection transformation is a transformation that can be applied to any level of a composable transformation (denoted by $T$ in the manuscript, by $f$ in (Nielsen et. al., 2020)) with available likelihood contribution (Table 2, Nielsen et. al., 2020), while the RUM model conditions on the whole $T$ and computes a (non-additive) likelihood contribution for the whole preferential flow.

In SurVAE Flows, the likelihood contribution of a deterministic bijective flow is the likelihood of a datapoint $\mathbf{x}$ given by the flow density $p$ at that point, that is $p(\mathbf{x})$ computed using the change of variable formula, which results in the Jacobian of $T^{-1}$. In contrast, a preferential flow uses a probabilistic modelling approach by having an additional likelihood function $\mathcal{L}$ which computes the likelihood contribution of (preference) data $\mathbf{x}$ conditioned on the flow density $p$, that is $\mathcal{L}(\mathbf{x} \mid p)$. In a preferential flow, although the mapping $T$ from a latent point $\mathbf{z}$ to a datapoint $\mathbf{x}$ is a deterministic bijective function, the likelihood contribution of preference data does not directly involve the Jacobian of $T^{-1}$, unlike in SurVAE Flows (Algorithm 1, Nielsen et. al., 2020).

Thus, the likelihood contribution is exact and the objective is not a lower bound. We extended the discussion section of the manuscript to discuss the relationship between SurVAE Flows and preferential flows.

The Appendix explores various settings of the parameter $s$ when the true value of $s$ varies, which shows that their choice to fix $s=1$ is reasonable. However, the authors almost mention using using $\lambda \propto 1$ as well to maintain tractability. The implications of this choice are unclear, did the authors run any validity checks to test the implications of their choice?

We indeed use $\lambda \propto 1$ for construction of the functional prior, but note that in all of the experiments we used as $\lambda(x)$ a mixture of a bounded uniform and a Gaussian. That is, we already conducted the experiments under conditions where the choice does not hold.

To further investigate the effect of the true sampling distribution, we now conducted an additional ablation study, where the true $\lambda(x)$ is varied. See the global response for description of the experiment and summary of the results confirming that the method works for a range of choices.

...The paper only considers the case of a single expert, with a flow model to that experts belief density. The authors may wish to comment on future work in this area — specifically, if their approach could be adapted to a multi-expert setting, if a single flow model is still sufficient or if a flow mixture model is preferred.

This is a good idea and we will extend the Discussion to cover also multi-expert settings.

The most common methods for aggregating expertise of multiple experts are behavioral aggregation and mathematical aggregation. The former is based on experts discussing their opinions and making consensus judgments for which an aggregate distribution is fitted (O'Hagan, 2019). This is agnostic to the elicitation algorithm and our method with a single flow could readily be used to capture the consensus. For mathematical aggregation we need a pooling rule that combines the elicited densities into a single one (EFSA, 2014). We think that a mixture of flows, which is equivalent to a linear pooling over multiple elicited flow densities, with equal mixture weights, would be a good default option. Further, Bayesian pooling such as having a hierarchical model on the joint preferential data from multiple experts could be investigated.

References

EFSA, E. F. S. A. (2014). “Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment.” EFSA Journal, 12(6): 3734. 2, 16\ O’Hagan, A. (2019). Expert knowledge elicitation: subjective but scientific. The American Statistician, 73(sup1), 69-81.

...At the moment all experiments use k = 5 and it would be interesting to see how the results vary with different values of k. Specifically, k = 2 is a very relevant real-life use case as this type of feedback is given a lot in e.g. chatbots. Similarly, it would be good to get some idea of how the flows converge to the ground truth density as a function of the number of preferential samples.

Thank you for the excellent suggestion. We re-ran both Onemoon2D with $k=2,3,5,10$ and LLM experiment with $k=2$ to $k=5$, as explained in the global response. The key result is that the method indeed works already with $k=2$, though naturally not as well. We also studied the convergence as a function of $n$, again describing the experiment and the results in the global response.

Minor points and typos:

Thank you for pointing out these errors. We corrected the errors for the revised manuscript and increased the font size of Fig. 1.

Is assumption 3 satisfied in the LLM experiment? If not, how does this affect the results?

The RUM model with this specific noise distribution serves as a theoretical model for the expert's choice, so it is highly unlikely that assumption 3 is satisfied in the LLM experiment. In fact, it most likely does not hold exactly for humans either. Hence this experiment shows that even under model misspecification that can be expected also in the real use-cases we can get sensible results. We hypothesize that a potential violation of assumption 3 can have similar effect as misspecification of the noise level, which can result in too high (or low) spread in the estimated flows (see e.g. 2nd col, 3rd row in Figure A.7).