AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modelling

We would like to thank the reviewer for their time and comments to improve our paper. We are pleased that they felt that this was commendable work that is well-written, easy to follow, and highly relevant for healthcare. We are glad that the methods were considered straight-forward, but effective, and that reviewer felt that the extensive experiments demonstrated the utility of our approach.

In our rebuttals, alongside our point-by-point responses, we present three new experiments.

Suggested improvements

Q1: A potential explanation for the LLM's success

We believe our method’s success is due to the LLM’s ability to translate information across domains. LLMs can effectively answer questions in medical domains and probability theory. They offer the ability to translate expert information in the form of natural language into probability distributions that represent prior beliefs on model parameters. We hypothesise that resampling the LLM multiple times mitigates the effects of hallucination for LLMs that are more likely to fabricate information. By combining many Gaussian components, we can approximate an LLM’s non-Gaussian prior. For our method to be successful, the LLM needs to have an understanding of the problem domain, Gaussian distributions, and linear models. We believe that this is why we see improved performance from chain-of-thought models, which can reason about the impact of each feature on an outcome. We will include this discussion in the final version of the work.

Q2: Further comparisons to a baseline

Zhu & Griffiths (2024) is not directly comparable to our work, as they do not provide a framework for eliciting the parameters of a predictive model. The most comparable work, Gouk & Gao (2024) (cited), generates synthetic data using an LLM to include in the training of linear classification models. To compare with our method, we extend their ideas to regression tasks and present new experimental results.

These are presented in our response to Reviewer 1 (xAxP) in section Q2, along with a detailed description of the baseline. In particular, Gouk & Gao’s method is susceptible to LLM data memorisation, putting it at risk of artificially providing improved posterior results on public datasets. To better compare the methods, we have split the results by the private and public tasks. We find that our approach yields improved posterior accuracy and mean squared error over Gouk & Gao’s method.

Q3: Add ethical approval statement

We have already obtained the appropriate ethics for our study but wanted to meet the double-blind requirement by not providing exact information on the ethics approval, including the identification number for our study. We include this information in the final version, alongside a description of the recruitment process. In the ethics statement, we have redacted the appropriate information to prevent breaking the double-blind requirements:

This study was performed in collaboration with INSTITUTION. Participants were recruited from the following: (1) health and social care partners within the INSTITUTION, (2) urgent and acute care services within the INSTITUTION, (3) social services who oversee sheltered and extra care sheltered housing schemes, (4) INSTITUTION and (5) specialist memory services at INSTITUTION. All participants provided written informed consent. Capacity to consent was assessed according to Good Clinical Practice, as detailed in the FRAMEWORK and the ACT. Participants were provided with a Participant Information Sheet (PIS) that includes information on how the study used their personal data collected in accordance with the ACT requirements. If the participant was deemed to lack capacity, a personal or professional consultee was sought to provide written consent to the study. Additionally, the capacity of both the participant and study partner is assessed at each research visit. Research staff conducting the assessment have completed the COURSE training and COURSE training. If a participant is deemed to lack capacity but is willing to take part in the research, a personal consultee is sought in the first instance to sign a declaration of consent. If no personal consultee can be found, a professional consultee, such as a key worker, is sought. This process is included in the study protocol, and ethical panel approval is obtained.

We feel the use of privately collected data from a real-world healthcare problem provides substantial evidence for the effectiveness of our framework, and we are indebted to our study participants.

Thank you for taking the time to review our work. We will implement the suggested improvements to the final manuscript. If we have answered your questions, then we would appreciate you considering raising your score. If anything is still unclear, we are happy to discuss further.

We would like to thank the reviewers for their feedback and time. We are happy that the work is considered well-written, with strong experiments, clear and informative figures, and a detailed appendix. We are pleased that the evaluation on a private dataset is appreciated and provides strong motivation for the work and that the inclusion of expert information is interesting.

In our rebuttals, alongside our point-by-point responses, we present three new experiments.

Suggested improvements

Q1: Discussion of related works in LLMs from a Bayesian perspective

As you note, numerous works discuss in-context learning from a Bayesian perspective. Our work is unique in that we empirically estimate the internal prior and posterior distributions of an in-context learner. With this, we use MCMC methods to compare the in-context posterior with a separately calculated posterior. This allows us to question whether an LLM is performing Bayesian inference on linear modelling of real-world numerical tasks. This is more relevant to our work than the previous natural language or theoretical perspectives. This contribution also provides a complete decision criterion to determine whether in-context learning or AutoElicit is more performant for a task.

We agree that this is an important area of the literature. We were restricted by the page length requirements of the submission and will include a complete description in the main section of the final paper with your suggestions.

Q2: Discussion on the interpretability of the priors

We use 100 mixture components in our prior to mitigate the effects of LLM hallucination, which is important in healthcare tasks. The prior distribution can be understood by visualising histograms of the parameter samples from the final mixture (linked below in a modified Figure 15):

FIGURE: https://imgur.com/a/gRRVmUz

As the density of these distributions are used for the prior directly, this provides a complete picture. For linear models, each feature's prior corresponds to its direction and importance when predicting the target. On the synthetic task (row 1), the prior distributions concentrate on the parameters that we expect. For the UTI task (row 2), the LLM provided prior distributions that suggest Feature 2 was negatively predictive of a positive UTI, whilst Feature 10 is a strong positive predictor. For Breast Cancer (row 6), the LLM produced two modes in its Feature 2 prior. Here, the LLM was either unsure of its belief, and so the elicited prior changed between task descriptions, or the LLM was unsure of the feature’s direction of correlation but guessed that the feature was important.

A chain-of-thought LLM like Deepseek-R1 could help us to better understand the elicitation process. We also found that using LLMs for predictions directly (in-context learning) underperforms in comparison to our approach.

Q3: Discussion on the number of mixture components

We investigate, in Rebbutal 2 for Reviewer 2 (3Pnd), in section Q3, how the posterior performance depends on the number of prior components. Here, we significantly reduce the number of mixture components whilst retaining the posterior accuracy and mean squared error.

Q4: New experiment evaluating an uninformative mixture of random Gaussians

Based on your suggestion, we compare our results to a new uninformative baseline. We sample 100 values from a standard normal distribution ($\mu \sim N(\mu | 0,1)$), and we use these as means of 100 Gaussian distributions ($\theta \sim N(\theta | \mu, 1)$). With these distributions we construct a mixture prior. As we keep the standard deviation at 1 for each component, the resulting prior has a greater range in the parameter space. The figure linked below shows the results:

FIGURE: https://imgur.com/a/STNjyEG

This new uninformative prior leads to no noticeable difference in the results for the synthetic task, Diabetes task, and Breast Cancer task. However, on the UTI task, we see marginally better accuracy than the previous uninformative prior, and on the Heart Disease and Hypothyroid tasks, we see greater improvements. Despite this, DeepSeek-R1-32B-Q4 or GPT-3.5-turbo continue to provide more informative priors. Therefore, the posterior improvement is not due to the complexity of the prior.

Q5: Clarity on the predictive performance

Predictive performance refers to either the posterior accuracy or mean squared error. To calculate this, we sample 5 chains of 5000 posterior parameter samples using MCMC methods. We use these sampled posterior parameters to predict the labels on our test set and calculate the metrics. This provides a distribution of performance.

Thank you for taking the time to review our work. We will implement the suggested improvements to the final manuscript. If we have answered your questions, then we would appreciate you considering raising your score. If anything is still unclear, we are happy to discuss further.

We appreciate the reviewer's kind words and time. We are pleased that our work was considered well-written with thorough experiments and appendices. We are happy that the extensive literature review, discussion of elicitation costs, and LLM memorisation experiments were appreciated.

In our rebuttals, alongside our point-by-point responses, we present three new experiments.

Suggested improvements

Q1: Further comparisons to a baseline

Zhu & Griffiths (2024) is not directly comparable to our work, as they do not provide a framework for eliciting the parameters of a predictive model. The most comparable work, Gouk & Gao (2024) (cited), generates synthetic data using an LLM to include in the training of linear classification models. To compare with our method, we extend their ideas to regression tasks and present new experimental results.

These are presented in our response to Reviewer 1 (xAxP) in section Q2, along with a detailed description of the baseline. In particular, Gouk & Gao’s method is susceptible to LLM data memorisation, putting it at risk of artificially providing improved posterior results on public datasets. To better compare the methods, we have split the results by the private and public tasks. We find that our approach yields improved posterior accuracy and mean squared error over Gouk & Gao’s method.

Q2: Using more challenging expert information

This is an interesting point. A synthetic example of an LLM using completely new expert information is given in the last paragraph of Appendix A.11 and Figure 13 (linked below).

FIGURE: https://imgur.com/a/QSZl2ZO

This demonstrates that giving increasingly more detailed information about the task (provided as natural language, Appendix A.11) allowed the LLM to update the elicited distributions correctly. We will lengthen the discussion in Section 5.3 and Appendix A.11 of the final paper to include this point.

Q3: New experiment on the variation of the task descriptions and number of mixture components

We will include examples of rephrased task descriptions in the appendix. We used a temperature of 0.1 to elicit the prior distributions so that the LLM reliably generated parameters of a Gaussian distribution and to mitigate the effects of hallucination, as we focus on healthcare tasks. When rephrasing the task descriptions, we used a higher temperature of 1.0 to get diverse descriptions. This is why we see variations in the priors elicited. An example rephrased system role is:

You are a simulator of a logistic regression predictive model for … Here the inputs are values from sensors around a home and the output is the probability of the presence of a urinary tract infection. Specifically, the targets are …

To:

You function as a logistic regression prediction model for … Inputs include sensor readings from a home, and the output is the probability of a urinary tract infection. The targets are …

The meaning of the description remains the same, but the rephrasing mitigates against changes in the priors elicited based only on the language used in the prompt.

To supplement this, we calculate the posterior performance with varied numbers of mixture components in our prior (linked below):

FIGURE: https://imgur.com/a/E3hE9Na

We see little variation (except for Breast Cancer with Deepseek) in the accuracy or mean squared error as we increase the number of components in the prior. This suggests our framework is robust to the number of task descriptions. We hypothesise that a greater number of mixture components makes the framework more resilient to LLMs that frequently hallucinate, especially on tasks with large numbers of features (such as Breast Cancer).

In Appendix A.12, we showed that the density of the priors changed with the number of mixture components, however, this has small effects on the posterior performance. This is likely because the sample space that provides good posterior performance is covered by the majority of elicited components, whilst the large changes in Appendix A.12 are due to more unique components.

Q4: Discussion on the interpretability of priors

In our response to Reviewer 3 (fJAR) in section Q2, we provide examples of interpreting the priors elicited from GPT-3.5-turbo. As suggested, it would be interesting to explore how these priors are understood by clinicians and if they are used separately from the predictive modelling. The points you have raised are valuable. This is future work we are currently organising with clinicians.

Related works

Thank you for the helpful suggestions. We will include these references with discussions in the final version.

Thank you for taking the time to review our work. We will implement the suggested improvements to the final manuscript. If we have answered your questions, then we would appreciate you considering raising your score. If anything is still unclear, we are happy to discuss further.

We would like to thank the reviewer for their comments and constructive feedback to improve our work. We are happy the work was found to be well-written, easy to follow, and well motivated. We are pleased that the use of strong experiments, evaluation of LLM memorisation, and use of our privately collected clinical dataset were appreciated.

In our rebuttals, alongside our point-by-point responses, we present three new experiments.

Suggested improvements

Q1: Methodological comparison to Selby et al. (2024)

Thank you for your question. We list key differences between our work and Selby et al. (2024) (cited), highlighting where the novelty of our approach lies. We will improve our description of this work in the final version of the manuscript:

• Selby et al.’s core focus is on data imputation; however, they show preliminary results for eliciting distributions and compare these to human experts.
• They use these in a single predictive task. However, their method elicits a prior over the targets directly, much like with in-context learning, and not over the parameters of a model as in our work. This means that Selby et al. do not update a model using available data and are therefore not training any predictive models using the priors. This also makes Selby et al.’s work incompatible as a baseline.
• Selby et al. do not perform repeated sampling of the LLM, which we hypothesise mitigates hallucination.
• Selby et al. do not combine experts and LLMs, an important aspect of human-AI interaction. We achieve this by allowing experts to provide natural language information in task descriptions, simplifying current prior elicitation methods. In this case, our elicited distributions contain knowledge from both the experts and the LLM’s training.
• We also present a grounded model selection criterion to decide between AutoElicit and in-context learning.

Q2: New experiment with Gouk & Gao (2024) as a baseline

A more compatible baseline presented by Gouk & Gao (2024) (cited) generates synthetic data using an LLM to provide an additional likelihood term when training linear classification models. This method uses an LLM to generate feature values before labelling them with zero-shot in-context learning. These generated samples are provided alongside real data during the training of a linear predictive model. Therefore, Gouk & Gao’s method is susceptible to LLM data memorisation, putting it at risk of artificially providing improved posterior results on public datasets.

We compare our results to this baseline and extend their ideas to regression tasks to compare with all the datasets we test. This previous work was described in our manuscript but we now provide new experimental results. The figure (linked below) will be included in the final manuscript:

FIGURE: https://imgur.com/a/3sM2baQ

This figure is split by publicly available tasks (four right-most plots) and privately available tasks (two left-most plots). In these experiments, our approach provides significantly greater posterior accuracy and lower posterior mean squared error for both private datasets.

In the public tasks, we see that the performance of Gouk & Gao’s approach varies considerably by LLM and task. It underperforms compared to the uninformative prior on five occasions. On the two occasions it outperforms AutoElicit (ours), it achieves surprisingly good accuracy or mean squared error (Heart Disease with GPT-4-turbo and Diabetes with GPT-3.5-turbo), suggesting that it might be reproducing parts of the public datasets rather than generating new samples. Given our analysis in Appendix A.8 showing the memorisation ability of LLMs, and Gouk and Gao’s performance on private data, we suspect the performance of this approach is artificially improved by reproducing the true dataset. Since for AutoElicit, we prompt the language model for a prior distribution over the parameters of a predictive model, using only the feature names and a description of the dataset, the memorisation of data points is unlikely to have had an impact on our method's results (discussed further in Appendix A.8.1).

In both of the private datasets (two left-most plots), where there is no risk of LLM memorisation, AutoElicit results in improved posterior accuracy and mean squared error. On the synthetic task, our improvement is orders of magnitudes better, and on the UTI task, our method significantly increases accuracy at all training sizes. In both cases, the approach proposed by Gouk & Gao underperforms compared to an uninformative prior.

We will include these new results in the final version of the paper, as it provides further context to our approach.

Thank you for taking the time to review our work. We will implement the suggested improvements to the final manuscript. If we have answered your questions, then we would appreciate you considering raising your score. If anything is still unclear, we are happy to discuss further.