LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language

Thanks for taking the time to review our paper and for the positive words about our contributions. We address your questions below.

Figure 9 is intended to demonstrate how predictions from an LLMP can be influenced by conditioning on various scenarios communicated via text prompts to the model. This experiment is a qualitative investigation with no specific ground truth information. However, we hope to show that a prompt containing two synthetic initial data points plus some descriptive text will yield samples that are meaningful.

We provide units for the axes in the prompts (detailed in Appendix J.1) as follows:

• Figure 9b) “The following are daily temperature measurements from Montreal in January in degrees Celsius”
• Figure 9c) “The following are daily temperature measurements from Montreal in May in degrees Celsius”
• Figure 9d) “In the following series, the first number is the number of Months from January and the second is the Monthly precipitation measurements in inches from San Diego, CA”
• Figure 9e) “In the following series, the first number is the number of Months from February and the second is the Monthly precipitation measurements in inches from Singapore”

We will add these units to the plots in the next revision.

You are correct that the median response in Figure 9d) does not match the ground truth in Figure 9f) particularly well. They match in the sense that both figures indicate that the precipitation is quite low all year round in Singapore, but it does not match the minor rising and falling trend. We think that this can be attributed to the fact that since the synthetic datapoints also do not match actual values very well, the LLMP had difficulty reconciling this mismatch between values and text information. We included this experiment primarily to demonstrate that the model is aware of this seasonal trend in San Diego vs. Singapore (Singapore has one of the least seasonal trends in precipitation on earth) and that can reasonably modify the LLMPs predictive distribution on the same collection of data values by varying the prompts.

Different from Figure 9, Figure 10 is a quantitative investigation on how incorporating textual information can improve predictive performance. In this experiment, we measure how correct this change in predictive distribution is by examining improvements in the Negative Log-likelihood (NLL) and the Mean Absolute Error (MAE) given by the predictive distribution when text is included in the prompt versus when it is excluded. Figure10 shows that both these metrics improve (i.e. are lower) when conditioning on text.

Please let us know if you have any more comments or questions. If we have adequately addressed your concerns, we kindly ask that you consider revising your score.

Thanks for taking the time to review our paper and for the positive words about our contributions. We address your questions below.

The main weakness in the approach that I potentially see is the runtime. From what I could tell, there is no runtime results in either the main paper or the appendix. For processing a sequence of input-output pairs, what is the general runtime exhibited, and how does it change as the sequence grows?

We agree that this is the main weakness our method. We view the main use case of LLMPs as being when the benefits of incorporating textual information into your regression problem outweigh the significant computational expenses involved. Appendix G “Additional Implementation Details” does contain some example runtimes. It states that processing times vary as a function of the:

• GPU used.
• length of the prompt.
• number of target points queried.
• number of tokens required to be generated for a particular target point.
• the number of samples taken at each target point
• and whether independent or autoregressive sampling is used.

However, Appendix G does not specifically address your questions, so below is some new data that hopefully will give you a more tangible understanding of runtime performance. Note that this new data is quantitative as oppose to general "Big O notation".

The table below shows the time to load the LLM into GPU memory, the time for the LLM to generate all samples at all target points, and the time to compute the probability distribution over the true target points. All runs used the Llama-2-7B LLM and were executed on an NVIDIA 3090 GPU with 24GB of memory with a batch size of 10. All times are in seconds.

From the table, we can see that the longer the prompt, the longer the computation time for each target point. For independent sampling (I-LLMP), the prompt length is constant and is only a function of the number of training points as each target point is processed independently. For autoregressive sampling (A-LLMP), the prompt length is a function of both the number of training points and the number of target points since each target point is appended to the prompt as it is sampled.

We will add these details in the next revision.

Thank you for the response to my question, this does indeed directly communicate the type of information I wanted to see regarding this issue.

As far as the paper, I do still feel strongly that this is innovative work with a great deal of experimental results backing the empirical findings. As such, I maintain my original score.

Thanks for taking the time to review our paper and posing insightful questions. Answers to your questions are below:

If I correctly understand the experimetnal part reports resutls on both training and only prompting. In this case make these two paradigms more clear in the organization of the experimental results.

Apologies if the organization of the experimental results section was not clear. We decided to organize our experiments into two main sections: the first section examines how LLM Processes (LLMPs) perform on purely numerical tasks, while the second section examines the influence of conditioning LLMPs on text and similar examples in-context. Note that for our proposed methods (I-LLMP and A-LLMP), no training is ever involved. These methods use prompting only. Only the Gaussian Process hyperparameters are trained to compare with LLMPs. We will clarify this point at the start of the experimental sections in the next revision of the paper.

The paper that is cited [21] : their experiments should be comparable to this work, why the results are not compared? while you apply more technical variations here what is the main message and your new contribution compared to them? can you put their results and yours side by side and compare them?

The primary reason we did not compare to the results in the “From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples” paper is that it was released on arXiv on April 11, 2024 only a little more than a month before the NeurIPS submission deadline of May 22, 2024. By the time we became aware of the paper, we had finished running our experiments and had started writing up the paper.

However, below are the results of a new experiment where we compare our results to theirs on the Original #1 dataset (see section 4.2 in the “From Words to Numbers” paper). The experimental set-up is as follows: There are 100 trials with each trial consisting of 50 training points and a single target point. The training and target points for each trial are randomly generated using the function described in Equation 2 in the “From Words to Numbers” paper. We use the code from their paper to generate the data and evaluate their approach and compare it to ours using identical numerical data. We use the Llama-2-7B LLM for both methods to ensure a fair comparison.

Our method (I-LLMP) achieved lower Mean Absolute Error (MAE) on 78 of the 100 trials when compared to their method. When the errors are averaged over the 100 trials, our average error was 0.836 and theirs was 3.137. These results indicate that our LLMP approach is clearly superior to the approach employed in the “From Words to Numbers” paper. This is due to the facts that a) we sort the training points according to distance to the current target point when creating the prompt whereas they do not, and b) we form a distributional estimate for the predicted point and then take the median sample value as the best estimate, whereas they generate a single point estimate. We will include this comparison in the updated version of our paper.

In our paper, we also compare to the LLMTime method in Figure 5 and Gaussian Processes in Table 1 and Figure 7. These results show that we significantly outperform LLMTime, especially in terms of negative log likelihood (NLL) and are better overall when compared to Gaussian Processes on a wide variety of functions.

While we also show that LLMs are capable of non-linear regression, our contributions go significantly further than both the “From Words to Numbers” paper and LLMTime:

• Our primary contribution is to condition on problem relevant text and demonstrate how this can improve prediction performance. Their work does not consider this.
• Our work presents how to elicit full numerical predictive distributions from LLMs using two different methods (sampling-based and logit-based) while their method employs only point estimates for predictions.
• We perform a comprehensive analysis of various prompt formats including ordering and scaling. They do no such experimentation and use only a single fixed prompt format.
• We present a novel auto-regressive LLM sampling approach that yields superior results to the independent sampling approach used in their paper.
• We compare our results to Gaussian Processes, viewed as the gold standard for probabilistic regression, while they compare to only simpler approaches.

Please let us know if you have any more comments or questions. If we have adequately addressed your concerns, we kindly ask that you consider revising your score.