DeLLMa: Decision Making Under Uncertainty with Large Language Models

Thank you so much for your positive feedback. We are delighted that you found DeLLMa to be original, clearly presented, and achieve competitive performance. We hope to address your questions and concerns below.

Context Window Limits

We agree that the context window may indeed pose some limits on real world problems. Nonetheless, most leading LLMs are endowed with a context window of 128K tokens, and our most context-intensive instances only require <21K tokens as contexts (shown in Table 5). We believe that this window, together with future advances in long-context modeling, can cover a broad range of decision making problems.

Sensitivity to State Forecasts

We did observe some sensitivity to the state forecasting step. But as we show in Table 2 of our paper—which shows DeLLMa performance given different state forecasting variations—DeLLMa seems to be quite robust to this step as long as the backbone LLM can perform the utility elicitation step well.

Simplicity of Pairwise Ranking

We have indeed tried a number of variations for utility elicitation! For human pairwise comparisons, we tried variants such as direct rankings and rating scales (from a range of 1 to 5). For utility fitting, we also tried different utility fitting methods provided in the choix package [1], such as the iterative variant of the Luce-Spectral Ranking method. We found that the methods described in section 3.3 worked the best for the types of problems we have in the experiments. However, as we expand the problem space in the future, we will be sure to evaluate other methods.

Reviewer Questions

How can DeLLMa be adapted to handle continuous state and action spaces?

This is a great question, which we are exploring:

• [Continuous State Space]: There are multiple possibilities for this—one direction we are exploring is tool usage (where we use probabilistic tools to yield a continuous state forecast), but we can also directly prompt an LLM to forecast a distribution over the continuous state variables.

• [Continuous Action Space]: For a continuous action space, we just need to adapt/update our expected utility maximization step to a continuous space. We can offload this to a suitable continuous optimization algorithm or program. For example, we can perform adaptive gradient steps over the action space when optimizing continuous actions. (c.f. Section 4.7.2 of [2])

The state forecasting method assumes independence bw latent factors. How does this impact total performance?

It’s a good point that the independence assumption will affect performance. While we show that this assumption works reasonably in practice in our current implementation (e.g., Table 1), we haven’t yet implemented a more general probabilistic model to assess the improvement—but it’s an important direction which we will do in future work.

Is DeLLMa able to handle cases where the user task cannot be distilled into a hard utility function?

In our current implementation, we assume that enough information to specify a utility function is given by the user prompt. But in the future we will definitely aim to address cases where the utility is only partially defined (or where there is ambiguity). Some prior works (e.g. [3]) have shown that preference elicitation from limited information is indeed possible.

[1] The choix package, https://choix.lum.li/en/latest/api.html#process-pairwise

[2] Decision Making Under Uncertainty, https://web.stanford.edu/group/sisl/public/dmu.pdf

[3] Eliciting Human Preferences with Language Models, https://arxiv.org/abs/2310.11589

Thank you for your feedback! We are glad that you found our formalism helpful in understanding our framework, and our results reproducible. We hope to address your questions and concerns below.

Fixed Decision Space

This is a great point! Indeed, many real-world decisions do not attain a predefined set of actions, and this is an important future direction that we aim to tackle. Nonetheless, there are many important problems that are endowed with a fixed action set. For example, in healthcare, physicians often make decisions from a finite set of treatment options based on a patient’s symptoms and diagnostic result [1]. Similarly, in supply chain management, decision-makers regularly work within a bounded set of distribution strategies and inventory policies, such as selecting from fixed shipping options or standard order quantities [2]. With a focus on this setting, DeLLMa is our first step towards a more comprehensive system for real-life decision making.

Limited Scope of DeLLMa Applications

Thank you for raising this concern! It is certainly not our intention to phrase DeLLMa as a general solution to all instances of decision making under uncertainty. In fact, in Appendix H, we emphasize that DeLLMa is only a first step towards this direction, and requires additional guardrails and human-AI collaboration before becoming a production-ready system. In this section in our paper, we have added some additional discussion which reads as follows:

Furthermore, our current experiments focus on specific domains (agriculture and stocks) as representative, controlled environments for testing decision-making under uncertainty. Future work will involve evaluating DeLLMa across a broader set of domains to better understand its strengths and limitations in diverse, real-world applications.

Missing Details on Human Annotation Study

We appreciate your criticism and apologize for any confusion this may have caused! In our global response, we have provided the annotation guidelines and updated annotation results. In short, we have solicited a total of 412 annotations, 200 of which are annotated by 5 external volunteers. From these, we obtained an annotator-LLM agreement rate of 66.75% and an inter-annotator agreement rate of 67.0% ($\pm$ 6.34%). All annotation results and evaluation scripts have been updated to our supplementary material (under DeLLMa-additional-annotation).

Reviewer Questions

How will DeLLMa perform on other tasks related to chain-of-thought or tree-of-thought methods discussed in related work?

Thank you for this question. DeLLMa is most well suited to decision-making under uncertainty tasks, which is a subset of reasoning tasks. While CoT / ToT are more-broadly applicable to a wider range of tasks, we've found that DeLLMa yields higher performance on these decision making tasks.

If scalability is not a significant issue in certain cases, will DeLLMa outperform other methods?

Yes, we observe in Figure 3 that DeLLMa does improve with scale, and its scaling behavior compares favorably with OpenAI o1 (shown in Table 3), a SotA method for scaling compute at inference-time aimed at reasoning tasks.

In the experiments, one task involves investing on December 1, 2023, and selling on the last trading day of that month (December 29, 2023) to maximize returns. Why were these specific dates chosen? Why not other dates?

Our initial experiments were conducted on the GPT-4-1106 checkpoint (i.e., a model released in November of 2023), and we use financial market data from December 2023 for evaluation since they are strictly out of the knowledge cutoff of the model.

[1] MIMIC-IV-Ext Clinical Decision Making: A MIMIC-IV Derived Dataset for Evaluation of Large Language Models on the Task of Clinical Decision Making for Abdominal Pathologies, https://physionet.org/content/mimic-iv-ext-cdm/1.1/

[2] Large Language Models for Supply Chain Optimization, https://arxiv.org/abs/2307.03875

Improving Stock Result Analysis

We appreciate your detailed feedback! Our intention was indeed to provide an initial hypothesis for this phenomenon rather than a definitive explanation, and we agree that the better performance of DeLLMa-Top1 may stem from our dataset or model processing constraints.

(1) some difficulties might be due to assessing particular actions (are these scores evaluated permuting through all the alternatives? (e.g., which 2 actions matters)?)

Yes, these scores are evaluated through all possible permutations of the alternatives.

(2) averaging across the accuracy score of different sizes matters. There I would go for higher weight to the higher number of actions.

Thanks for this great suggestion! In our current version, we opted for unweighted average accuracy as our performance metric due to its simplicity, but we agree that assigning higher weights to more complex tasks is a sensible approach. We also recognize the importance of evaluating decision-making capabilities across different levels of task complexity. To this end, we are developing a follow-up benchmark that will incorporate stratified analysis to better assess performance across various complexities.

Based on your feedback, we have revised our phrasing in this paragraph, which reads as follows:

Additionally, DeLLMa-Top1 performs better than DeLLMa-Pairs in the stocks data. This difference may stem from the simplicity of utility elicitation from only the top action choice, which requires less data processing and may mitigate noise compared to enumerating all state-action pairs. We hypothesize that in high-volatility data like stocks, LLMs may struggle with pairwise comparisons due to potential hallucination issues, particularly when the model attempts to rank options without a clear ground truth. By focusing on the top choice, DeLLMa-Top1 could avoid accumulating noise from these internal rankings, achieving better performance.

DeLLMa vs o1

Apologies for the confusion. In Table 3, we are comparing our largest DeLLMa variant, with a per action sample size of 64 and an overlap percentage of 25% against OpenAI o1. While smaller DeLLMa instantiations can outperform o1 (shown in Figure 3), we used our large variants as they attain similar costs (Line 519).

Writing Organization

Thank you for your thoughtful feedback on our paper structure! Due to page limitations, we had to reference the appendix for certain details, which may have affected readability. We’ll do our best to address this. Specifically, we’ll explore ways to consolidate key explanations within the main text and adjust figure placements to improve clarity.

[1] Approaching Human-Level Forecasting with Language Models, https://arxiv.org/abs/2402.18563

[2] MIMIC-IV-Ext Clinical Decision Making: A MIMIC-IV Derived Dataset for Evaluation of Large Language Models on the Task of Clinical Decision Making for Abdominal Pathologies, https://physionet.org/content/mimic-iv-ext-cdm/1.1/

[3] Large Language Models for Supply Chain Optimization, https://arxiv.org/abs/2307.03875

Thank you for your careful review and constructive comments! We are glad that you found DeLLMa to be interesting while grounded in classical decision theory. We hope to address your questions and concerns below.

Missing Details on Human Annotation Study

We appreciate your criticism and apologize for any confusion this may have caused! In our global response, we have provided the annotation guidelines and updated annotation results. In short, we have solicited a total of 412 annotations, 200 of which are annotated by 5 external volunteers. From these, we obtained an annotator-LLM agreement rate of 66.75% and an inter-annotator agreement rate of 67.0% ($\pm$ 6.34%). All annotation results and evaluation scripts have been updated to our supplementary material (under DeLLMa-additional-annotation).

Strong Assumption on K Independent Factors

We agree that this is indeed a strong assumption. In our initial experiments, we opted for this assumption due to its simplicity and its reasonable end-to-end performance in practice. But future works can indeed consider a more meaningful probability factorization (e.g., a probabilistic graphical model). Our additional calibration results indicate reasonable forecasting performance, which we discuss next.

Additional Analysis of Calibration Performance

Thank you for pointing this out. We provide additional details and reference values for our evaluation of the forecasting distribution (i.e., Section 4.1, Table 1).

In Table 1, we aim to show the calibration performance of our LLM-based forecasting method. To contextualize the quantitative statistics reported in this table, we additionally report two sets of ECE statistics:

• ECE under the uniform distribution forecast (ECE-Uniform).
• ECE under the LLM forecast, but with random choice as the ground truth (ECE-Random). We report the standard deviation from 100 random trials.

This yields the table below. Together, these results indicate that our state forecasting algorithm attains much improved performance in comparison with these baselines. We will upload code to reproduce these results as additional supplementary materials.

Note that, to calculate the ECE, for a fixed set of latent factors (e.g., fruit price change, climate conditions), we manually find and annotate ground truth values — using the USDA report and web search. We can then compute metrics such as the expected calibration error (ECE) for forecasts of these latent factors given by a model.

Fixed Number of Alternatives

This is a great point—indeed, many real-world decisions do not attain a predefined set of alternatives, and this is an important future direction that we aim to tackle. Nonetheless, there are many important problems that are endowed with a fixed action set. For example, in healthcare, physicians often make decisions from a finite set of treatment options based on a patient’s symptoms and diagnostic result [2]. Similarly, in supply chain management, decision-makers regularly work within a bounded set of distribution strategies and inventory policies, such as selecting from fixed shipping options or standard order quantities [3]. With a focus on this setting, DeLLMa is our first step towards a more comprehensive system for real-life decision making.