Thank you for the detailed feedback on our work. We are delighted that you found our approach novel and promising and our paper well-written.

More comprehensive details on how DSPy generates candidate values for module parameters (instructions, field description, and example input/output) based on signatures.

Thank you for the suggestion. We now include a new appendix I (pages 28, 29, 30) which include snippets of our optimized prompts, when compiling with Llama2-13b-chat. This should help the reader concretely visualize the parts of a typical prompt constructed from a signature. In particular, it contains the instructions (which, if omitted, are inferred from the fields), the field organization, and the demonstrations.

In short, the module defines a parameter space. A teleprompter can then optimize these different pieces. For instance, we have an ongoing research project that optimizes the instructions of DSPy modules automatically. In this paper, we focus on optimization with random search over demonstrations, where we automatically created traces of each module, and then select different traces as few-shot examples (or finetuning training data), even if there are no hand-written labels for these modules.

Can we easily implement common and advanced prompting techniques with signatures? When it comes to dealing with complex user intentions, do you think additional hand-written comments are necessary alongside signatures?

A prompting technique is best abstracted as a Module in DSPy. It is indeed possible that a specific approach to prompting requires adding new modules to DSPy, which should work for any signature (i.e., any input/output patterns).

Our vision is that any minor hard-coded parts of the modules (e.g., “Let’s think step by step.”) should eventually be treated as variables to optimize. Nonetheless, like in traditional DNNs, there are always hyperparameters that may be selected by hand, as long as the overall model at the end is optimized automatically.

Since the submission of this work, we have built (or helped others build) a number of new sophisticated systems with DSPy. In some of them, we found that users like to include additional comments (or instructions) in the extended DSPy format that we present in Appendix B “Advanced Signatures”. To make this more robust, we have been exploring techniques to automatically optimize these prompts and to allow for automatic backtracking logic to correct errors. Considering the scope, length, and complexity of this paper, we leave exploring these additional components to future publications related to DSPy.

How can users debug their DSPy programs effectively? [...] it raises the question of how users can determine which modules are not functioning correctly. Is it necessary for users to repeatedly rewrite the entire DSPy program?

One of the key reasons we built DSPy as a “define-by-run” abstraction (like PyTorch, or the “eager mode” of TensorFlow 2.0) is to facilitate debugging.

In particular, users of DSPy can simply insert checkpoints or use print statements between different steps of their DSPy programs. For example, in the multi-hop program, we can print or log every search query generated (or even manually change these queries) in order to understand which module is responsible for different mistakes.

Because it is possible to debug one module at a time, we do not anticipate any need for users to repeatedly rewrite their programs. Instead, they can focus on isolating individual types of errors and then improving the responsible module. Of course, DSPy is not a panacea and sometimes development or debugging may require more end-to-end tracing of the full program. To this end, we recently started adding support for tracing of the full program to visualize all the prompts during execution.

[continued below]

[continue...]

How much does the optimization cost in terms of computing resources or API usage?

Thank you for emphasizing this element. To address this, we have added a new appendix A on Latency & Throughput of using DSPy. To illustrate that the cost of using DSPy is manageable, we consider a sophisticated program from our evaluation, i.e. the multi-hop search and QA program. We report on compiling it with OpenAI GPT-3.5 for $3.00 in just six minutes. Inference with this program takes approximately 3 seconds per example (single-threaded) and can easily exceed 150 questions per minute (with ten DSPy threads).

Future work should explore efficiency in a much broader evaluation to characterize this emerging space of LM programs. Just like the cost of training a new model, the cost of compiling a DSPy can vary greatly depending on the complexity of the program and the optimizer. However, we hope this small-scale evaluation sheds some light on the typical cost of compiling a program like this.

In section 5, Hypothesis 2, the paper mentions the possibility of DSPy outperforming expert-written prompts. However, is there an evaluation comparing DSPy with handwritten prompts to support this claim?

This is indeed an important aspect of DSPy. In our evaluations, the CoT program for GSM8K includes comparison between the “bootstrap” compilation approach and the “fewshot +human_CoT” setting (Table 1). The former generates its own optimized few-shot examples (including, in this program, the reasoning chains). The latter samples expert-written step-by-step reasoning chains, which are the typical way other papers build prompts for GSM8K (e.g., the original CoT paper by Wei et al). Another instance of this is the ReAct evaluations in Table 2, which compare handwritten prompting against our bootstrapping compilation approaches. Across these evaluations, we found that the compiled systems match or exceed the quality achieved by hand-written prompts and reasoning chains. These results can also be compared with systems from the literature (e.g., the original ReAct scores on HotPotQA), which generally use hand-written prompts (e.g., see Appendix E).

In Section 5 of the paper, it states, "... ‘how they compare on GSM8K with program P when compiled with strategy S’, which is a well-defined and reproducible run.” How can the reproducibility of these results be ensured?

To reproduce a specific program, DSPy supports saving a dump of the compiled program. This is akin to saving a trained model, to draw an analogy with deep neural networks. As long as the LM can produce reproducible outputs, loading the saved program can be used to replicate old runs, if the LLMs are set for greedy decoding (i.e., temperature=0.0). DSPy also supports caching (and sharing the cache for) all invocations of the LM(s), which is another way to ensure reproducibility as needed.

More generally, it's also possible to re-run each setup several times (with higher temperature) and average the runs. Either way, like DNNs, DSPy offers a more self-contained setup than hand-drafting prompts (or hand-tuning classifier weights), which aids reproducibility of the key ideas in the the more general sense that the optimization process itself is automated and hence can be reapplied to new settings. In contrast, hand-tuning prompts is more of an art than a science and is hard to recreate.

In Case Study 1 using the GSM8K dataset, GPT-3.5 with 5 few shots achieves a score of 57%, while vanilla+GPT-3.5+fewshot only scores 33.1%. What factors contribute to the decline in performance, considering the similarity between these two approaches?

Thank you for the remark. Let us clarify how the different approaches for few-shot prompting work on GSM8K.

In Table 1, we report the following results with gpt-3.5-turbo-instruct (released in September 2023).

1. Using the fewshot (LabeledFewShot) teleprompter in DSPy, on the vanilla program.
2. Using the fewshot teleprompter on the CoT program.
3. Using the fewshot teleprompter on the CoT program, with the extended “+human CoT” training set.

In the first two, we prompt the LM with few-shot examples that do not contain reasoning chains. That is, the demonstrations in #1 and #2 only include the final numerical answer. In the first one, we also do not request the model to engage in a step-by-step reasoning chain, unlike #2 and #3. In the third one, the few-shot examples include human-written reasoning chains.

These three approaches score 33%, 65%, and 78% on our development set. We test the last one on the official test set, and it scores 72%. In this case, the most comparable run to the original 5-shot GPT-3.5 evaluation by OpenAI is the third option. As you can see, our implementation here scores more highly (72% > 57%), which may be a result of using a newer release of GPT-3.5 (September vs March 2023), using extra demonstrations (5-shot vs 8-shot), and variance across samples (we average 5 runs).

[continued...]

1. “In the abstract you say DSPy programs are "compiled to" a language model.”
2. “If I understand correctly [teleprompters] are limited to gradient-free optimization techniques. This detail warrants mentioning.”
3. “Typo: GMS08K -> GSM8K (page 2, paragraph 4) Typo: format -> formats (page 4, Section 3.2) Typo: grammar: but -> or (page 5, Section 3.3) Typo: "As we assumes" in the example at the end of Section 3.3 Typo: these -> this, ensembles -> ensembling (page 6, stage 3) Typo: "specific math problems or particular LM." -> "specific to math problems or a particular LM." (page 7)”

Thank you for this extensive list of notes and typos. We believe we have now addressed all of these in the PDF. (We will update the submission abstract outside the PDF too, if the work is accepted. The system does not permit this change currently.)

Strong claim: The paper claims DSPy is the first programming model that translates prompting techniques into parameterized declarative modules that can be optimized. [...] Dohan 2022 is closest to the claim made in the paper.

We agree to soften our statement in the introduction, in light of this note. In short, Dohan et al. (whose work we cite as early as our first paragraph) present a rich self-described “position paper” on the conceptual analogy between what they call “LM cascades” and probabilistic graphical models. While we find that work highly valuable, we are unable to immediately extrapolate from this conceptual analogy to a programming model in which prompting techniques can be optimized directly, as we do with DSPy. Nonetheless, to avoid any unintended implications, we have softened our statement in the introduction (marked in blue in page 2) in line with your comment.

Do the compilation approaches account for overfitting the validation set? [...] How do you think about safety during SQL generation, since SQL queries can modify or delete tables. How do you approach using tools with side effects during compilation?

These are both good examples where using DSPy does not replace the value of good, thoughtful experimental design. In particular, care must be taken if the program has access to tools that can modify (or destroy) records or execute code, like running it in a sandbox. We have a small Python execution sandbox in DSPy to aid with this, but it is not an alternative for thoughtful design.

Similarly, as with standard ML techniques, overfitting is an important consideration, especially when the training and validation sets are small. In our experience, we found limited signs of detrimental overfitting (i.e., selecting low-quality systems) but found evidence of sets of different difficulty such that the absolute scores of the systems are different across validation or evaluation sets.

For good experimental design, the teleprompters allow us to distinguish between the training, validation, development, and test sets. For instance, when compiling the multi-hop program on HotPotQA, we use the first 50 examples of the training set for training, and examples 50 through 200 for validation. We compare different approaches on a separate development set (300 examples). And we finally report the quality on the test set (1000 separate examples). While the teleprompters enable such good experimental design, this cannot be enforced. It is always possible for misuse of the optimization process to overfit.

In the RAG example at the end of Section 3.2, is it complete without a definition of Retrieve provided? How can a DSPy user introduce a new Retrieve implementation?

At the start of a script that uses DSPy, the user declares the “default” language model and retrieval model. These choices can also be changed for specific code blocks. In our experiments, we set the default retriever to be a ColBERTv2 (Santhanam et al., 2022) index over a Wikipedia corpus. However, this is very easy to extend. Our users have introduced many integrations for retrieval, from Pyserini to Pinecone to Weaviate, among other popular systems.

Thank you for the thoughtful review and the in-depth feedback on our work. We are glad to see that you found our programming paradigm 'novel' and our 'focus on expressiveness and usability', as evidenced in the 'composability' of programs and optimizations, promising for the ICLR community.

My impression is that the value-add of DSPy for any of the listed strategies is somewhat small. E.g. implementing any of CoT, Reflection, or ReAct without DSPy does not require much code. [...] I expect this value-add grows when working with many such strategies at once, and additionally that DSPy provides organizational value both as the programs and compilers grow in complexity [...] Being able to compose different strategies and compiler techniques is also one of the key beneficial properties of the system.

Thank you for the nuanced discussion. We agree that much of the power of DSPy comes from enabling rich compositions of modules and compilation strategies. However, we wish to emphasize that even simple programs (e.g., those that rely on a single module, like CoT or ReAct) stand to benefit from DSPy. To illustrate, it’s indeed true that expressing CoT (or, say, ReAct) does not require much code without DSPy. However, expressing them well for a particular task generally requires writing a good prompt, often with high-quality few-shot examples for reasoning or tool use. Adapting this prompt across LMs may also pose a challenge. While these challenges for a single, simple module are not very large (and indeed more research is needed to make general claims about them), we believe the DSPy paradigm offers a consistent way to optimize both simple and complex programs alike.

The evaluations performed do not provide a measure of compute usage or time (both for compilation as well as inference of the compiled programs)

Thank you for emphasizing this element. To address this, we have added a new appendix A on Latency & Throughput of using DSPy. To illustrate that the cost of using DSPy is manageable, we consider a sophisticated program from our evaluation, i.e. the multi-hop search and QA program. We report on compiling it with OpenAI GPT-3.5 for $3.00 in just six minutes. Inference with this program takes approximately 3 seconds per example (single-threaded) and can easily exceed 150 questions per minute (with ten DSPy threads).

Future work should explore efficiency in a much broader evaluation to characterize this emerging space of LM programs. Just like the cost of training a new model, the cost of compiling a DSPy can vary greatly depending on the complexity of the program and the optimizer. However, we hope this small-scale evaluation sheds some light on the typical cost of compiling a program like this.

There are no examples of compiled programs provided in the paper or appendices. [...] how do they differ from the types of prompt programs that people write by hand or using other frameworks? how do programs compiled for small LMs differ from those compiled for large LMs? are there any obvious patterns in the compiled programs?

We now include a new appendix I (pages 28, 29, 30) which include snippets of our optimized prompts, when compiling with Llama2-13b-chat. We believe an extensive analysis of these prompts, especially when varying LMs and varying optimization strategies, is a highly exciting future direction.

Nonetheless, it may be useful to consider Figure 11 (page 30) which reports on the Llama2-13b prompt for the second hop of query generation for HotPotQA. We see that it teaches the model two different strategies for generating search queries. The first bootstrapped few-shot example is a well-formed question "When was the first of the vampire-themed fantasy romance novels published?". The second is a keyword-based search query, "Jeremy Paxman birth year". It is possible to hypothesize this duality contributes to the selection of this prompt during optimization. Future work may try to characterize these patterns on a principled basis.

'how about obvious shortcomings or irregularities, where additional hand-optimization would be easy?'

One of the limitations of the BoostrapFewshot set of teleprompters we discuss in this work is that they focus on generating and selecting demonstrations of the input/output behavior of each module. To expand this toward more open-ended optimization, we have started multiple research directions that explore optimizing the instructions and execution logic (e.g., backtracking on failure) for the modules.

[continue below...]

Thank you for the thoughtful comments on our response. We are glad that you found our additions to the text valuable. We also appreciate that you considered raising the score (but couldn’t, because the system only accepts 10/10 as the next higher score).

some measure of cost in order to make the comparisons across programs and compilers in Tables 1 and 2 more meaningful

We agree that readers will benefit from a comparison of the costs associated with different programs. In principle, measuring the compilation costs requires us to re-compile our programs (as we did for Appendix A, for one program) while disabling our caches. Considering that this is a significant experimental endeavor, if our work is accepted, we will include compiling and inference costs in the camera-ready version. This can potentially serve as a new column in the results tables or an expansion of Appendix A.

We hope that the addition of Appendix A conveys the observation that the cost of compiling (which is a one-time, offline step) is fairly low. (As a ballpark, compiling is generally faster than typical finetuning jobs for instance.) Because of this, we typically recommend that DSPy users focus primarily on the quality and cost of the resulting programs, since compiling is an offline operation with manageable costs.

Nonetheless, to provide a preliminary comparison of inference costs, we took four rows of increased complexity for HotPotQA (Table 2) and tested them with gpt-3.5-1106, while disabling the cache (which would otherwise skip any repeated computations, e.g. retrieval queries or LM calls).

We ran 100 questions from HotPotQA with a single thread, and we report the average latency below. Each numbered bullet below represents a program (with compiler strategy in parentheses).

1. Vanilla (fewshot)

Average Latency: 0.3 seconds per question

Average Cost: $0.0005 per question (1x)

2. CoT_RAG (fewshot)

Average Latency: 1.1 seconds per question

Average Cost: $0.0013 per question (2.6x)

3. Multihop (fewshot)

Average Latency: 2.6 seconds per question

Average Cost: $0.0018 per question (3.6x)

4. Multihop (bootstrap)

Average Latency: 2.6 seconds per question

Average Cost: $0.0041 per question (8.2x)

As this shows, these programs are within an order of magnitude of the cost and latency of the simplest one, even though we see major quality improvements from vanilla to multihop. We hope this provides deeper insight into our evaluations, and will expand on this analysis if our work is accepted.

I would still encourage you to provide some discussion of the qualitative aspects of these generated programs.

We appreciate this suggestion. If the DSPy paper is accepted, we will use the time allocated for the camera ready (due in Feb) to expand Appendix I with a brief discussion of the prompts observed when compiling, and how that aids with debugging and analysis in our view, building off the feedback from (and discussion presented in our responses to) reviewers zPoD and cQiR.

Thank you for the helpful feedback. We are thrilled that you found our results 'very impressive' and that you appreciate the 'simplicity of the user experience' of the DSPy framework.

missing related work [1], which takes a similar approach of expressing LM pipelines as programs [...] I would be happy to increase my score if the paper is revised to include a discussion comparing the two approaches.

Thank you for pointing out LMQL. We have revised the related work section to compare LMQL and DSPy. We highlight this new part in blue. In short, LMQL is query language for efficiently constraining the decoding algorithms of LMs to generate only strings that fulfill logical constrains (e.g., lists of bullets or values formatted correctly for a calculator). LMQL’s lower-level interface for controlled decoding can be beneficial to implement specific advanced modules within DSPy.

Given this, we believe that LMQL has fundamentally distinct design goals from DSPy. DSPy focuses on optimizing the quality of a pipeline of LM calls toward a given metric (e.g., optimizing the prompts of a multi-hop program to achieve high answer exact match). In contrast, LMQL helps make sure LM outputs adhere to specific formatting characteristics. The DSPy optimizations have an arbitrary continuous metric in mind, whereas LMQL optimizations focus on the efficiency of decoding under logical constraints.

A more minor concern is that the optimization techniques demonstrated here are relatively limited [...] Do the authors have any ideas for future optimization strategies (or "teleprompters") that could be implemented within this framework?

Absolutely, we have a wide range of ideas for future optimization strategies. Our focus in this work is to show that the DSPy formulation of the problem (i.e., LM programs with optimizable modules, based on natural-language signatures) creates a very large space of potential optimizations. We then focus on showing that simple strategies for creating demonstrations (for prompting or finetuning), selecting between them, and ensembling them yield very large quality gains.

Since we wrote this paper, we have also explored optimizing the instructions directly in a range of different ways, as well as strategies for automatic backtracking logic. We believe these extensions fall outside the scope of this paper, and we leave scientifically evaluating them for future work.

---

Lastly, we appreciate your offer to raise the score given a discussion of LMQL vs DSPy. We believe this discussion of LMQL has helped us shed more light on the unique aspects of DSPy in the updated paper.