Chain-of-Experts: When LLMs Meet Complex Operations Research Problems

We sincerely appreciate the constructive suggestions and comments on our work, which have definitely helped us to enhance the paper. The following are our detailed responses to each of the reviewer's comments.

Comment 1:

The results on ComplexOR seem not sense. A too small dataset.

We understand the reviewer's concern. ComplexOR can be viewed as the first dataset that has been well annotated for complex operations research problems and it has great potential to evaluate the reasoning capabilities of LLMs. The dataset is not large-scale because it is rather labour intensive for domain experts to help annotate the problems. To construct an annotated problem instance, we need to

1. Find problems with suitable complexity from sources like academic papers, textbooks, or tutorials
2. Manually extract the textual description of the problem
3. Construct the mixed integer programming model and double check if it is correct
4. Programming for this model
5. Design several test cases with varied difficulty

Since we are firmly aware of its value to the community, we continued to collect more samples after the paper submission. Till now, we have expanded the size of ComplexOR from 20 instances to 37 instances. The new experimental results on ComplexOR have been updated in the revision.

Comment 2:

The description of CoE is not clear. It should be well-moviated and started with several backgrounds.

In revision, we will follow the reviewer's advice to better motivate the description of CoE. Our thought started with Solo Performance Prompting, a multi-agent system. While SSP shows potential in solving complex tasks like logical grid puzzles, it falls short in more domain-specific taks such as the OR problem. We identified two primary reasons for this limitation. Firstly, in SSP, the agents are prompt templates initialized by leader personas, which restricts their ability to integrate extensive knowledge. To address this limitation, we introduce 'experts' to play specific roles in the problem-solving process. Secondly, we find that the collaborative approach in SSP is too straightforward to tackle challenging problems. It lacks the ability to selectively choose experts based on the specific problem and refine based on external feedback. To this end, we propose forward thought construction and backward reflection mechanisms inspired by real-world collaboration processes.

Comment 3:

Is the CoE suitable for other reasoning tasks? What is the difference if applied to other tasks?

This is a good point. In this paper, the agents/experts for collaborative reasoning are customized from the domain of operations research. As to agent design, the terminology interpreter who is responsible for supplying knowledge about specific terminology is integrated with a scenario modeling knowledge base. The programming expert is designed to generate modeling code, specifically in the LP file format.

To extend CoE to support other reasoning tasks, the first step is to reconstruct proper agents for the target domain. Luckily, the collaborative reasoning framework among the agents with forward thought construction and backward reflection is re-usable.

To evaluate the reusability of our framework, we conducted a preliminary experiment on Text2SQL task. Specifically, we selected 12 hard test instances related to a manufacturer scenario from the spider dataset[1]. In this experiment, we configured a CoE with two experts: a manufacturer terminology interpreter and a dataset grammar expert integrated with an SQLite grammar document[2]. We compared this setup with a standard prompting baseline. The results are as follows:

We regret that time constraints prevented us from conducting more comprehensive experiments. In our upcoming revisions, we are committed to adding a formal experiment to further validate our approach.

Comment 4:

I suggest the paper give more attention to the CoE framework.

In this work, we focus on automatically solving complex OR problems. It is motivated by the real demand of our industry partners. Such an automatic tool can greatly help them to reduce the overhead of modeling and programming for operations research problems from diversified industry sectors.

We totally agree with your comment that it would generate more impact if we focus on CoE and apply it to more types of reasoning tasks. In the previous comment, we have validated its potential to support complex Text2SQL task. We believe this is an exciting avenue for future research to explore if we can further improve the generality of our reasoning framework among various reasoning tasks.

Reference:

[1] https://paperswithcode.com/sota/text-to-sql-on-spider

[2] https://www.sqlite.org/lang.html

Thank you for your insightful comments. We have thoughtfully incorporated them into our revised paper. Below, we restate your comments and provide our point-by-point responses.

Comment 1: test out CoE on some of the well-established benchmarks

This is a good point. In this paper, the agents/experts for collaborative reasoning are customized from the domain of operations research. As to agent design, the terminology interpreter who is responsible for supplying knowledge about specific terminology is integrated with a scenario modeling knowledge base. The programming expert is designed to generate modeling code, specifically in the LP file format.

To extend CoE to support other reasoning tasks, the first step is to reconstruct proper agents for the target domain. Luckily, the collaborative reasoning framework among the agents with forward thought construction and backward reflection is re-usable.

To test the reusability of CoE, we conducted a preliminary experiment using the Text2SQL dataset. Specifically, we selected 12 hard test instances related to a manufacturer scenario from the spider dataset[1]. Here, we configured a CoE with two experts: a manufacturer terminology interpreter and a dataset grammar expert integrated with an SQLite grammar document[2]. We compared this setup with a standard baseline. The results are as follows:

We regret that time constraints prevented us from conducting more comprehensive experiments. In our upcoming revisions, we are committed to adding a formal experiment to further validation.

Comment 2: ComplexOR is too small

We understand the reviewer's concern. ComplexOR can be viewed as the first dataset that has been well annotated for complex operations research problems and it has great potential to evaluate the reasoning capabilities of LLMs. The dataset is not large-scale because it is rather labour intensive for domain experts to help annotate the problems. To construct an annotated problem instance, we need to

1. Find problems with suitable complexity from sources like academic papers, textbooks, or tutorials.
2. Manually extract the textual description of the problem.
3. Construct the mixed integer programming model and double check if it is correct.
4. Programming for this model.
5. Design several test cases with varied difficulty.

Since we are firmly aware of its value to the community, we continued to collect more samples after the paper submission. Till now, we have expanded the size of ComplexOR from 20 instances to 37 instances. The new experimental results on ComplexOR have been updated in the revision, as shown in Table 1 (overall performance), Table 2 (ablation study), and Table 3 (Robustness under different LLM).

Comment 3: a baseline that use the same model

This is an inspiring comment. We followed the reviewer's advice to implement a baseline that uses the same model. In this experiment, we use a uniform system prompt across all roles, without any additional knowledge bases. We also find that the removal of experts' features leads to a decrease in accuracy, which suggests that the CoE benefits from using specialized experts over a singular, generalized model. We have added this baseline in our revision.

Comment 4: role of theta in Eqs.2 and 3

We sincerely appreciate your comment and it actually pushes us to think about the roles of $\theta$ deeply. We agree that the use of $\theta$ in Equations 2 and 3 could be misleading as CoE don't actually generate new parameters through prompting. To clarify, we've revised our approach to align with the notation used in the paper Reflexion[3]. Here, we treat the Conductor as a policy, with its parameters indicated in the subscript. The revised equation is as follows:

$Conductor_{\mathcal{F}^{\theta'}(\mathbb{PT}_t)}(e | s) = P_r\{E_{\phi_t} = e | S_t = s\}$

Reference

[1] https://paperswithcode.com/sota/text-to-sql-on-spider

[2] https://www.sqlite.org/lang.html

[3] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. CoRR, abs/2303.11366, 2023. doi: 10.48550/arXiv.2303.11366.

We sincerely appreciate the constructive suggestions and comments on our work, which have definitely helped us to enhance the paper. The following are our detailed responses to each of the reviewer's comments.

Can CoE be Represented by the SSP Framework?

Why can't the method proposed in this paper be represented through Solo Performance Prompting, and where exactly does it differ from Solo Performance Prompting?

There are two key differences between Solo Performance Prompting and our CoE. First, in terms of agent generation, Solo Performance Prompting prompts a single LLM to identify and simulate multiple agents. In contrast, the agents/experts in CoE are tailored to collaboratively solve an OR problem. For example, the Modeling Knowledge Supplement Expert is integrated with the knowledge of games-cutting edge modeling to its knowledge base. Second, in terms of collaborative reasoning framework, the agents in Solo Performance Prompting cooperate in a round-robin manner. In contrast, our CoE works with forward thought construction and backward reflection.

We also offer a case study in Appendix 4.3 to illustrate the differences of reasoning processes between Solo Performance Prompting and our CoE.

Techniques Designed for OR Problems?

While the paper focuses on tackling complex operations research problems, it doesn't seem to introduce any techniques specifically tailored for operations research challenges.

This is a good point. In this paper, the agents/experts for collaborative reasoning are customized from the domain of operations research. As to agent design, the terminology interpreter who is responsible for supplying knowledge about specific terminology is integrated with a scenario modeling knowledge base. The programming expert is designed to generate modeling code, specifically in the LP file format.

To extend CoE to support other reasoning tasks, the first step is to reconstruct proper agents for the target domain. Luckily, the collaborative reasoning framework among the agents with forward thought construction and backward reflection is re-usable.

To test the reusability of CoE, we conducted a preliminary experiment using the Text2SQL dataset. Specifically, we selected 12 hard test instances related to a manufacturer scenario from the spider dataset[1]. Here, we configured a CoE with two experts: a manufacturer terminology interpreter and a dataset grammar expert integrated with an SQLite grammar document[2]. We compared this setup with a standard baseline. The results are as follows:

We regret that time constraints prevented us from conducting more comprehensive experiments. In our upcoming revisions, we are committed to adding a formal experiment to further validation.

Comparison with Other Planning with Feedback Methods

I think the authors should offer a more in-depth comparison with these existing works. Moreover, though the methodology is described as a multi-expert framework, it essentially relies on deploying various prompts to the same LLM.

We thank the reviewer for providing the reference. The unique features of our CoE include forward agent selection and backward reflecion. We followed the reviewer's advice and provided an in-depth comparison with these existing works in the following table.

Reference:

[1] https://paperswithcode.com/sota/text-to-sql-on-spider

[2] https://www.sqlite.org/lang.html

We sincerely appreciate the constructive suggestions and comments on our work, which have definitely helped us to enhance the paper. The following are our detailed responses to each of the reviewer's comments.

Comment 1:

Lack of evaluation on individual expert agents, as well as the conductor

In our original submission, we have conducted an ablation study to evaluate individual expert agents. The reviewer can check Figure 5 in Appendix A.4.2. Due to space limit, we regret that it was not put in the main body of the paper.

Comment 2:

The comparison with other models might not be fair, since they call the LLM differently. Maybe add some measurements of how different methods use the LLMs.

In our experiments, we unify the way to use the LLMs. Each method within an experiment utilizes the same model API, facilitated by the 'chat complete' interface LangChain framework[1]. Please let us know if we did not interprete the reviewer's comment correctly.

Comment 3:

If we use other less competent LLMs, like smaller models or open sourced models, how much the performance will be affected?

This is a good angle. If we use smaller models such as Llama 2, the performance degrades dramatically. We have conducted an experiment in chapter 4.7 to show the sensitivity to different types of LLMs. For smaller models such as Llama 2, we also conducted additional experiments on dataset LPWP and presented the results in the following:

In this experiment, we use the Llama2 13b model[2] as a foundational model, applying it across various baselines and the CoE on the LPWP dataset. Our findings revealed that the Llama2 model was largely ineffective in resolving the OR problems. Approaches like CoT and Reflexion showed no significant impact. Only 2 out of 289 instances were successfully resolved using ReAct and CoE, which incorporated external knowledge. These results suggest that Large Language Models must achieve a certain scale to effectively tackle complex problems.

Reference

[1] https://github.com/langchain-ai/langchain

[2] Zhengliang Liu, Yiwei Li, Peng Shu, Aoxiao Zhong, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Jie Luo, Cheng Chen, Sekeun Kim, Jiang Hu, Haixing Dai, Lin Zhao, Dajiang Zhu, Jun Liu, Wei Liu, Dinggang Shen, Tianming Liu, Quanzheng Li, Xiang Li: Radiology-Llama2: Best-in-Class Large Language Model for Radiology. CoRR abs/2309.06419 (2023)