LUMOS: Towards Language Agents that are Unified, Modular, and Open Source

Dear Reviewer UN8U,

Thank you for your thoughtful review! We are glad you found the modular framework to be well-motivated and noticed the potential of our dataset to help train better, smaller models. Thank you also for noting the effectiveness and generalizability of our method. Below we answer your questions:

1. Reason why Lumos-O is better than Lumos-I on math benchmarks

A potential reason why Lumos-I is not better than Lumos-O on math tasks is because the calculation results of the intermediate reasoning subgoals planned by Lumos-I are not useful for generating the next subgoal. Suppose we are solving a math problem like “James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week?” Even if the agent calculates how many times James sprints, which is 3*3=9, the mere number 9 does not affect the next subgoal generation, which could be “Calculate the total meters James runs a week”.

2. Performance-efficiency tradeoff between Lumos-O and Lumos-I

We compute the inference time for Lumos-O and Lumos-I across 100 instances on GSM8K and HotpotQA, respectively. The experiments are run with 2 NVIDIA A6000 48GB GPUs with inference batch size 16. We find that Lumos-O is much more efficient than Lumos-I on both datasets.

Lumos-O completes its inference in a single round, whereas Lumos-I necessitates multiple rounds of inference until it autonomously concludes its planning. The iterative planning and grounding in Lumos-I contribute to a higher time cost for solving individual instances. Nevertheless, Lumos-I is better at generating an appropriate subgoal based on the current external environment compared to Lumos-O.

For example, when tackling a complex question “What government position was held by the woman who portrayed Corliss Archer in the film Kiss and Tell?”, Lumos-I is able to first identify the woman who portrayed Corliss Archer in Kiss and Tell, Shirley Temple, then asks the government position she held.

However, though Lumos-O is able to first ask who the woman is, without the interaction with external knowledge such as Wikipedia pages, it will generate a subgoal which inquires the government position held by Madonna, a random entity totally irrelevant with the original question. Hence, Lumos-O is a more efficient solution, but not as effective as Lumos-I due to the lack of the adaptability to external environments.

3. Typo in final planning module annotation

Sorry about this typo. The previous turn in this conversation actually asks “Should we stop planning?”, instead of “Should we keep planning?” We will fix the typo in the camera-ready version.

Thank you!

​ Thank you very much for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion. If you find that our response addresses your concerns, would you kindly consider defending acceptance for our paper or even raising your rating score? We greatly appreciate your consideration.

Dear Reviewer xkWX,

Thank you for your thoughtful review. We appreciate your note that we contribute a solid framework to help even smaller models achieve success as language agents. Below we answer your questions:

1. Similar architecture to existing agent framework

Rather than merely presenting a plan-ground-execute modular architecture, our research is more inclined towards exploring two key areas: 1) developing suitable formulations to effectively train a superior open-source language agent, and 2) studying how to construct training annotations that are in line with our proposed formulations for planning and grounding modules.

Regarding the training formulations, very few previous studies on agent fine-tuning studies which formulations might enhance agent training. Many established agent frameworks (like ReAct and ReWOO) depend solely on a single model for both planning and grounding tasks. Whereas, we are unsure about the real efficacy of using a single model for learning both planning and grounding skills.

Moreover, there's a possibility that simple chain-of-thought fine-tuning might suffice for complex tasks such as math and complex QA tasks [1]. Open-source LLMs such as LLAMA-2 have already learned a substantial amount of basic math and general knowledge during their pre-training phases. Through chain-of-thought fine-tuning, these models could potentially acquire planning skills fast, and utilize their stored knowledge without any external math and QA tools. As indicated in Section 4.3, we uncover the performance of various open-source agent training formulations. This could serve as a practical guide for developing more advanced open-source agents in the future.

[1] Distilling Reasoning Capabilities into Smaller Language Models. Shridhar et al., 2023. ACL Findings 2023.

2. Analysis on the error patterns of Lumos-O and Lumos-I on math tasks

By observing the cases where Lumos-I makes mistakes, we find that Lumos-I is slightly worse than Lumos-O to terminate the planning. A typical example is shown as follows. There’s a math problem: “James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week?” Lumos-I will only solve part of the problem by merely calculating 3*60=180, while Lumos-O could generate the perfect solution. Another example is “Kylar went to the store to buy glasses for his new apartment. One glass costs $5, but every second glass costs only 60% of the price. Kylar wants to buy 16 glasses. How much does he need to pay for them?” Lumos-I will repeatedly generate subgoals “Calculate the cost of the third/fourth/fifth/sixth … glass.”

Another possible reason why Lumos-I is not better than Lumos-O lies in the ineffectiveness of the intermediate reasoning subgoals formulated by Lumos-I for the generation of subsequent subgoals. Consider the aforementioned mathematical problem such as, “James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week?” Even if the agent calculated how many times James sprints a week, which is 3*3=9, the mere number 9 does not affect the next subgoal generation, which could be “Calculate the total meters James runs a week”, without any mentioning of the previous calculation results.

3. Discussion about the comparison between Lumos and UA-T

UA-T represents a training approach that teaches a single LM to simultaneously handle both planning and grounding tasks. However, due to the substantial differences between the goals of planning and grounding, training a single model on both skills might impede the development of each skill individually.

Moreover, incorporating grounding instructions and processes into the planning stages significantly lengthens the model's input data. When generating the last subgoal, the input size for UA-T is 359.5 words, 2.63 times longer than the original input dimensions used for the Lumos planning module. When generating a new subgoal, this increase in input length could lead to the model partially losing track of earlier planning contexts. Consequently, this might adversely affect the accuracy of subsequent subgoal generation.

Thank you!

We're grateful for your thoughtful questions and helpful advice. Please feel free to reach out if you have any additional queries. If you feel that our response adequately addresses your concerns, would you like to consider raising your rating score for our paper? Thank you for your consideration!

Dear Reviewer 7VK9,

Thank you for engaging with our work and for noting the quality of our methodology and experimental setup. Below we answer your questions:

1. Clarifications about evaluation metrics

We followed the evaluation metrics adopted in the well-known agent evaluation papers such as AgentBench: a holistic agent evaluation benchmark [1], ReAct [2], AgentLM [3], FiReAct [4], and ReWOO [5]. F1 is actually not the mainstream metric to evaluate agents on complex interactive tasks.

For more comprehensive evaluation as the reviewer suggested, we leverage character-level F1 score, one QA evaluation metric used in ReWOO which evaluates the extent of character overlap, to evaluate agent performance on HotpotQA. Here’re the results:

Via the new metric character-level F1, we find that our LLAMA-2-7B-based Lumos still achieves competitive performance against gpt-3.5-turbo-based agents such as ReAct and ReWOO. It also surpasses GPT-3.5-CoT by a large margin.

[1] AgentBench: Evaluating LLMs as Agents. Liu et al., 2023. Arxiv 2308.03688.

[2] ReAct: Synergizing Reasoning and Acting in Language Models. Yao et al., 2023. ICLR 2023.

[3] AgentTuning: Enabling Generalized Agent Abilities for LLMs. Zeng et al., 2023. Arxiv 2310.12823.

[4] FireAct: Toward Language Agent Fine-tuning. Chen et al., 2023. Arxiv 2309.05653.

[5] ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models. Xu et al., 2023. Arxiv 2305.18323.

2. Reason why we don’t study publicly available APIs

GPT-based APIs can be excessively costly, especially for tasks involving lengthy contexts like web agent tasks, where encoding HTMLs is a necessity. Also, these APIs are seldom deterministic, making agent reproduction challenging. Moreover, the closed-source nature of these LLMs hinders a comprehensive understanding of their architectures and internal behaviors.

Moreover, commercial use constraints may force companies to develop their proprietary agents based on open-source models. Such reliance on closed-source LLM-based agents is not conducive to the growth of the research and industry community, suggesting a shift towards the use of open-source LLMs.

Lastly, instead of simply showing the effect of fine-tuning, we discuss which training formulations could further improve the agent fine-tuning performance and which data construction methods could contribute to better training annotations for planning and grounding modules. Lumos is one of the first works that study the key elements in agent training process in detail.

3. Clarification about the adopted results in Table 1

• “On GSM8K, ReWOO achieves 62.4% accuracy”: According to the first sentence of Section 3.2.1 in ReWOO paper, “Table 2 shows the main evaluation results on public benchmarks and curated dataset based on gpt-3.5-turbo”, 62.4% indicates gpt-3.5-turbo results.
• “Cannot find the claimed LLAMA-7B results in Xu et al. 2023”: Figure 6 in the ReWOO paper demonstrates their LLAMA-7B-based Planner results.
• “Achieves 66.6% accuracy on StrategyQA”: The 66.6% accuracy is also from Table 2 in the ReWOO paper, which is about gpt-3.5-turbo result.

4. Comparisons on Mind2Web

In a recent paper AgentTuning published (10/19) after ICLR submission deadline, they fine-tune LLAMA-2-70B with Mind2Web training data. 7B-size Lumos (27.7% step success rate) still exceeds AgentLM-70B (13.5% step success rate) by a large margin. We will append the new comparison results and discussion in Section 4.2.

5. Typo fixing

Thanks for pointing out the typos! We will address these editorial comments, proofread, and improve the writing of the paper in the revision.

6. Other questions

6.1. Main commonality between the related works in Self-Instruct paper and the ones in ours:

Self-Instruct mainly discusses the method of improving LLMs with their self-generated instruction tuning data. One of the main contributions of Self-Instruct is the automatic approach to distilling GPT-3 for generating instruction-tuning data for further training. As Lumos constructs the training annotations with the aid from GPT-4, one of the common parts of both paper’s related works involve “language models for data generation” which is the second paragraph in Section 5 of Self-Instruct paper.

Also, the common usage of Self-Instruct in recent literatures such as Alpaca [6] and Symbolic Chain-of-Thought Distillation [7] is to synthesize annotations from scratch for training small LMs. Since the goal of Lumos is to equip small LMs with agent ability, one intuitive solution is distilling agent tuning data from LLMs for training. Hence, another common part of the related works is about “knowledge distillation” which is the fifth paragraph in the Self-Instruct paper.

[6] Stanford Alpaca: An Instruction-following LLaMA model. Taori et al., 2023. https://crfm.stanford.edu/2023/03/13/alpaca.html

[7] Symbolic Chain-of-Thought Distillation: Small Models Can Also" Think" Step-by-Step. Li et al., 2023. ACL 2023.

6.2. Main difference between the related works in Self-Instruct paper and the ones in ours:

Self-Instruct focuses on relatively simple instruction following settings, e.g., studying whether LLMs can write an email to a professor. However, Lumos focuses on evaluating if language agents can accomplish the complex interactive tasks that require multi-step reasoning with interaction with external environments.

Thus, we discuss language agents which are not mentioned in Self-Instruct. Besides, the instructions generated by Self-Instruct are usually much simpler than our studied complex interactive tasks. Relying simply on Self-Instruct to create complex interactive task annotations can lead to a significant number of errors and diminish the overall quality of the annotations.

In Lumos “Improving Capacity of Smaller Models” paragraph, we further discuss the motivation of transferring the existing annotated gold reasoning steps into the formats aligning Lumos formulations, instead of directly applying the Self-Instruct method.

6.3. Impact and future works

As we mentioned in the previous rebuttal part, closed-source API-based agents could suffer from issues regarding affordability, reproducibility, and intellectual property. Pushing forward the development of building strong open-source language agents would be a more proper solution that considers the capabilities and alleviates the aforementioned issues.

Lumos is one of the very first works comprehensively studying which training formulations and data sources should be incorporated to facilitate the performance on complex interactive tasks. In terms of the long-term future works, we hope that based on Lumos training formulations and high-quality training annotations, the future language agent research could shed light on designing more effective, efficient and controllable methods to obtain powerful open-source language agents.

With regard to short-term goals, we plan to train the next-generation Lumos with self-reflection ability. In other words, the next-generation Lumos should realize the planning mistakes they have made and automatically correct the previous planned subgoals until the mistake disappears. Also, we aim to harmonize the language agents with reinforcement learning that makes the planning more adaptive to the external environment and further optimizes the planning modules.

6.4. Limitations

Currently, Lumos is unable to replan and reground once the grounded actions cannot be executed. A more flexible agent framework with self-correction ability is desired. Besides, when testing the agents based on Lumos-Iterative (Lumos-I) formulation, we sometimes encounter a situation where the agent cannot terminate the planning at the proper time. A controllable stopping mechanism for the iterative agent formulation is also expected.

Thank you!

Thank you for your insightful questions and valuable advice. If you have any more questions, we're more than willing to continue the discussion. If you find that our response addresses your concerns, could you please consider increasing your rating score for our paper? Your consideration is highly appreciated.

Dear Reviewer 9Mw2,

We thank the reviewer for engaging with our work, and recognizing the importance of the unified framework we contribute in this paper.

We would like to address your concerns in detail below.

1. Better baseline model comparison

Prior to delving into our recent comparative experiments, it's essential to provide context and a timeline concerning the agent evaluation and modeling studies.

As of the ICLR submission deadline (9/28), there existed only ONE public version of AgentBench [1], which was released on 8/7. All the results referenced in Lumos are derived from this initial version. The subsequent version, referred to in the review, was released later on 10/25. Therefore, we are not able to cite results from Vicuna-13B v1.5 based on LLAMA-2 from AgentBench Version 2 [2] prior to the ICLR deadline. We will update it in the following versions.

[1] (Version 1) AgentBench: Evaluating LLMs as Agents. Liu et al., 2023. https://arxiv.org/abs/2308.03688v1

[2] (Version 2) AgentBench: Evaluating LLMs as Agents. Liu et al., 2023. https://arxiv.org/abs/2308.03688v2

Additionally, prior to 9/28, the dominant method in language agent research up to that date was few-shot in-context learning, exemplified by models like ReAct, ReWOO, and Reflexion. Very few studies had explored the fine-tuning of smaller LMs to compete with the performance of GPT-series or other significant open language agents.

Consequently, we did not specifically emphasize in-context learning settings at that time. After the submission deadline, novel approaches to language agent fine-tuning began to emerge, including FiReAct (10/9) and AgentLM (10/19), marking the beginning of research into enhancing smaller LMs with language agent capabilities through fine-tuning.

In terms of the comparison with LLAMA-2-based agents, we note that some LLAMA-2-based agent performance is already shown in Table 1 and we have already provided some comparison. To further address the reviewer’s concern, we compare 1) more baselines of which the base models belong to LLAMA-2 series, and 2) the concurrent two language agent fine-tuning methods, FiReAct [3] and AgentLM [4].

[3] FireAct: Toward Language Agent Fine-tuning. Chen et al., 2023. Arxiv 2309.05653.

[4] AgentTuning: Enabling Generalized Agent Abilities for LLMs. Zeng et al., 2023. Arxiv 2310.12823.

2. Lumos vs. other LLAMA-2-based models

• For math tasks, some baselines listed in Table 1(b) are based on LLAMA-2. Described in Table 3 and 4 of MAmmoTH [5], the base model of Orca-Platypus-13B baseline is LLAMA-2. It is trained with many math CoT data constructed upon FLAN Collection. 7B-size Lumos-O outperforms Orca-Platypus-13B on GSM8K and SVAMP 12.1% and 8.6% accuracy.
• For web agent task, Mind2Web, shown in Table 3 of the latest AgentBench v2, Vicuna-13B v1.5 only gets a 12% step success rate, ~15% underperforming Lumos. LLAMA-2-70B-chat also lags behind Lumos by a large margin.
• For WebShop, also shown in Table 3 of the latest AgentBench v2, Lumos outperforms LLAMA-2-13B-chat 14.5 average reward.

[5] MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning. Yue et al., 2023. Arxiv 2309.05653.

3. Lumos vs. FiReAct and AgentLM

• For math tasks, Lumos achieves 18.1% accuracy improvements over AgentLM-13B on GSM8K (based on Table 4 of AgentBench).
• For complex QA tasks, we use Exact Match (EM) as the evaluation metric to keep it consistent with FiReAct and AgentLM. Despite FiReAct being fine-tuned with in-domain HotpotQA annotations, Lumos, without any fine-tuning on HotpotQA annotations, still presents an impressive improvement. In particular, Lumos surpasses the 7B-scale FiReAct by 3.2% EM. Lumos also has a 7.1% EM improvement over AgentLM.
• For web agent tasks, Mind2Web, Lumos surpasses AgentLM-70B fine-tuned on Mind2Web 14.1% step success rate.

Overall, we show that Lumos is still competitive with many LLAMA-2-based agent baselines and has superior performance to the concurrent works on most of our evaluated tasks. We will update our paper with these new results.

4. More few-shot generalization experiments

We extend the few-shot generalization experiments on WebShop to 13B-scale. Specifically, We adopt LLAMA-2-13B as the fundamental model for both the planning and grounding modules. We show the results as follows:

Experimental results suggest that 1) Lumos still has better generalizability than the other LLAMA-2-13B-based models even API-agent agents under the same few-shot in-context learning settings.; 2) Lumos-I trained with unified annotations also outperforms the ones trained with domain-specific annotations Lumos-I-math/complex-QA/web-agent on unseen tasks.

5. Novelty on model architecture

Instead of simply presenting a plan-ground-execute modular architecture, we focus more on studying 1) how to train a better open-source language agent, and 2) how to prepare high-quality planning and grounding annotations that align with the proposed formulations.

In terms of the training formulations, since there are very few prior works exploring agent fine-tuning, it is unclear which formulations would be more beneficial to train a better agent. In fact, many well-known agent frameworks (e.g., ReAct and ReWOO) fully rely on a single model to perform planning and grounding. However, we are unsure about whether training a single model is the most appropriate approach to managing planning and grounding skills.

Additionally, it is also possible that chain-of-thought fine-tuning is already good to tackle math and complex QA tasks [6]. As shown in Section 4.3, we unveil the effect of possible open-source agent training formulations. It could provide a practical reference to lead the path for training more powerful open-source agents in the future.

Regarding the training annotation construction, one intuitive solution is to simply leverage methods such as Self-Instruct to generate tasks, plans and grounded actions by LLMs from scratch. Whereas, these methods are not suitable for generating high-quality annotations for complex interactive tasks, as even GPT-4 performs badly on the complex interactive tasks we study, such as Mind2Web. Relying on these methods to create complex interactive task annotations can lead to a significant number of errors and diminish the overall quality of the annotations. To this end, it is necessary to think of potential novel methods that can create large-scale and high-quality annotations to train planning and grounding modules.

Finally, we propose a prompting-based conversion method that utilizes the ground-truth reasoning steps in existing benchmarks to generate the subgoals and actions and transfer to conversational formats that fit the proposed formulations. Appendix E demonstrates that by controlling the training annotation size to be the same, our annotations can still help get better performance than the ones produced by the Self-Instruct method and passed by rigorous execution sanity checking.

Overall, Lumos involves the discussion beyond merely presenting an architecture. We care more about what training strategies can construct a better open-source agent. It covers the comprehensive study on the choices of the key elements in the agent training process including training formulation and annotation perspectives.

[6] Distilling Reasoning Capabilities into Smaller Language Models. Shridhar et al., 2023. ACL Findings 2023.

6. Clarification about baseline model comparison

To address the reviewer’s concern about baseline result comparison in Table 1, we conduct the prompting experiments for each task type’s strongest baselines of which the calling prompts are publicly released in AgentBench and ReWOO [7] papers. Especially for WebShop, we adopt the prompts described in AgentBench paper and rewrite their in-context examples to be consistent with ours. The results are shown as follows:

We show that Lumos-I still exceeds the compared baselines in Table 1 after the reproduction.

[7] ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models. Xu et al., 2023. Arxiv 2305.18323.

—

7. Other questions

“Is it true that most other baseline LLM agents are applied under few-shot / in-context learning setting?” - Yes, we will add this information in the latter version. Recently, two fine-tuned language agents, FiReAct (10/9) and AgentTuning (10/19) were published after the ICLR submission deadline (9/28). As discussed in the rebuttal subsection Lumos vs. FiReAct and AgentLM, Lumos outperforms them on most of our evaluation tasks.

—

Thank you!

​ We appreciate your excellent questions and suggestions. Please feel free to reach out if you have additional questions. If you find that our response addresses your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.

3. Novelty of Lumos architecture

​ Instead of simply presenting a plan-ground-execute modular architecture, we focus more on studying 1) how to train a better open-source language agent (Section 2, 3 and 4.3), and 2) how to prepare high-quality planning and grounding annotations that align with the proposed formulations (Section 3 and Appendix E).

Concretely, in terms of training formulations, we discuss the detailed input and output contents of each module, how to build the collaboration between the modules, where the supervision signals could be constructed, and comparison with multiple potential agent training formulations.

Regarding annotation construction, we propose a prompting-based conversion method to acquire high-quality large-scale annotations that align with the formulations of planning and grounding modules. We conduct rigorous experiments to show the quality of our annotation construction method with a comparison to the commonly used Self-Instruct method by controlling the number of annotations.

To our best knowledge, Lumos is one of the first studies concentrating on investigating the detailed components (training formulations and data) involved in agent training strategies prior to ICLR submission deadline.

Thank you!

​ We sincerely thank the reviewers for their constructive suggestions and questions to enhance our paper. Please reply if you have any further questions, and we will be more than happy to continue the discussion.