ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents

1. It is a technical report that lacks a scientific component. It should go to a software engineering conference, for example, International Conference on Software Engineering (ICSE) or IEEE/ACM International Conference on Automated Software Engineering (ASE). The paper can also go to Datasets and Benchmarks tutorials at AI conferences. Whereas the work is technically impressive and valuable for the software engineering community, there is no scientific value in the assessment of how good LLMs are able to use APIs. The problem at hand is purely technical and as a clear indicator for this is that over two thirds of references are links to web pages rather than scientific studies.

There are many excellent benchmark studies presented at top conferences such as ICLR, NeurIPS, and ACL. We are unsure what specific concerns you have about my work, and we are unclear whether your comments are directed at AI conferences or top software engineering conferences. If you have any concerns, please provide specific feedback, as it will help us address them more effectively.

Regarding your comments, such as "It is a technical report that lacks a scientific component" and "there is no scientific value in assessing how well LLMs can use APIs", we must respectfully disagree. Our work establishes a pipeline for rapidly collecting real-world action sequences and provides findings that we believe will be helpful to others, backed by substantial analysis, thought, and effort.

The top-tier conference papers dedicated entirely to datasets/benchmarks include, but are not limited to, the following:

• (ICLR’24) SmartPlay : A Benchmark for LLMs as Intelligent Agents
• (ICLR’24) AgentBench: Evaluating LLMs as Agents
• (ICLR’24) ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
• (ICLR’24) MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
• (NeurIPS’24) FinBen: An Holistic Financial Benchmark for Large Language Models
• (NeurIPS’24) ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination

Evaluation has always been a critical component in guiding the development of AI, so we do not understand why you believe "there is no scientific value in assessing how well LLMs can use APIs". We would appreciate further clarification on what you define as "science" and "engineering", and we look forward to any insights you might offer.

As for your comment, "The paper can also go to Datasets and Benchmarks tutorials at AI conferences", we would like to clarify that when we submitted to ICLR, we chose the Primary Area of "Datasets and Benchmarks", and the keywords we selected included "Benchmark". ICLR has accepted many high-quality benchmark evaluation papers.

Regarding your concern that "two thirds of references are links to web pages", we must emphasize that the web pages we referenced were considered necessary to clearly explain our methodology. I apologize if this caused any inconvenience. We will strive to reduce the use of web citations and ensure that we present our work as clearly as possible with minimal references.

Finally, thank you for your kind words about our paper being "technically impressive and valuable".

2. The analysis of the performance of GPT 4o with the structured output capability was not performed. https://openai.com/index/introducing-structured-outputs-in-the-api/

Structured output refers to a function that controls the output format to ensure it is correctly formatted, such as in JSON. This is not directly related to the main focus of our research (API selection, parameter filling, and awareness in requesting necessary input). Furthermore, as mentioned in the paper, we did not conduct evaluations with GPT-4o due to budget constraints.

3. The used language appeals to emotions rather than states the scientific value (ex: “We made great efforts to evaluate…”).

We apologize for any confusion caused by this statement. We will revise it in the next version.

4. Several references are outdated, pointing to arxiv whereas the paper has been already published at a conference (ex: ToolLLM).

We apologize for not updating the correct citations in a timely manner. We will carefully review and ensure the accuracy of all citations in the next version.

Thank you once again for your feedback. I truly hope that my responses above have helped clarify any concerns you may have. We would be very grateful if you could carefully consider your final rating. Please don’t hesitate to reach out with any additional questions or comments—I would be more than happy to assist further.

We sincerely hope our response has addressed your concerns. We kindly ask you to carefully consider your scoring. If you have any further questions, we are always happy to assist.

Thank you for your understanding! We have thoroughly reviewed your comments and addressed them in the updated PDF.

Concern 3:

We have removed the relevant statements from the text, particularly in the later paragraphs of the Introduction. Please take a look!

Concern 4:

We have removed most URL references in the text and used descriptive explanations instead. Additionally, we updated citations for all papers with formal references. For conferences like ICLR/NeurIPS, where no official references were available, we used the OpenReview versions of the papers.

Additional Notes:

As the discussion phase concludes on November 26 (AOE), we have implemented all the changes we committed to so far. However, there may still be some oversights, and we will use the remaining time to carefully review and refine our revisions.

Thank you for your kind words about the technical strength of our work. We hope that after addressing your concerns, you will further consider your final score. If you have any additional questions, we are always happy to assist!

We have conducted additional experiments on language models fine-tuned specifically for Agents, hoping to provide further insights. If you’re interested, feel free to check out the replies in XY2i’s thread.

Thank you again for your valuable feedback. If you have any further questions, we are always here to help!

1. Limited evaluation: The paper primarily assesses the model’s ability to choose correct actions based on ground-truth sequences but doesn’t evaluate its end-to-end task success rate (as done in, for example, AppWorld [1]). Experiments linking these aspects are missing.

Thank you for your comment. We will consider building a relevant dynamic runtime environment on the macOS platform in future work. We have already undertaken some preliminary tasks, such as validating shortcut functionality, enabling apps exclusive to iOS to run on macOS via decryption, and translating API calls generated by LLMs into shortcut commands. To fully evaluate whether a shortcut is "successful", Apple Scripts are required to run the shortcuts and capture signals from the associated apps to determine the final status. Otherwise, manual evaluating would be necessary, and we regret that we cannot "manually verify" all model-generated shortcuts in the short term.

Nevertheless, we believe that ShortcutsBench offers distinct features and advantages compared to existing datasets/benchmarks, we believe we have provided a substantial amount of evaluation results, covering areas such as API selection, parameter filling, and feedback awareness in agent-based API execution. We hope these contributions will inspire future research in the field.

We will also add a citation for AppWorld and discuss relevant similarities and differences in our paper.

Thank you again for your valuable comment.

2. No human validation: Given the synthetic nature of the benchmark, it’s uncertain whether all tasks are truly solvable or what the benchmark’s upper bound is. Including human performance as a reference would add clarity.

Regarding "whether all tasks are truly solvable", please refer to our response to reviewer "XY2i" on their first question. In summary, we implemented three steps to ensure the quality of our dataset, with both Step 2 and Step 3 involving manual verification. Since these shortcuts are available in the Shortcut Store specifically to address "real user needs" and have undergone human review, we believe there are no inherently "unsolvable" shortcuts.

As for "what the benchmark’s upper bound is", we have not conducted a manual evaluation of API selection and parameter filling for each shortcut. However, we would like to note that none of the existing works referenced in Table 2, and also, AppWorld, as you mentioned, have performed this level of manual validation.

We are very open to performing a manual validation of API selection and parameter filling. However, it is challenging to complete this in the short term. API calls require skilled individuals with specific knowledge of Shortcut programming, and parameter filling entails complex annotation tasks. Moreover, ShortcutsBench includes a substantial number of shortcuts, further adding to the workload.

We sincerely appreciate your valuable feedback, and if you have any further questions, we are always happy to assist.

3. A notable feature of this benchmark seems to be scalability -- extracting real-world APIs and action sequences from Shortcut apps and sharing sites seems relatively easy. Providing quantitative details on dataset construction—such as time taken for each step and the actual extent of manual effort involved—would be nice.

Thank you for your valuable feedback. Scalability is indeed a key feature of our framework. In fact, it’s entirely possible to set up a pipeline where shortcut programming is performed manually to generate action sequences, allowing us to efficiently construct a large number of action sequences.

As for the specifics regarding the time required to build the dataset, we regret that we did not keep detailed records, but we can provide a general estimate:

• Collecting shortcut sites: 3 days; this phase was entirely manual.
• Scraping links via Selenium automated testing tool: 2 weeks; this step required custom code for each site.
• Shortcut deduplication, API validity check, and shortcut validity check: 4 weeks; shortcut deduplication was automated through iCloud links and action content cleaning. API validity checks require manual verification. Shortcut validity checks combined with automated and semi-automated methods: automated filtering was conducted using Apple Scripts to run shortcuts for initial filtering, followed by manual loading of shortcuts into the Shortcuts app for verification.

There were also various additional manual and automated checks, which we will not detail here.

Thank you for recognizing our work. I hope my previous responses have addressed your questions. If you have any new questions or if anything remains unclear, we’re always happy to provide further clarification.

Thank you for your timely response and encouraging feedback! We have thoroughly reviewed your comments and addressed them in the updated PDF.

Concern 1:

We added a reference to AppWorld with a brief introduction in the second paragraph of Section 2: Related Work.

Concern 2:

In Section 3.2 (Query and action sequence), we elaborated on our efforts to ensure query quality, with further details available in Appendices A.1 and A.2:

• The first paragraph discusses measures taken to maintain shortcut usability.
• The final paragraph highlights actions to improve query accuracy.

Additionally, the full processing logs and records are publicly accessible at https://anonymous.4open.science/r/ShortcutsBench.

Concern 3:

In Contribution 2 of the Introduction, we added the statement: “We hope this approach to dataset construction will inspire more researchers”. We added descriptions of the estimated time for each operation in Appendix A.1.

Additional Notes:

As the discussion phase concludes on November 26 (AOE), we have implemented all the changes we committed to so far. However, there may still be some oversights, and we will use the remaining time to carefully review and refine our revisions.

Thank you once again for your positive feedback! If you have any further questions, I’d be happy to assist.

We have conducted additional experiments on language models fine-tuned specifically for Agents, hoping to provide further insights. If you’re interested, feel free to check out the replies in XY2i’s thread.

3. The paper does not make it very clear what is their most important finding in the abstract/intro/conclusion. It would be good if they could highlight their most important findings in the abstract/intro/conclusion. For example, they could discuss how open source models perform comparably to closed source models on simpler tasks but not harder tasks.

In the second-to-last paragraph of the “contributions” in the Introduction, we have highlighted the findings we believe may be of the greatest interest to the reader. We believe each finding may impact different readers in various ways. You can find the detailed findings in Section 4.

1. Section 3 is not well-elaborated. readers will benefit from clearer description of this process. For example, for (2), the authors say 'after duplicating based on icloud link, ....', it is not very clear what is duplicated and why this step helps. It would be good if the authors could refine their descriptions on their methodology.

With the sentence "After deduplicating based on 'iCloud link' (Apple, 2024b), we got the source files of all 8675 shortcuts", we intend to convey the following:

• During the data construction process, we extracted shortcuts from multiple different Shortcut repositories. Since these repositories store shortcuts as iCloud links, the same shortcut might appear in multiple repositories. Additionally, the actual shortcut content (i.e., the action sequences) linked by these iCloud links could also be identical. Therefore, we applied deduplication to ensure that each shortcut in the final dataset was unique.

Beyond deduplication, we performed several additional operations, such as filtering out shortcuts dependent on non-existent APIs and importing all shortcuts into the Shortcuts app to verify their validity and functionality. For conciseness, only key steps are summarized in the paper. The complete process is available at https://anonymous.4open.science/r/ShortcutsBench, and further supplementary details can be found in the Appendix.

If you have any further questions, please don’t hesitate to let us know. We would be glad to assist in any way we can.

2. The authors cite each work too many times in the paper, for example, a research paper is is cited five times in one paragraph in Section 2. Referencing previous works is good practice, but referencing too many times affects readability. It would be good if they could remove repetitive references.

We apologize for any inconvenience this may have caused. Each cited paper is referenced multiple times because each has a distinct focus and contributes uniquely to different aspects of our discussion. In cases where methods fall under various classification schemes, multiple references to the same work allow for more accurate contextual alignment with each specific point.

We apologize for any inconvenience caused by the extensive citations or repeated references across sections. For the final published version, we will adjust by moving some citations to the appendix to streamline the reading experience.

4. Why is all API-based agents the authors evaluate based on the ReACT framework, it would be good if they could provide additional evaluation/analysis on other frameworks, such as CodeAct.

ReACT is a generate-feedback alternating execution method, and our reasons for using the ReACT are as follows:

• It aligns with the settings of prior work, such as MetaTool, ToolLLM, ToolAlpaca, and ToolQA mentioned in Table 2, whereas APIBench, APIBank, and ToolBench use zero-shot and multi-shot settings.
• A simple yet effective framework better reflects the model’s capabilities, rather than the capabilities of the framework itself.

Our research focuses on evaluating API-based agents, whereas CodeAct—mentioned in your feedback—is a valuable approach for generating Python code, as referenced in the second paragraph of our related work. However, it is not directly relevant to our research.

We will include a discussion of CodeAct in the related works section.

Thank you once again for your feedback. We truly hope that our responses above have helped clarify any concerns you may have. We would be very grateful if you could carefully consider your final rating. Please don’t hesitate to reach out with any additional questions or comments—I would be more than happy to assist further.

Hello,

The issues you raised include:

Weakness 1: Concerns about the clarity of certain statements in the paper. We sincerely apologize for any confusion caused and hope you can understand our choices. ShortcutsBench explores an innovative perspective by constructing a pipeline to extract high-quality data from existing Digital Automation Platforms for benchmarking. Inevitably, this involves specific terminology and expressions related to these platforms. While we have made efforts to simplify the language, add footnotes where ambiguity might arise, and include detailed processes in the appendix, we acknowledge that we may not have conveyed every statement as clearly as possible.

Weakness 2: Concerns about the frequency of certain citations. We regret any inconvenience caused. Our intention in using multiple references across different sections was to ensure that every claim in the paper is properly supported by relevant paper, and we hope for your understanding in this regard.

Regarding your question about the use of the ReACT framework, please refer to our detailed response for clarification.

ICLR emphasizes innovative ideas, rigorous work, and full openness. To our best known, ShortcutsBench represents the first attempt to leverage data from Digital Automation Platforms to construct high-quality benchmarking datasets. To ensure robustness, we conducted experiments on agents constructed using 10 open-source and proprietary models, while maintaining full openness in data acquisition, processing, and evaluation, with comprehensive logs throughout the process.

We greatly appreciate your evaluation of the paper’s Soundness (3/4) and Contribution (3/4). We hope our responses can address your concerns regarding Presentation (1/4). If you have more specific feedback on Presentation, we welcome your suggestions. We sincerely hope you will take all aspects of ShortcutsBench into consideration when determining your final score.

Thank you again for your thoughtful feedback.

Thank you so much for your prompt response and positive feedback! We have carefully reviewed and incorporated all your comments and concerns into the revised PDF.

Specifically:

Concern 1:

We made substantial revisions to Section 3, aiming to better convey our intentions more precisely. Key updates include:

• Redrawing Figure 1:
  • Removed unnecessary icons (e.g., the spider icon representing crawlers).
  • Adjusted the placement of critical steps (e.g., “Deduplication”) for greater accuracy.
  • Refined the depiction of app sources, added a legend, and introduced Step 6 to indicate data filtering.
• Reorganizing Section 3.1:
  • Removed references to most URLs.
  • Expanded on “after deduplicating based on iCloud links, …”.
  • Reorganized the discussion of app sources, relocating details about “WFActions.json” to the second paragraph.
  • Further elaborated on API deduplication in the second paragraph. For a detailed discussion on why we perform API deduplication, please refer to Appendix A.1.
  • In the final paragraph, we remind readers to refer to Appendix A.1 for further details on data acquisition.
• Streamlining Section 3.2: Simplified descriptions of APIs for improved readability.
• Adding Context in Section 3.2 (Query and action sequence):
  • Included descriptions of efforts to maintain shortcut usability in the first paragraph.
  • Highlighted efforts to ensure query accuracy in the last paragraph.
• Refining Section 3.3: Simplified citations and phrasing for clarity.
• Additional edits for smoothness and coherence throughout the section.

Concern 2:

We substantially reduced and refined citations across the paper. Previously, we aimed for comprehensive references for each point, but now we’ve retained only the most relevant citations:

1 Introduction:

Significantly trimmed references, especially in the second paragraph (“Some existing benchmarks/datasets…”). Initially, every critique of prior work was accompanied by specific citations. Now, we have removed most of these references and instead direct readers to Table 2 for a concise summary.

2 Related work & 3 Dataset:

Significantly streamlined references: we ensure that each section contains no more than two citations, with the same paper cited only once per paragraph.

4 Evaluation

All references to LLMs have been removed, and citations for related work have been streamlined. We believe the paper provides a sufficiently precise description of the models, enabling readers to independently locate and utilize them without citation.

Concern 3:

We relocated the findings you highlighted to the “key contributions” part at the end of the introduction.

Concern 4:

In Section 4.1 (“Prompt Template”), we added citations for the ReACT method under “Following existing work” while limiting the overall number of references for brevity.

Additional Notes:

We understand that the detailed process of collecting, cleaning, and constructing ShortcutsBench cannot be fully covered within the constraints of a 10-page paper. Thus, we focused on ensuring the content is both concise and clear. For more detailed explanations, please refer to the Appendix, where we revised the descriptions of the entire collection, cleaning, and construction process. Additionally, the implementation details and logs are all available at https://anonymous.4open.science/r/ShortcutsBench.

In addition to addressing your feedback, we have also made revisions to various parts of the paper based on comments from other reviewers. You may refer to the specific comments for details.

As the discussion phase concludes on November 26 (AOE), we have implemented all the changes we committed to so far. However, there may still be some oversights, and we will use the remaining time to carefully review and refine our revisions.

Thanks again! We greatly appreciate your feedback and look forward to your further comments! If you have any questions further, please feel free to reach out—I’m happy to assist.

1. Although the paper emphasizes that the benchmark includes high-quality human-annotated action sequences from shortcut developers and queries derived from real user demands, it only mentions the shortcut developers are our annotators. Further details in this area would be beneficial.

   In addition to the inherent quality assurance of shortcuts developed by shortcut developers in ShortcutsBench, we implemented three additional steps to ensure the quality of action sequences:

   1. As mentioned in the final paragraph of Section 3.1 (and detailed in Appendix A.1), we removed duplicate shortcuts by iCloud links. Specifically, we removed duplicate shortcuts based on their iCloud links and action sequences; any shortcuts with either of these identical elements were deduplicated.
   2. Additionally, we manually filtered out shortcuts that relied on outdated or inactive APIs, as well as those requiring APIs from paid apps. This ensured that all included shortcuts use operational APIs. You can review the details of this process in our open-source documentation. This was a manual process, involving two reviewers who tested on macOS Sonoma and iOS 17, downloading all relevant apps and verifying API functionality. APIs deemed obsolete were removed directly; you can view a list of these APIs here. Apps used in this evaluating process can be downloaded following the guidance in our README.
   3. Finally, we signed all remaining shortcut files, as described in ShortcutsBench documentation, and loaded them into the macOS Sonoma Shortcuts app, where unsigned shortcuts cannot be imported. We tested these shortcuts by running them individually, as loading thousands of shortcuts at once would cause the app to crash. To facilitate evaluating on macOS Sonoma, we added the required apps to the Applications folder, allowing the Shortcuts app to capture and execute the corresponding shortcut actions. Note that some iOS-only apps require specific modifications (known as Clutch) to run on macOS. Throughout this process, we manually excluded empty or low-quality shortcuts, resulting in the final dataset of 7,627 shortcuts as described in our paper.

   For details on how we ensured the quality of queries, we combine the response to this question with the Question 2 you mentioned.

   If you have any further questions regarding the details, please feel free to ask. We would be happy to assist you.

2. In section 3.2, the paper describes using GPT-4o to simulate user queries. However, it would be helpful to include the steps taken to verify the correctness and ensure the diversity of these user queries.

To ensure the accuracy of our constructed queries, we employ two main strategies:

1. As detailed in Section 3.2, we leverage a more advanced model, GPT-4o, for query construction. GPT-4o was selected following a preliminary experiment comparing GPT-3.5 (active at the time of evaluating), GPT-4o, and Gemini-1.5-Pro across 100 queries. Human evaluators scored GPT-4o-generated queries highest, outperforming the other two models. GPT-4o effectively captures required parameters and provides a clear query description, meeting our criteria in 94 out of 100 cases. In contrast, GPT-3.5 often missed specific parameters, and Gemini-1.5-Pro performed slightly below GPT-4o.

   Although we cannot guarantee all 7,627 queries precisely meet our requirements, we consider our approach reasonable. Similar work, such as ToolLLM, was published in ICLR '24 with Spotlight, using GPT for extensive query and action sequence generation without guaranteeing full accuracy.

2. As described in Section 3.2, our query data includes a range of information—not only API data but also action sequences and functional descriptions provided by humans for each shortcut command. For the 100 queries generated, 94 met our manual verification criteria, requiring that the query adequately describes the shortcut’s function and integrates both primitive and enum data type parameters necessary for API calls. These parameters were extracted from the full action sequences, and the template we used is available in Appendix A.2.

Regarding diversity, we did not actively introduce diversity into our queries; instead, the diversity of our dataset derives from the shortcuts themselves. Each shortcut corresponds to a unique constructed query. As mentioned in “Further Processing” in Section 3.3 (or Appendix A.3), we categorized our shortcuts into 8 categories, following the classification method used in the Apple App Store, which further reflects the diversity of our dataset.

If you have any additional questions about the details, please don't hesitate to reach out. We would be glad to provide further clarification.

3. The evaluation primarily features selected proprietary models like GPT-4o and Gemini. For open-source LLMs, it mainly compares with general LLMs such as Qwen-2-70B and LLaMA-3-70B. Considering comparisons with more robust, specifically developed AI Agent models such as AgentLM (70B from https://github.com/THUDM/AgentTuning) and xLAM (8x7b or 8x22b from https://github.com/SalesforceAIResearch/xLAM) could provide more insights.

In our experimental design, we categorized the LLMs to be evaluated from both closed-source and open-source perspectives. Within each category, we selected models that are considered to have varying reasoning capabilities, which we believe is a reasonable classification. Moreover, the LLMs we evaluated surpass existing works in both quantity and diversity, as shown in Table 2:

Moreover, closed-source language models such as Gemini-1.5-Pro, Deepseek-2-Chat, Deepseek-2-Coder, Gemini-1.5-Flash, GPT-4o-Mini, and ChatGLM-4-Air we used have been specifically fine-tuned for function calling.

ShortcutsBench differs significantly from AgentLM/xLAM. The goal of ShortcutsBench is to provide a benchmark that is sufficiently challenging, rich, and distinctive, aligning more closely with AgentBench and MetaTool. I believe the experiments conducted within ShortcutsBench have effectively validated the problems mentioned in the Introduction. In contrast, AgentLM and xLAM focus on specialized fine-tuning of language models to enhance their API-calling capabilities in specific domains. It is entirely feasible to use the APIs provided by ShortcutsBench in conjunction with the data generation methods of AgentLM/xLAM to generate data that improves a language model’s performance in specific areas.

Despite this, we have fully accounted for the potential evaluation and analysis of additional models. Once relevant models are deployed, evaluating results can be obtained with a single command using Python's openai package. Detailed instructions on how to run our code for new model evaluations are provided in our code and data repository, accessible via this link: https://anonymous.4open.science/r/ShortcutsBench/experiments/README.md. You only need to make minor modifications to a bash script, such as https://anonymous.4open.science/r/ShortcutsBench/experiments/restart_all_experiments_llama_3_72b.sh, and add the model in https://anonymous.4open.science/r/ShortcutsBench/experiments/all_experiments.py. This will allow you to conduct the evaluation with one command. Our code is designed to handle network errors automatically, reattempting execution until all results are successfully obtained.

If you have any other questions or need further clarification on the details, please feel free to let us know. We’d be happy to assist you.

4. The source of the reported numbers in Table 1 could be more clearly specified. There is uncertainty regarding whether models like Qwen-2.5-7B and LLaMA-3-8B consistently achieve more than 90% on ToolBench and over 80% on ToolLLM. Furthermore, given that MetaTool (https://github.com/HowieHwong/MetaTool) does not provide comprehensive details or code for model evaluation and metrics, more information is necessary to verify the accuracy of the table data.

In the second paragraph of the Introduction, we mention, “Our evaluation of these less intelligent LLMs on 3 representative existing benchmarks/datasets,” meaning that the results in Table 1 are from our own evaluation. We evaluated the API selection performance on 3 representative datasets across various smaller-scale language models, which are generally considered to have lower reasoning capabilities. All data, code, and results used in these tests are fully open-sourced and available at https://anonymous.4open.science/r/ShortcutsBench/pre_experiments/README.md, where you can review them to verify the source of the results.

Regarding your comment that “MetaTool does not provide comprehensive details or code for model evaluation and metrics”, this does not impact our experiments. We used https://github.com/HowieHwong/MetaTool/blob/master/dataset/plugin_des.json as the tool description and selected https://github.com/HowieHwong/MetaTool/blob/master/dataset/data/all_clean_data.csv as the query data, evaluating only the accuracy of API selection.

Furthermore, to ensure a fair comparison, we used the same prompt template for testing as described in Section 4.

Thank you once again for your valuable feedback and insights. I truly hope that our responses above have helped clarify any concerns you may have. We would be very grateful if you could carefully consider your final rating (out of 10). Please don’t hesitate to reach out with any additional questions or comments—I would be more than happy to assist further.

Your concerns regarding Weaknesses 1 and 2, which primarily focus on the dataset construction process, have been carefully considered. We have implemented three additional steps to ensure the quality of action sequences and two additional steps to enhance the quality of user queries. For detailed explanations, please refer to our specific responses. To ensure transparency and accessibility, we have made all processes, including methods, details, and logs, available at: https://anonymous.4open.science/r/ShortcutsBench/.

For Weakness 4, regarding the source of results in Table 1, we have open-sourced the complete dataset, methods, and operational instructions used during testing at: https://anonymous.4open.science/r/ShortcutsBench/pre_experiments/. You are welcome to review the relevant results or reproduce them yourself. For Weakness 3, concerning additional experiments on 10 Agents, please refer to our detailed responses.

We sincerely appreciate your positive evaluation of the paper’s Soundness (4/4), Presentation (4/4), and Contribution (3/4). ShortcutsBench approaches the topic from a previously unexplored perspective, constructing a pipeline to extract high-quality data from existing Digital Automation Platforms as a benchmark dataset. This benchmark represents a challenging end-to-end process for API-based agent tasks, including API selection, parameter filling, and awareness for requesting missing information. It also uncovers many intriguing findings about API-based agents that we believe add significant value.

If you have any further concerns that might influence the final score, please feel free to share them. We would be more than happy to provide additional clarification.

Thank you for your valuable feedback and insights. I hope our responses have addressed your concerns. With the Discussion Period ending on November 26, I am eager to ensure all your questions are fully resolved. We would greatly appreciate it if you could consider your final rating (out of 10). Please don’t hesitate to reach out with any further questions or comments—I’m here and ready to assist.

Thank you for your feedback. We have carefully addressed all your comments in the revised PDF and look forward to hearing your response.

Specifically:

Concern 1 & Concern 2:

In Section 3.2 (Query and action sequence), we added descriptions of our efforts to ensure the quality of queries, with more detailed explanations provided in Appendices A.1 and A.2:

• Descriptions of efforts to maintain shortcut usability are included in the first paragraph.
• Efforts to ensure query accuracy are highlighted in the final paragraph.

You can also review the complete processing logs and records, which are open-sourced at https://anonymous.4open.science/r/ShortcutsBench.

Concern 3:

We added an introduction and comparison of AgentLM and xLAM in Section 4.1:

“We did not compare with specialized API-calling fine-tuned LLMs like AgentLM or xLAM for two reasons. First, our selected models already cover a range of intelligence levels, including closed-source models fine-tuned for API-calling tasks. Second, our goal is to provide a benchmark that is challenging, rich, and distinctive, which has been validated under the current setup. While AgentLM and xLAM focus on fine-tuning LLMs for API usage in specific domains, the APIs and methods in ShortcutsBench could be combined with their approaches to generate data for enhancing performance in targeted areas.”

Concern 4:

The sentence in the second paragraph of the Introduction, “Our evaluation on API selection of these less intelligent LLMs…,” indicates that this evaluation was conducted by us. You can further explore the fully open-sourced materials at https://anonymous.4open.science/r/ShortcutsBench/pre_experiments/README.md to review the experimental results.

Additional Notes:

As the discussion phase concludes on November 26 (AOE), we have implemented all the changes we committed to so far. However, there may still be some oversights, and we will use the remaining time to carefully review and refine our revisions.

We greatly appreciate your high evaluation of the paper’s Soundness (4/4), Presentation (4/4), and Contribution (3/4). ShortcutsBench offers a novel perspective, creating a benchmark dataset from Digital Automation Platforms to evaluate API-based agents on tasks like API selection, parameter filling, and handling missing information. It also reveals valuable insights into these agents’ performance.

We hope you will consider the contributions of ShortcutsBench when providing your final score. Please don’t hesitate to reach out with any questions—I’d be happy to assist.

We have placed the results of the 6 LLMs fine-tuned for agent scenarios alongside their corresponding base models in the same cell of the table for easy comparison. If the fine-tuned model outperforms the original, the corresponding result is highlighted in bold.

The details of how we processed the experimental results are as follows:

• We used gpt-4o-mini to convert outputs with non-standard formats into our standardized format for evaluation. We manually verified 50 randomly selected samples and confirmed that all conversions were correct. The conversion script can be found at: https://anonymous.4open.science/r/ShortcutsBench/data_for_agent_llm/reformat_output_2_correct_format.py. The reason for this processing step is that we observed poor instruction-following capabilities in llama-2-7b, llama-2-13b, agentlm-7b, and agentlm-13b (as described is Section setup in the initial version of ShortcutsBench), particularly when it came to generating outputs in the specific formats required by the prompts. For llama-2-7b, llama-2-13b, agentlm-7b, and agentlm-13b, this limitation may arise directly from the models’ inherent deficiencies in instruction-following capabilities. Incorrect output formats can lead to errors in result parsing, ultimately underestimating the experimental results.
• Since the context lengths of llama-2-7b, llama-2-13b, llama-2-70b, agentlm-7b, agentlm-13b, agentlm-70b, and lemur-70b-chat-v1 are limited to 4k tokens, it is challenging for these models to handle tasks in the more difficult range of (15,30]. Therefore, we only report results for the first 3 difficulty levels.
• The models we evaluated include several 70B models, which require 257GB of storage and long time for download. Additionally, these LLMs, fine-tuned for agent scenarios, do not have publicly available APIs. Even with models like Llama-2, there are few callable APIs, so we had to deploy them ourselves. Running 70B models demands significant resources, so we spent much time on the setup.

The detailed experimental results have been updated at: https://drive.google.com/drive/folders/1kYkqTWrukdb2nOMuK-O5TZo_cJ7SVF3E?usp=sharing

Files without the .after_format suffix represent the outputs prior to formatting.

• Performance on Recognition of Need for Input:

  • All evaluated LLMs performed exceptionally poorly in recognition of need input, significantly lagging behind the General LLMs mentioned in the paper of ShortcutsBench.
  • Compared to the performance of the base LLMs before fine-tuning, neither the AgentLM and Lemur series models fine-tuned on code nor the xLAM series models fine-tuned specifically for API-based agents showed any improvement in this aspect. This suggests the need for targeted fine-tuning specifically aimed at this capability.

We hope our experiments can address your concerns! If you have any further questions, please don’t hesitate to reach out to us. We would be delighted to assist you and look forward to hearing from you!

We have carefully addressed the remaining Concern 3 and look forward to your feedback. With the Discussion Period ending on December 2 (AoE) and only one day remaining, We want to ensure that all your questions have been fully resolved. We eagerly await your response.

Thank you for your response! We’re delighted to have your approval. Unfortunately, as the deadline for modifying the PDF has passed (November 27), we are unable to make any changes or re-upload it at this time. However, we assure you that once the option to modify the PDF becomes available, we will make every effort to integrate the results into both the main text and the appendix, with clear and prominent markings to indicate their location.

As the Discussion Period concludes today December 2 (AoE), we are eager to ensure that all questions have been thoroughly addressed. We hope our responses have sufficiently resolved your concerns. Your feedback is invaluable to us, and we would greatly appreciate it if you could take a moment to provide a final rating and share your thoughts.