FlowAgent: a New Paradigm for Workflow Agent

Thank you for your detailed review. We have revised the paper based on your feedback, with major updates highlighted in blue in the PDF.

Prompt-Based and Rule-Based Methods Example (Section 3 and Appendix A.1)

Thank you for the suggestion. We have revised the introduction and added references in the updated version, specifically in the second and third paragraphs. A more detailed discussion on prompt-based and rule-based workflows is now in Section 3.

Using the "Book Apartment Viewing" example from the STAR dataset, we illustrate the differences between prompt-based and rule-based methods. Below is the natural language workflow (prompt-based), which describes the process for the agent:

1. Ask the user for their name.
2. Request the name of the apartment the user is interested in.
3. Ask the user for the day they wish to view the apartment.
4. Ask for the start time of the viewing.
5. Inquire if the application fee has been paid.
6. Request any custom message the user might want to add.
7. Check the availability for the requested viewing.
8. If the viewing is available, inform the user and ask if they want to proceed with booking.
9. If the user says yes, book the viewing and inform them that the booking was successful.
10. If the user says no, ask again for the apartment name to modify search criteria.
11. If the viewing is not available, inform the user that the viewing is unavailable.
12. Finally, ask if there's anything else the user needs assistance with.

Natural language workflows can be imprecise, especially with nested branches like lines 7-11, which may confuse LLMs.

On the other hand, in a rule-based workflow, the entire process is rigidly controlled by software, and the LLM is only called as a node within this process. For example, in the Dify workflow [1], we need to configure a fixed sequence, such as using LLM_1 to complete the first step, "Ask the user for their name." Only after this step is completed can we move to the second step (LLM_2 to "Request the name of the apartment the user is interested in"). However, in reality, users may not respond according to this flow, such as saying, "Tell me first if I can book." In this case, the process gets stuck at the first step, and LLM_1 will continue to ask, "Please provide your name first".

Our proposed PDL syntax is as follows:

Name: Book Apartment Viewing
Desc: A process to assist users in booking a viewing for an apartment.

SLOTs:
  - name: RenterName
  - name: ApartmentName
  - name: Day
  - name: StartTimeHour
  - name: ApplicationFeePaid
  - name: CustomMessage
  - name: RequestType

APIs:
  - name: BookApartmentViewing
    request: [RenterName, ApartmentName, Day, StartTimeHour, ApplicationFeePaid, CustomMessage, RequestType]
    response: [Message]

ANSWERs:
  - name: ViewingAvailable
    desc: Inform the user that the viewing is available and ask if they want to proceed with booking.
    precondition: [BookApartmentViewing]
  - name: BookingSuccess
    desc: Inform the user that the booking was successful.
    precondition: [BookApartmentViewing]
  - name: ModifySearchCriteria
    desc: Ask the user for a different apartment name to modify search criteria.
    precondition: [BookApartmentViewing]
  - name: ViewingUnavailable
    desc: Inform the user that the viewing is unavailable.
    precondition: [BookApartmentViewing]
  - name: FurtherAssistance
    desc: Ask if there's anything else the user needs assistance with.
    precondition: [BookApartmentViewing]

PDL: |
  [RenterName] = ASKSLOT()
  [ApartmentName] = ASKSLOT()
  [Day] = ASKSLOT()
  [StartTimeHour] = ASKSLOT()
  [ApplicationFeePaid] = ASKSLOT()
  [CustomMessage] = ASKSLOT()
  [RequestType] = "CheckAvailability"
  [Message] = API_BookApartmentViewing([RenterName, ApartmentName, Day, StartTimeHour, ApplicationFeePaid, CustomMessage, RequestType])
  if Message == "Viewing Available":
      ANSWER_ViewingAvailable()
      UserWantsToProceed = ASKSLOT()
      if UserWantsToProceed == "Yes":
          [RequestType] = "BookViewing"
          [Message] = API_BookApartmentViewing([RenterName, ApartmentName, Day, StartTimeHour, ApplicationFeePaid, CustomMessage, RequestType])
          ANSWER_BookingSuccess()
      else:
          ANSWER_ModifySearchCriteria()
          [ApartmentName] = ASKSLOT()
  elif Message == "Viewing Unavailable":
      ANSWER_ViewingUnavailable()
  ANSWER_FurtherAssistance()

Compared with the formal two syntaxes, PDL has the following features:

1. Flexibility: Unlike rule-based workflows, PDL allows for more flexibility, addressing issues like user deviations.
2. Clarity: Compared to natural language workflows, PDL provides a clear structure, aiding LLMs in understanding the process.

[1] https://docs.dify.ai/guides/workflow

Thank you for your detailed review. We have revised the paper based on your feedback, with the main updates highlighted in blue in the PDF. Below, we address your concerns point by point:

Implementation Details of Controllers

We apologize for the initial lack of detail. We have added more information in Section 4.2 of the revised paper and included Algorithm 1 to demonstrate the operation of the controllers. Here’s a brief response to your questions:

1. The controllers are indeed based on deterministic logic, implemented in Python.
2. Each pre-decision controller checks the current state and organizes the results into the LLM’s prompt. Each post-decision controller checks the LLM-generated decisions to prevent undesired outcomes. We have provided pseudocode to illustrate this process in Algorithm 1.
3. For clarity, we have simplified the relevant code, shown in Python below.

for i in range(agent_max_retry):
    # Pre-decision controllers: traverse and use their messages to generate the prompt
    pre_messages = []
    for c in pre_controllers:
        pre_messages.append(c.process())
    prompt = gen_prompt(template, pre_messages, ...)
    # Agent generates action
    agent_output = agent.process(prompt, ...)
    # Post-decision controllers: traverse and check the agent's action
    if_pass = True
    for c in post_controllers:
        post_result = c.process(bot_output)
        if not post_result:
            if_pass = False
            break
    if if_pass: # Break the loop if all post-decision controllers pass
        break

Performance Metrics

Thank you for highlighting the importance of performance metrics. Here’s how we address your concerns:

1. LLM Performance: We focus on TTFT (Time To First Token) and Total Time (from request to final token). These metrics are influenced by the length of prompts and responses. The table below summarizes the average token counts and timing metrics for FlowAgent and three other syntaxes on our in-house dataset. Despite FlowAgent having longer prompts, the impact on TTFT is minimal. Since all approaches use the ReAct format for output, their output lengths are similar, resulting in comparable Total Time.
2. Controllers: As they are simple Python scripts, their computational cost is negligible compared to the LLM. In our scenarios, the number of graph nodes is typically less than 100, and the time to run checking algorithms is under 0.2ms, making it insignificant relative to LLM latency.

class NodeDependencyController(BaseController):
    def _build_graph(self, pdl:PDL):
        g = PDLGraph()
        for node in pdl.nodes:
            node = PDLNode(name=node["name"], preconditions=node.get("precondition", None))
            g.add_node(node)
        self.graph = g

    def pre_control(self) -> str:
        next_turn_invalid_apis = self.graph.get_invalid_node_names()
        if len(next_turn_invalid_apis) > 0:
            msg = f"APIs {next_turn_invalid_apis} dependency not satisfied: The precondition API has not been called."
        else:
            msg = ""
        return msg

    def post_check(self, bot_output:BotOutput) -> bool:
        next_node = bot_output.action
        node = self.graph.name2node[next_node]
        if node.precondition:
            for p in node.precondition:
                if not self.graph.name2node[p].is_activated:
                    add_system_message(f"Precondition check failed! {p} not activated for {next_node}!")
                    return False
        node.is_activated = True
        add_system_message(f"Check success! {next_node} activated!")
        return True

class PDLGraph:
    def get_invalid_node_names(self) -> List[str]:
        invalid_nodes = []
        for node in self.name2node.values():
            for precondition in node.precondition:
                if not self.name2node[precondition].is_activated:
                    invalid_nodes.append(node)
                    break
        return [node.name for node in invalid_nodes]

Thank you for your detailed review. We have revised the paper based on your feedback, with the main changes highlighted in blue in the PDF. Below, we address your concerns point by point:

Intuition and Effectiveness of PDL (revised Section 4.1)

Design Principles of PDL: We apologize for not clearly explaining the design principles of PDL in the initial submission. We have revised the relevant sections and provide a brief explanation here, focusing on compliance and flexibility:

1. Flexibility: To enhance the flexibility of workflow agents, we adopted a prompt-based approach for applying PDL (see Section 3.2). Unlike other workflow representations (natural language, code, flowchart), which cannot perceive the state of the workflow, PDL uses explicit, structured node definitions. This allows the workflow agent to detect its current state during execution.
2. Compliance: In PDL's node design, we introduced dependency nodes to define state relationships during workflow execution. Based on this, we designed a series of controllers to regulate agent behavior, enhancing compliance during the workflow process (see Sections 4.1 and 4.2).

PDL Performance in Complex Scenarios:

1. Empirical Analysis: In preliminary experiments, we conducted an empirical analysis comparing PDL with other workflow representations. We found that in more complex workflows (20+ APIs, 10+ branching nodes), FlowAgent using PDL syntax, along with control modules, executed tasks more accurately than those using natural language, code, or flowchart. Execution success rates increased from below 50% to over 65%.

2. Performance Metrics: We divided the SGD dataset workflows into three subsets based on the number of lines of workflow code: (0,15], (15,30], and (30,60], representing different complexity levels. The table below shows model performance using PDL and code syntax across these subsets. PDL shows more stable performance on complex workflows compared to code.

Performance of Different Model Sizes

Thank you for the suggestion. We have added results for different sizes of the Qwen2 series models. Using the same method to divide the SGD dataset, we evaluated the Task Progress metric. The results are shown below:

Observations:

1. Smaller models (0.5B, 1.5B) struggle with this task.
2. The 7B model partially understands the task but shows a significant performance gap compared to the 72B model, especially on more complex workflows.
3. Further analysis reveals issues with the model's ability to adhere to the specified format and accurately invoke tools when needed. We plan to optimize smaller models' performance regarding these issues in future experiments.

Experimental Details (Appendix A, B and C)

We have added detailed experimental information in the appendix of the updated paper:

1. Hyperparameters: For the GPT series, we used models gpt-4o-2024-05-13, gpt-4o-mini-2024-07-18, and gpt-4-turbo-2024-04-09, with a temperature set to 0.2 for stable outputs. For the Qwen2 series, we used the vllm framework, also with a temperature of 0.2.
2. Prompts: Key prompts used are included in the appendix, covering user simulation prompts during dataset construction, prompts for FlowAgent and baselines during inference, and adjusted prompts for evaluation based on FlowBench.
3. Dataset Examples: We added more dataset examples in the appendix, including a complete PDL workflow, a user profile example, and dialogue examples from the STAR and WikiHow datasets.

We hope these additions enhance the paper's clarity and reproducibility.

Figure 4

Based on your suggestion, we have revised Figure 4. (now is Figure 5 in Appendix E.3)

Thank you for your thorough review and insightful comments. We have revised the paper based on your feedback, with the main updates highlighted in blue in the PDF. Below, we address your major points and provide detailed responses to your questions.

Clarifications

C1: Using LLMs for User Simulation (Section 5.1)

This relates to Weakness 2 and Question 7. (see Appendix B.1 for generated conversation and Appendix C.2 for our pormpts)

We appreciate your observation regarding the use of LLMs for user simulation. We recognized that using LLMs to simulate users might not fully capture real-world interactions, as they tend to behave as overly cooperative users. To address this, we implemented the following strategies:

1. Diverse User Profiles: We generated diverse user profiles in stages. Initial personas were based on [1], with added details such as gender, job, interactive patterns (e.g., impatient or distracted), and user intentions.
2. Response Diversity: During interactions, we randomly introduced additional instructions (OOW requests, see #C2) based on these profiles.

See the "Conversation Example on STAR" below, where in line 16, the LLM-simulated user employs an intent_switching/detail-switching strategy by modifying the appointment time.

C2: OOW Setup and Data Generation

This relates to Questions 2, 3, and 11.

The OOW categories (Section 5.2) were derived from analyzing historical real-world user interactions with a traditional TOD system. We identified and summarized the following three categories:

1. Intent Switching: Sudden changes in request details or needs, such as slot modifications or cancellations.
2. Process Jumping: Deviating from the established workflow, including skipping steps or ignoring rules.
3. Irrelevant Answering: Providing unrelated responses, such as topic shifts or counter-questions.

OOW data generation involves two modes:

• Turn-level: We use a single LLM to generate a complete dialogue ("script" mode in [0]), randomly truncate it, introduce OOW queries, and continue the conversation to create reference sessions.
• Session-level: LLMs simulate users ("agent" mode in [0]), as discussed in #C1.

C3: WikiHow Setup and Experiments (Section 6.5 and Appendix E.3)

This relates to Weakness 3 and Question 8.

Our extension to WikiHow stems from a business need to handle user-uploaded product manuals for QA, while respecting privacy. We tested on randomly selected 20 pages from four categories: Cars & Other Vehicles, Computers and Electronics, Finance and Business, and Food and Entertaining [3]. We converted these into PDL format and conducted preliminary experiments. Manual testing showed that models could respond accurately to user queries under PDL syntax (see "Conversation Example on WikiHow"). However, we acknowledge the lack of established benchmarks for "workflow-based QA" and are working on building a standard benmark and an interaction environment.

Regarding Weakness 3, we note:

1. No pre_conditions were set for WikiHow as LLMs do not invoke tools in this setting, avoiding the issue you mentioned.
2. Defining pre_conditions for complex scenarios is challenging, and we plan to explore model performance in sparse DAG settings to test PDL robustness. (See the "Intuition and Effectiveness of PDL" section of our response to #mPiU)

...
BOT: Great, Alex. What’s the name of the apartment you’re interested in?
USER: Maple Apartments.
BOT: When would you like to view the Maple Apartments?
USER: I’d like to do it on Friday.
BOT: What time on Friday would you prefer for the viewing?
USER: How about 3 PM?
BOT: Have you paid the application fee?
USER: Yes, I have.
BOT: Would you like to add any custom message for the viewing?
USER: No, that's fine.
BOT: <Call API> book_apartment_viewing({'RenterName': 'Alex', 'Name': 'Maple Apartments', 'Day': 'Friday', 'StartTimeHour': '15', 'ApplicationFeePaid': 'Yes', 'Message': '', 'RequestType': 'CheckAvailability'})
SYSTEM: {'Status': 'Available'}
BOT: The viewing is available. Would you like to proceed with booking?
USER: Actually, I might need to change the time. Can we do 4 PM instead?
    (OOW type) intent_switching/detail-switching
BOT: <Call API> book_apartment_viewing({'RenterName': 'Alex', 'Name': 'Maple Apartments', 'Day': 'Friday', 'StartTimeHour': '16', 'ApplicationFeePaid': 'Yes', 'Message': '', 'RequestType': 'CheckAvailability'})
SYSTEM: {'Status': 'Available'}
BOT: The new time at 4 PM is also available. Shall I book it for you?
USER: Yes, please book it.

Name: Find the Publication Date of a Website
Desc: A guide to finding the publication date of a webpage 

SUBTASKs:
  - name: CheckPageAndURL
    steps:
      - Look underneath the headline of an article or blog post for the publication date.
      - Check the bottom of the web page for a copyright date.
      - See if the date is part of the URL.
      - Look at the timestamps on any comments to get an estimate of the publication date.
  - name: UseGoogleSearch
    ...
  - name: UseWaybackMachine
    ...
  - name: SearchSourceCode
    steps:
      - Right-click the web page and select `View Page Source`.
      - Press `Ctrl+F` (Windows) or `⌘ Command+F` (Mac).
      - Search for the term “date" or "published".
      - Look for the date listed in year-month-day order.

PDL: |
  [publication_date] = SUBTASK_CheckPageAndURL()
  if publication_date is None:
    [publication_date] = SUBTASK_UseGoogleSearch()
  if publication_date is None:
    [publication_date] = SUBTASK_UseWaybackMachine()
  if publication_date is None:
    [publication_date] = SUBTASK_SearchSourceCode()

U: How to find a website's publication date using code?
B: To find a website's publication date, you can inspect the page source and search for terms like "date" or "published" to locate the date in the metadata.
U: How to inspect the page source?
B: To inspect the page source, you can right-click the webpage and select "View Page Source" in your web browser.
U: I got "<meta property="article:published_time" content="2024-11-20T17:45:37.542Z"/><meta property="article:modified_time" content="2024-11-20T17:45:37.542Z"/><meta property="article:author" content="Ronan Farrow"/>"
B: Great! The `<meta>` tags you found contain the publication and modification dates for the article. The `article:published_time` shows when the article was first published: 2024-11-20.
...

Thank you for your detailed response and updated manuscript! It resolves some of my concerns about experimental setup. I still have concerns about this work.

• I believe that since the baselines have been automatically converted using GPT and the PDL is manually crafted, the evaluations are unfair.
• For future versions of this manuscript, the authors should add a scale to Figure 1 (c) plot. It is good for a product presentation, but the plot is not "scientific" without scale and metrics.
• It is still unclear to me how the LLM adds OOW nodes to the DAG.
• The authors perform "manual interaction" with the wikiHow agent, but they do not perform any experiment that suggests the broad applicability of FlowAgent to real-world workflow-based QA tasks. I highly suggest the authors run human studies for session-level evaluation (in addition to LLM-simulated users) and evaluating applicability of flowagent to real-world applications.
• I also have reproducibility concerns over the work. The authors provide no code or data which can be used to reproduce the results.