OKR-Agent: An Object and Key Results Driven Agent System with Hierarchical Self-Collaboration and Self-Evaluation

Thanks for your comments, below are the feedback of your concerns:

Q1 One of my major concerns is the limited technical novelty. Apart from the cute prompting, I did not see many method improvements from both technical and theoretical perspectives.

A1 We want to emphasize that the proposed method is not a simple prompt engineering. In fact, the complexity of the tasks we undertake exceeds the inference capabilities of existing Large Language Models (LLM). To avoid bottlenecks in LLM's performance, we have design the OKR based method, which allows the LLM to focus both on the global objectives and the details of the task. This approach enables the LLM to autonomously decompose tasks and evaluate the results of the decomposition, ensuring the accurate execution of target outcomes within the LLM's performance range.

Q2 Limited evaluation in a narrow domain with limited baselines. I would expect more complex open-world LLM-agent benchmarking results, and more language models to be tested. Currently, only simple comparisons with ChatGPT (not sure which model version as well) can not justify why the proposed prompting framework can be generalized to different models.

A2 Thanks for the suggestion and we will open-source our code and all experiments including intermediate results. Our framework can work with other LLMs, the main difference is the stabilities of the generation caused by capacities of different LLMs.

Q3 More analysis is needed on the agent objectives, key results, and evaluation criteria to provide insights into what is being learned.

A3 We offer additional ablation studies.

Q4 Agent coordination details are lacking - how are dependencies managed?

A4 The tasks for agents are automatically assigned by the Large Language Model (LLM) without any manual explicit assignment. The LLM autonomously decomposes the tasks based on their type and content, and allocates them to the appropriate agents.

Q5 Safety mechanisms to prevent unbounded generation of objectives/agents need to be addressed.

A5 The generation of objectives and agents is bounded by the evaluation, where large number of objectives could lead to a low score in the evaluation. We also set the overall hierarchical levels to further prevent the possible unbounded generation.

Q6 How does the performance scale with increasing levels of hierarchy? Is there a sweet spot?

A6 Yes. Through ablation experiments, we compared the generation effects of 1-3 levels of hierarchy. Subjectively, three layers are currently the most suitable, with more than three levels showing no significant improvement in effectiveness.

Q7 Can the approach work for very open-ended creative tasks without clear sub-objectives?

A7 There are two methods that can have an impact: 1) By using the Few-Shot approach to inform the LLM of the required use cases. 2) By fine-tuning the LLM.

Q8 Has the approach been tested on goal-driven tasks like process planning vs just content generation?

A8 In our paper, "Multi-day Travel Planning" is an example of goal-driven tasks. Due to the length constraints of the paper, we did not include another task: "Wedding Planning." If permitted, we will supplement this in the accompanying materials.

Q9 What techniques are used to elicit useful evaluation criteria from the agents?

A9 We enable the LLM to customize assessment methods based on task objectives and assign them to different agents, thereby selecting the best inference results. The effectiveness would be further enhanced with the introduction of human interactive evaluation.

Q10 How sensitive is performance to the quality of the initial user prompt? How difficult is it to construct such initial prompts from ambiguous and general tasks?

A10 We have adopted a method that allows the LLM to independently understand and decompose user input tasks (for example, providing the LLM with some cases to guide it in imitating methods that can successfully decompose objectives). Therefore, our algorithm is currently not sensitive to user input; the LLM will automatically analyze user objectives. Of course, more detailed descriptions lead to more effective results.

Thanks for your comments, below are the feedback of your concerns:

Q1 Limited Comparative Benchmarking / Heavy Reliance on Subjective Evaluation / Lack of Evaluation Depth and Diversity

A1 While we acknowledge the limited objective evaluations in our current work focusing on creative content generation, the assessment primarily relies on the subjective judgment of industry experts, which is a valid approach for this context. To align more closely with expert expectations, we have employed various engineering methods to enhance inference outcomes. These include using Few-Shot learning to improve the initial user input to OKR information conversion success rate, employing multi-round generation and assessment voting for optimal results, and integrating global and local information to enhance current inference outcomes. Additionally, our ablation studies on OKR generation outcomes further validate the effectiveness of our methods.

Q2 Insufficient Ablation Studies and Lack of Error Analysis

A2 We have conducted several ablation studies, we will include them in main text or the supplementary materials based on the page limits.

Q3 No Discussion on Model Robustness and Generalizability

A3 Due to the length constraints, we experimented two different types of experiments, including creative writing and goal planning in the submission. We will add more varied types of experiments if space allows.

Q4 How does the OKR-Agent manage the computational complexity and resource allocation when scaling to more complex tasks and a larger number of agents?

A4 We decompose OKRs into 3 hierarchical levels, and in hierarchical level, the objectives include 3-5 keys, which can address most tasks we tested.

Q5 In cases of ambiguous or poorly defined tasks, what mechanisms does the OKR-Agent use to ensure effective decomposition into Objectives and Key Results? Could you provide insights into how the OKR-Agent might be adapted for use in highly specialized or technical domains that differ from those tested?

A5 We have adopted the method of allowing the LLM to independently understand and decompose user input tasks (for example, providing the LLM with some cases to guide it to imitate methods that can successfully decompose objectives). For the key results generated by the Agent, we implemented "self-assessment" and "global assessment," hoping to enhance the effectiveness of the agent's execution.

Q6 Are there any plans to conduct ablation studies to pinpoint the most critical components of the OKR-Agent's architecture for its task-solving performance?

A6 We have added ablation experiments for the OKR generation stage. In these experiments, the results demonstrate that the customization of OKR objectives and the number of layers in OKR execution have a significant impact on the outcomes generated by the algorithm.

Q7 How does the OKR-Agent interface with external data sources, and what strategies are implemented to ensure the integrity and applicability of this external data?

A7 There are two methods that can have an impact: 1) By using the Few-Shot approach to inform the LLM of the required use cases. 2) By fine-tuning the LLM.

Thanks for your comments, below are the feedback of your concerns:

Q1. The paper does not include enough details to be able to reproduce the work ...

A1. Thanks for the suggestion and we will open-source our code and all experiments including intermediate results. Our framework can work with other LLMs, the main difference is the stabilities of the generation caused by capacities of different LLMs.

Q2. The examples shown do not use quantitative assessments ...%. There are human subjects evaluations but no indication of which differences are statistically significant, so saying the method proposed outperforms other methods does not seem to be strongly supported by the results presented.

A2. Yes, our current primary focus is creative content generation. The evaluation of such tasks primarily relies on the subjective assessment of industry experts. We will provide more ablation experiments on the OKR generation outcomes, which demonstrate their impact on the effectiveness of generation.

Thanks for your comments, below are the feedback of your concerns:

Q1: The biggest draw back of this paper is that, it is just not clearly written. I will be great to include code examples and intermediate solving results in the experiment session to help understanding.

A1: Thanks for the suggestion and we will open-source our code and all experiments including intermediate results.

Q2: If I understand correctly, for each key result the LLM has to be invoked at least multiple times during solution proposal and review. So overall this system seems costly and slow from an inference perspective. It will be great if the authors can compare with other methods from this perspective.

A2: Yes, there is always a trade-off between the cost of inference and the overall quality of solutions. Our inference process is divided into two stages: OKR generation and Agent working stage. In the OKR generation phase, well-designed prompts enable the LLM to effectively improve the OKR inference results, generating reasonable keywords. This process relies on the model's inference capability rather than content generation capacity, so it does not require excessive Token consumption. In the Agent working stage, the overall cost is directly proportional to the number of tokens. Here, we validated several different LLM base models (including gpt-3.5-turbo, gpt-4, flan-alpaca-xl-3B, etc.) under the constraints of OKRs, all capable of producing satisfactory content generation results. This observation also helps in reducing generation costs. Based on GPT-API's pricing, the cost for a single creative writing generation is about $0.5 to $0.6, and even lower, less than $0.1, if using local models.

Q3: The proposed system will decompose an task into many hierarchies. So who is deciding the number of hierarchies needed? From algorithm 1 it looks like an hyper parameter.

A3: We set the number of hierarchies empirically. Based on our experiments, 3 layers are sufficient for the tasks we tested and there is no significant improvement with more than 3 layers. We will provide this ablation in the revision.

Q4: Since the task will be broken into many objectives, are there causal or temporal links between different objectives? For example, how objective 2 can be solved should depend on objective 1's solution. If so, how can the different agents coordinate to solve the objectives in the correct order? From algorithm 2 it seems that each agent is just independently updating a portion of the final answer.

A4: There is no explicit link between objects although there could be shared components of task for different objects. Both the tasks and agents are automatically assigned by the Large Language Model (LLM) without any manual assignment.

Q5: What if one agent cannot give a satisfying answer? What will happen to the whole task? Are there any "replannings" where certain objectives should be removed/reconsidered?

A5: Once OKRs are generated, they will still fixed. We use an assessment and adjustment process during the OKRs creation stage to ensure the OKRs are correct and necessary. After an agent completes the execution of OKRs, the outcomes produced will be "evaluated" and "revised" again according to the OKR objectives.