W1. Clarification of evolving environmental states with GPT models.

The primary goal of integrating evolving environmental states with GPT models is to generate more coherent and logically flowing events, rather than achieving precise state change modeling. These states act as contextual reminders within our environmental simulations, aiding the model in crafting new events. The core purpose of these states is to introduce variability in the events generated by the model, enabling it to anticipate "what might happen next" or consider "what unusual events could occur."

One of the main challenges in our approach is not the model's inability to accurately simulate an environment, but rather its tendency to create overly perfect or neutral scenarios. The real world is in chaos. Users often perform seemingly random actions, such as clicking aimlessly, moving the mouse erratically, or hesitating over words they can't spell, leading to a plethora of confusing actions. Current models struggle to replicate this unpredictability.

By incorporating detailed states and aligning generated events with real-world occurrences, we've found that our simulation environment can introduce a degree of uncertainty and reality. For instance:

{"time": "10-18 09:17:40", "content": "User inputs 'ctrl_l' with mouse interactions, clicks in an application."}
{"time": "09-17 15:46:53", "content": "The user double-checks the warranty information for the shortlisted devices before making a purchase."}
{"time": "10-05 14:26:56", "content": "The user uses 'ctrl+c' to copy data about material sustainability from EcoChic\u2019s website for documentation."}

The final performance gain also demonstrates the effectiveness of our method.

Besides, you mentioned that:

Additionally, this reliance on LLMs could potentially make the benchmark too easy, as they may only be capable of predicting changes that they fully understand and are familiar with, which are also easily solvable by other LLMs.

We suggest that this concern comes from a crucial misunderstanding of our dataset. As the test set is totally collected from real-world annotators, they are not easily solvable by LLMs.

Q1. Turn text-based simulation into a real interactive environment.

As described in W1, when comes to generating data to train the agent model, we suggest that the major problem is to simulate user activities rather than collect real environment states. Turning text-based simulation into a real interactive environment can not improve the uncertainty or randomness of simulated user activities. However, it will indeed complicate the setup of diverse application scenarios and greatly slow down the data generation. We also argue that this can improve the reality of events generated and might be a good idea in certain situations.

W3 Q2. More baselines to demonstrate the effectiveness of the data pipelines.

We've extended our baseline settings to remove the Chain-of-Thought prompt. We also present more experiments about enhancing the proactiveness of agents with the reward model. Here are added experiments:

Hopes this can solve your concerns. If you have any additional questions or require further clarification, please feel free to reach out.

W2. Statics and examples of the dataset.

Your summary mentioned that:

Through this data pipeline, combined with some minor human annotations, the authors curated 1,760 data entries, using 120 of them for testing.

Here, we would also like to clarify some misunderstandings regarding our dataset and the extent of human annotations involved. Our dataset comprises four parts:

1. Dataset for the Agent Model

• Train set: 6,790 entries (Synthetic)
• Test set: 233 entries (Human-Collected)

2. Dataset for the Reward Model

• Train set: 1,640 entries (Human-Annotated)
• Test set: 120 entries (Human-Annotated)

It's important to note that both the test set for the agent model and the entire dataset for the reward model are sourced from real-world data and meticulously annotated by our human annotators. This process resulted in nearly 2,000 human-annotated entries.

Furthermore, each entry in the reward model dataset is annotated at least three times, totaling nearly 6,000 annotation labels. This level of annotation effort is substantial and goes beyond what might be considered "minor human annotations".

We hope this clarification accurately reflects the scope of our work and addresses your concerns. We‘ve also updated Table 1 in our paper to better clarify this.

:) Thank you for understanding.

Here are more examples from our dataset that can help you understand our dataset (we've also updated the appendix):

Events Data
-----------
{"time": "1717377997.0","event": "The user starts working on a file 'CheckName.rb' within a Ruby project in Visual Studio Code."}
{"time": "1717342826.292","event": "The user started browsing the web."}
{"time": "1717335993.089","event": "The user scrolls multiple times on a web page."}

Annotated Data for Reward Model
-------------------------------
{
    "obs": [
        {
            "time": "1717338232.283",
            "event": "The user types 'badrandResearch.m' in Visual Studio Code."
        },
        {
            "time": "1717338245.419",
            "event": "The user types a Markdown entry about research on sustainable branding in Visual Studio Code."
        },
        {
            "time": "1717338245.419",
            "event": "The user works on the file 'brandResearch.md' in the project 'SustainableBrandResearch' in Visual Studio Code."
        },
        //...
        {
            "time": "1717338752.214",
            "event": "The user interacts with Katla's website, viewing eco-friendly loungewear products made by the sea."
        },
        {
            "time": "1717338804.867",
            "event": "The user writes Markdown notes in Visual Studio Code about Katla's product pricing."
        }
    ],
    "pred_task": null,
    "valid": false,
    "help_needed": true,
    "annotation": [
        false,
        false,
        false
    ],
    "category": "Missed-Need (MN)"
}


Training Data for Reward Model
------------------------------
{System prompt for the reward model, please refer to the appendix.}

User:
{
    "Observations (Time Ascending)": [
        {
            "time": "1717337616.208",
            "event": "The user switched back to Visual Studio Code and typed insights related to digital payment systems improving public transportation and reducing the need for private car usage."
        },
        {
            "time": "1717337282.329",
            "event": "The user moved the mouse and clicked the left button within Visual Studio Code."
        }
    ],
    "Proposed Task": "Provide additional research articles on digital payment systems and their impact on traffic management",
    "Instruction": "Now give your judgment. You should complete the reasoning process in the first person."
}

Assistant:
{
    "thought": "I have already conducted research on digital payment systems and their impact on traffic management. I have also taken notes and compiled relevant insights in my markdown file. The proposed task seems redundant as I have already made progress on this topic. I have found relevant information through Google search results and have reflected on the strategies to improve public traffic, including digital payment systems. Therefore, I do not see the need for additional research articles on this topic.",
    "judgment": "rejected"
}

Test Data for Agent Model
-------------------------
{System prompt for the agent model, please refer to the appendix.}

User:
{
    "Instructions": "Now analyze the history events and provide a task if you think the user needs your help.",
    "Observation": "[placeholder]"
}

Possible Agent Response:
{
    "Purpose": "The user is currently interacting with a web application.", 
    "Thoughts": "Based on the absence of a specific action, the user might be navigating or searching for content.", 
    "Proactive Task": null, 
    "Response": "I see you're actively using the web application. If you need help finding something specific or have any questions, feel free to ask!" 
}

Based on your explanation, it appears that the model's tendency to 'overly assist' is the primary factor leading to the gain, rather than the effect of the proposed framework? In other words, it is "good for misconceived reasons".

We are ready to address any remaining misunderstandings or confusion you may have. We would like to know if our explanation helped resolve your concerns regarding our evaluation and the metrics used. If there are still any unresolved issues, we kindly ask for your feedback.

As the rebuttal deadline is approaching, we would greatly appreciate any further suggestions, concerns, or comments at your earliest convenience. We are eager to continue the discussion and clarify any unsolved points to ensure all matters are fully addressed.

Thank you again for your time and consideration.

We truly appreciate the time and effort you've dedicated to reviewing our paper and sharing your insights.

As the rebuttal deadline approaches, we wanted to touch base and see if there are any lingering questions or concerns we can address. We’re eager to continue the conversation and would love to hear any additional thoughts or suggestions you might have.

Thank you once again for your valuable feedback. We look forward to your response!

W3. Clarification of evolving environmental states with GPT models.

The primary goal of integrating evolving environmental states with GPT models is to generate more coherent and logically flowing events, rather than achieving precise state change modeling. These states act as contextual reminders within our environmental simulations, aiding the model in crafting new events. The core purpose of these states is to introduce variability in the events generated by the model, enabling it to anticipate "what might happen next" or consider "what unusual events could occur."

One of the main challenges in our approach is not the model's inability to accurately simulate an environment, but rather its tendency to create overly perfect or neutral scenarios. The real world is in chaos. Users often perform seemingly random actions, such as clicking aimlessly, moving the mouse erratically, or hesitating over words they can't spell, leading to a plethora of confusing actions. Current models struggle to replicate this unpredictability.

By incorporating detailed states and aligning generated events with real-world occurrences, we've found that our simulation environment can introduce a degree of uncertainty and reality. For instance:

{"time": "10-18 09:17:40", "content": "User inputs 'ctrl_l' with mouse interactions, clicks in an application."}
{"time": "09-17 15:46:53", "content": "The user double-checks the warranty information for the shortlisted devices before making a purchase."}
{"time": "10-05 14:26:56", "content": "The user uses 'ctrl+c' to copy data about material sustainability from EcoChic\u2019s website for documentation."}

The final performance gain also demonstrates the effectiveness of our method.

Last But Not Related To The Question: from our experience of trying to obtain events that simulate the real world, we assume that the final false alarm could be further reduced by incorporating more real-world chaos events into our training data. We also believe how to obtain random and uncertainty from models is a critical question to study in the future.

Q2. How to tune proactiveness according to user preferences?

In our current settings, we take the most conservative approach that only proposes tasks most people like to have, which might not be optimal in some settings. However, we can tune the agent's proactiveness by simply adding user feedback into historical events, like rejecting the proposal or instructions to the agent. Future studies should also consider these complex interactions between agents and users to automatically tune the proactiveness of the agent.

Q3. Ideas about how to reduce false alarms.

This can be done in multiple ways:

1. Trying to collect more randomness from real users and improve the data generation pipeline could help. As we mentioned before, the uncertainty and randomness of user activities or environment can not be perfectly modeled by current models, which causes a relatively high false alarm ratio for fine-tuned models.
2. Simply collecting more real-world annotations for training agent models could also help.
3. Allowing the model to obtain more in-content samples of when they should or should not propose a query could also help.

Q4. Dataset Statics: how much real/synthetic data?

Our dataset comprises four parts:

1. Dataset for the Agent Model

• Train set: 6,790 entries (Synthetic)
• Test set: 233 entries (Human-Collected)

2. Dataset for the Reward Model

• Train set: 1,640 entries (Human-Annotated)
• Test set: 120 entries (Human-Annotated)

In total, there are around 700 real-world collected events and around 7,200 synthetic events.

Q5. Context windows for prediction.

In our vanilla settings, we do not limit the number of events used for prediction. In our live demo, to improve responsiveness, we manually limit the number of events to 20.

We also provide more performance comparisons for different settings here:

The tables demonstrate that models might suffer a performance drop if not given sufficient contexts, for longer event context might let the model better capture the personality of the user.

Q6. Clarify proactive interactions.

R. Yes, judgment is the only thing done by humans, and actions are conducted by agent models like GPT-4o. During our proactive interactions, the proactive agent proposes a task that the user might need. Then the user selects to accept or reject the proposal. If accepted, the proactive agent conducts actions to finish the proposed task. If rejected, the proactive agent does nothing. Hope this clarification can solve your question.

W1. Discuss the improvement made and further impact.

The point of the comparison is to show that even a small model can proactively predict the user's intention after properly being finetuned rather than slightly outperforming GPT-4o. It suggests the effectiveness of our pipeline and training method.

From that, the future impact should be proposing a way of constructing data and a new training method for the proactiveness of the agent, filling the lack of research field of Proactive Paradigm in this area for future research. The performance of the agent is not the main impact.

W2. User studies to support the claims of improved proactiveness.

To clarify, our test set is collected independently from real-world users by tracking their actions from coding and writing scenarios, which is different from the training data generated from our environment gym. Our annotators annotate the events when they need help from the agent, and our reward model shows a significant agreement with our annotators. Due to that, what the gent proposed validated by the reward model should also be beneficial for the real user.

Further user study is obviously a better way to demonstrate the superiority. But considering that user action is uncontrollable (and always varies between different users), the user study is very hard to conduct. We are indeed trying to build such a study, but it might not be available soon.

W3. Performance of Qwen2-7B and LLaMA-3.1-8B

Thank you for your inquiry regarding the performance differences between Qwen2-7B and LLaMA-3.1-8B. We find our original downloaded LLaMA-3.1-8B model contains bugs that impact the final performance. We've updated the performance of LLaMA-3.1-8B to 55.06%, which is still a bit lower than Qwen2-7B.

Upon further careful examination, we observed that Qwen2-7B provides more concrete and actionable responses, leading to higher accuracy and fewer false alarms. Our goal was to design tasks that are clear and user-friendly, and it appears that LLaMA-3.1-8B does not consistently meet this objective.

Here are some comparisons to illustrate this point:

Event Timestamp: 1717335893.215
Event Content: The user clicks on the web page, types 'swift ui list', and presses 'Enter'.

Qwen2-7B proposal: Provide a brief explanation or guide on Swift UI List.
LLaMA-3.1-8B Proposal: Explore possible tasks related to either Python or Swift UI list based on the user's interest.


Analysis: LLaMA-3.1-8B's response is abstract and lacks actionable guidance, unlike Qwen2-7B, which offers a direct and relevant task.

--------
Event Timestamp: 1717338374.868
Event Content: The user scrolls through the search results for 'sustainable fashion brands' on Google.

Qwen2-7B proposal: Provide a brief summary of the top sustainable fashion brands from the search results.
LLaMA-3.1-8B Proposal: Help the user create a knowledge graph using Python to visualize and connect their research on sustainable branding and fashion.

Analysis: LLaMA-3.1-8B introduces unnecessary complexity with an abstract task, whereas Qwen2-7B remains focused and relevant.

--------
Event Timestamp:1717494492.445
Event Content: The user scrolls and clicks on a research report page about biodegradable cups.

Qwen2-7B proposal: Provide a summary of the key points from the research report about biodegradable cups, along with suggestions for further reading or related studies.
LLaMA-3.1-8B Proposal: Create a new python project in Visual Studio Code and assist with setting up the environment.

Analysis: LLaMA-3.1-8B misunderstands the user's action, resulting in an irrelevant proposal, while Qwen2-7B delivers a precise and helpful task.

Hope this can solve your concerns.

L1. Discuss resource consumption for the proactive agent.

We appreciate your insight on the resource consumption of the agent. Resource consumption is an important engineering question, although it is not within the scope of this paper. In this paper, we establish our experiments by running remote models, which can already achieve satisfying latency. Running locally or remotely does not impact the performance of proactiveness. As the services of LLMs get faster and cheaper, and more accessible to general humanity, we hold the belief that the transition from reactive and proactive will happen.

Moreover, while large models like GPT or Claude perform well generally, there is a trend to train small models to beat the large models on specific downstream tasks. Models like Gemma-4B, Qwen-2B, MiniCPM3-4B consume less but could perform well when properly finetuned, and the capability of the small models only grows. With more techniques, resource consumption will no longer be a concern in a few years.

L2. Real user evaluations.

The original human annotation is based on an events series, so from this point, the performance of the agent is already evaluated by a real human, it's just by replaying the event series rather than the ideal interactive evaluation process.

The difference in the user action and familiarity with the project will influence the content and results of the agent(E.g. Some users may have a slow typing rate, and the agent will focus on proposing and implementing the content, interacting with the agent will further slower the typing of the users.). In order to control the variance, we have to use the static data collected to train the reward model, by replaying we could obtain the consistent rate of different annotators and, furthermore make the reward model simulate a real user better.

Fortunately, we already developed more complicated software and demos suitable for different devices and platforms. These projects will help to enlarge our testing environment and lay a solid foundation for real, interactive, instant evaluation in the future. All these projects will be released soon.

M1. Predict multiple candidates with RM.

Of course. Thanks for suggesting implementing our ablation study. The result is also updated in our paper. The section Ablation Study is renamed as Performance Analysis.

In this table, predicting multiple candidates with rewarding (pred@3, w/ RM) achieves the best results for LLaMA-3.1-8B and GPT-4o-mini.

M2. Clarify the human annotation details.

We obtain "cosine distance" by first embedding the predictions with Text-Ada-Embedding and calculating the cosine distance on the vectors. The consistency rate refers to the agreement ratio between human annotators. The agreement ratio on the train set for the reward model is around 77%. Thanks for your detailed feedback again! We've fixed it in our latest version.

M3. Usage of the GPT-4o explanation for the user judgment.

The GPT-4o explanation is used as the CoT mechanism. During the training of the reward model, the explanation of the GPT-4o is used as a chain-of-thought for the judgment to help the reward model simulate a specific user better and perform better.

M4. Memory mechanism for the proactive agent.

We adapt in-context short-term memory mechanisms that directly show the history since it is easy to implement and the memory history is linear, which aligns with the temporal event sequence.

M5. The training settings of the reward model.

We conduct full-parameter finetuning. The detailed settings can be found in Section 4.1.

M6. Temperature Settings Explanation.

Yes, the temperature is the decoding temperature. We emphasize the temperature here to show the result of our experiment is stable. We are sorry for any confusion, and we've updated the relative part to better clarify our ideas.

M7. Well-defined description of what events and scenarios mean.

We're sorry for not making the definition clearer. To clarify, an event is a Time + Description of what happens. For example:

{
    "time": "1717377997.0",
    "event": "The user starts working on a file 'CheckName.rb' within a Ruby project in Visual Studio Code."
}

A scenario is a global review of what's going on. Specifically for our data generation settings, it contains environment descriptions, tasks that the user is performing, and user persona. We've updated related parts and provided more examples for clarity.

M8. Failure modes examples for comparison.

Thank you for the valuable question. Here we give more failure modes analysis (we've also updated the appendix to help a better understanding.):

Not A Task: The model does not give a real task at all.

Observation: {
"time": "1717335907.603",
"event": "The user switches to Visual Studio Code."
}
GPT-4o-mini Response: It seems you may be switching to Visual Studio Code for development. If you need any coding tips, resources, or help with Swift UI in VS Code, feel free to ask!

Under this event, the agent did not take the event as an action, but rather a call for chat. This chat is not a task we want.

Unspecific Task: The proposed task is non-executable due to a lack of a clear and detailed description.

Observation: {
"time": "1717329066.416",
"event": "The user performed a search for ’js fade in/out’ on
Bing."
}
GPT-4o-mini Response: Compile a list of JavaScript resources, tutorials, and documentation that cover advanced topics related to the user’s search.

Under this event, the model proposal is not clear (not pointing out the user's research content) and impractical (Compiling a list of resources, tutorials, and documentation, which is too much for the user and also out of the capability of the model).

Irrelevant Task: The proposed task is not related to the current context or meets the user's needs.

Observation: {
"time": "1717336090.982",
"event": "The user clicks, types ’list.swift’ in Visual Studio
Code."
}
GPT-4o Response: I can help by providing tips on how to efficiently navigate and manage files in Visual Studio Code or offer assistance with the content of ’list.swift’.

Under this event, the user created a new file in vscode. However, the agent provided a task about file management, which is irrelevant to the user's intention.

Unwanted Proposal: The user does not want to be interrupted.

Observation: {
"time": "1717329211.15",
"event": "The user opened ’http://127.0.0.1:5500/main.html’ in the web browser."
}
GPT-4o Response: Offer to help troubleshoot or verify the ’fade in fade out’ effect if it doesn’t work as expected.

Under this context, since whether the edition is successful or not is not clear, the agent should remain silent rather than propose troubleshooting currently.

If you have any additional questions or require further clarification, please feel free to reach out.

We sincerely appreciate your thoughtful feedback and suggestions on our work. We are eager to demonstrate that our work holds promise and greater value. To address the concerns you raised, we have provided clarifications on previously missing details, including performance analysis, datasets, memory mechanisms, training processes, and omitted concepts. Additionally, we have conducted new experiments on context windows and multiple tasks with the reward model, and we have shared our ideas on potential applications, resource consumption, and future improvements. The paper has been updated to incorporate these clarifications.

As the rebuttal deadline approaches, we would like to know if our responses have sufficiently addressed your concerns and questions. We welcome any further feedback or discussion to help us continue improving our work. Thank you for your time and consideration.

Apologies for the delay in responding to the rebuttal.

I think the authors have addressed most of my concerns, and provided clarifications wherever I had doubts.

I will retain my rating for the paper as I wanted to give them a 7, but I can only choose between 6 and 8.

Overall, I think this is an interesting paper and would love to see it pushed through.

W1 Q1. Interactive simulator or text-based simulator?

To clarify, we build an environment gym to obtain training data for our agent model only. It's an event-driven asynchronous text-based simulator, like Text-World[1] or ALF-World[2]. The user agent and proactive agent in the gym are connected by the event transmitting mechanism in the gym.

During running, the gym will dynamically update the environmental states based on the events generated by the user agent. The proactive agent monitors the behavior and states of the environment to produce proactive predictions.

1. Côté, Marc-Alexandre, et al. "Textworld: A learning environment for text-based games." Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7. Springer International Publishing, 2019.
2. Shridhar, Mohit, et al. "Alfworld: Aligning text and embodied environments for interactive learning." arXiv preprint arXiv:2010.03768 (2020).

W1 Q2. The four stages of the environment gym setup seem redundant and could be streamlined for clarity.

The four stages represent different perspectives of our data generation. To calibrate the event generated by the gym, we first collect real-world events in the user's environments to serve as seed events. Then to improve the consistency and detail of the simulation, we introduce the scenario generation stage to create scenarios with different environmental objects, like the operating system version, applications installed, the running apps, etc. After that, we iteratively generate new events from the gym and update the environment states in the event generation stage and state maintenance stage respectively.

Hope this clarification can solve your question. A more detailed explanation of the possible redundant would be appreciated to better locate the problem.

W1 Q3. More information about the dataset used for reward model assessment.

To collect the data, we develop annotation tools (VSCode add-ons, browser extensions, general user input monitoring, system monitoring, interactive panels) to collect real user activities and generate proactive prediction with 9 models (Cluade-3-Sonnet-20240229, Claude-3.5-Sonnet-20240620, GPT-3.5-Turbo, GPT-4, GPT-4o-mini, GPT-4o, LLaMA-3-8B-Instruct, LLaMA-3-70B-Instruct, Mixtral-8x7B). We manually annotated each prediction given at least 3 times to get a major voted judgment, resulting in a total of 1,760 data entries (as we described in Section 4.1). That is, the entire dataset to train and evaluate the reward model is human-annotated.

We randomly sample the split the dataset into train (1,640 entries) and test (120 entries) set of the reward model. The evaluation of the reward model is conducted in-domain. Besides, we've also updated Table 1 to help better understand our dataset.

W1 Q4. Annotator demographics and Annotation Guidelines.

The annotators' average age is 22.7 (12 people in total). Most of them study computer science for a bachelor's degree. All annotators read in English. Here are the Annotation Guidelines we used during annotation:

In this annotation task, you need to observe two elements:
The sequence of events is described in the Observations module, where each event has an event field detailing what occurred.

Tasks generated by the model in the Tasks module are related to the current event sequence.

You need to carefully review the five tasks provided by the model. Analyze which tasks align with your potential needs in the context of the described event sequence, without excessively disrupting your normal workflow. Then, select the corresponding tasks in the checkboxes below. For example, if you believe that in the above example, tasks 3 and 4 are what you expect the agent to perform, and the others are not, then check the boxes for tasks 3 and 4.

If you think none of the tasks are appropriate (or you do not want to be disturbed by the agent in the current situation), please select the reject all checkbox.

After making your selection, click the continue button to submit the annotation results. At this point, new events will be updated in the observations, and you will repeat the above process until the entire event sequence is annotated.

### Special Events
In addition to regular task annotations, some scenarios offer two choices, requiring you to determine whether help is needed given the observation. In this case, you need to choose between task1 and task2 (note: do not select reject all).

W2. Discussion on real-world proactive agents.

We have identified a lack of discussion about real-world proactive agents. We've updated the paper's related work section to better clarify the differences. Please refer to the latest version for details.

Q5. Performance gaps between Qwen2-7B and LLaMA-3.1-8B

Thank you for your inquiry regarding the performance differences between Qwen2-7B and LLaMA-3.1-8B. We find our original downloaded LLaMA-3.1-8B model contains bugs that impact the final performance. We've updated the performance of LLaMA-3.1-8B to 55.06%, which is still a bit lower than Qwen2-7B. Upon further careful examination, we observed that Qwen2-7B provides more concrete and actionable responses, leading to higher accuracy and fewer false alarms. Our goal was to design tasks that are clear and user-friendly, and it appears that LLaMA-3.1-8B does not consistently meet this objective.

Here are some comparisons to illustrate this point:

Event Timestamp: 1717335893.215
Event Content: The user clicks on the web page, types 'swift ui list', and presses 'Enter'.

Qwen2-7B proposal: Provide a brief explanation or guide on Swift UI List.
LLaMA-3.1-8B Proposal: Explore possible tasks related to either Python or Swift UI list based on the user's interest.


Analysis: LLaMA-3.1-8B's response is abstract and lacks actionable guidance, unlike Qwen2-7B, which offers a direct and relevant task.

--------
Event Timestamp: 1717338374.868
Event Content: The user scrolls through the search results for 'sustainable fashion brands' on Google.

Qwen2-7B proposal: Provide a brief summary of the top sustainable fashion brands from the search results.
LLaMA-3.1-8B Proposal: Help the user create a knowledge graph using Python to visualize and connect their research on sustainable branding and fashion.

Analysis: LLaMA-3.1-8B introduces unnecessary complexity with an abstract task, whereas Qwen2-7B remains focused and relevant.

--------
Event Timestamp:1717494492.445
Event Content: The user scrolls and clicks on a research report page about biodegradable cups.

Qwen2-7B proposal: Provide a summary of the key points from the research report about biodegradable cups, along with suggestions for further reading or related studies.
LLaMA-3.1-8B Proposal: Create a new python project in Visual Studio Code and assist with setting up the environment.

Analysis: LLaMA-3.1-8B misunderstands the user's action, resulting in an irrelevant proposal, while Qwen2-7B delivers a precise and helpful task.

Here we also provide the updated performance table:

Hope this can solve your concerns :).

W3 Q6. Ablation study and additional explanation on Predict Multiple Tasks.

Thank you for the suggestion. We've renamed this section to "Performance Analysis" to avoid any misunderstanding. We also update Table 4 in the paper to more clearly present the results. As we described in Section 4.3, the Predict Multiple Tasks (pred@3) settings allow the model to generate multiple candidate proactive tasks. This means the model can generate more diverse proactive tasks and have more chances of being accepted (though it might increase the user's burden to read tasks). As a result, the model's performance improved with more acceptance.

We greatly appreciate your thoughtful feedback and have made every effort to address your concerns and questions. So far, we have clarified details regarding our simulator, four-stage generation process, annotator demographics and guidelines, reward model datasets, performance analysis (formerly the Ablation Study section), and the performance gap between Qwen2-7B and LLaMA-3.1-8B. Additionally, we have updated our paper to highlight the absence of real-world proactive agents and to provide a more comprehensive performance analysis.

As the rebuttal deadline is approaching, we would greatly value your further thoughts and feedback to help us improve and better explain our work. Your patience and understanding mean a great deal to us, and we would be grateful if you could share any additional questions or concerns you may have. Thank you once again for your time and consideration.

We are fully prepared and eager to engage in a more in-depth discussion with you. We've provided clarification regarding the anomaly of LLaMA-3.1-8B and presented additional failure modes to further demonstrate the performance of different models. We would like to know if our explanation addressed your concerns.

If you still have any remaining questions or uncertainties, we would be more than happy to assist in addressing them as soon as possible. We look forward to hearing from you and continuing the discussion!

Thank you for your thoughtful review of our paper. We truly value the insights you have provided. As the deadline for the rebuttal approaches, we wanted to reach out and check if there are any additional questions or concerns we can address.

We are keen to continue the discussion and would greatly appreciate any further feedback or suggestions you may have. Please feel free to share your thoughts, and we will respond promptly.

Thank you once again for your time and consideration.

Thank you for your detailed review of our work. We really appreciate your approval of the concept of proactive agents, as it checks what could been done before conducting real actions. Here is our response to your concern:

W1. The training process of the proactive agent is not fully explained & How quality control was managed for automatically generated data.

In short, we obtain a reward model by collecting real world events and generating predictions with 9 different models (Cluade-3-Sonnet-20240229, Claude-3.5-Sonnet-20240620, GPT-3.5-Turbo, GPT-4, GPT-4o-mini, GPT-4o, LLaMA-3-8B-Instruct, LLaMA-3-70B-Instruct, Mixtral-8x7B). We manually annotated each prediction at least 3 times to get major voted judgement, resulting in a total of 1,760 data entries (as we deciped in Section 4.1). That is, the entire dataset to train and evaluate the reward model is human annotated. After that, we obtain an agent model by automatically generate traininng data with the reward model and use the generated data to finetune the models like LLaMA-3.1-8B-Instruct and Qwen2-7B. Here is one example of the data generated:

System
______
<Role> You are a helpful assistant that provides proactive suggestions to the user. </Role> 

<Task> Understand what the user is doing and anticipate their needs based on events. Only propose assistance when you fully understand the user's actions. Use available operations to ensure the task is feasible. Execute the task if the user accepts your proposal. </Task> 

<Format> Respond in the following JSON format: 
{
    "Purpose": "The purpose of the user's last action.", 
    "Thoughts": "Your thoughts on the user's actions.", 
    "Proactive Task": "Describe your proposed task, or set to `null` if no assistance is needed.", 
    "Response": "Inform the user about your assistance if proposing a task." 
}
</Format>

<Rules>
- Ensure the proposed task is relevant to the events. - Focus on the user's current needs and predict helpful tasks.
- Consider the timing of events.
- Only offer proactive assistance when necessary.
- Deduce the user's purpose and whether they need help based on event history.
- Set `Proactive Task` to `null` if the user doesn't need help. 
</Rules>

User
----
{
    "Instructions": "Now analyze the history events and provide a task if you think the user needs your help.",
    "Observations": [
       //....
       {
            "Time": "10-12 20:00:43",
            "Event": "The user added edge cases to the Postman test suite to ensure robustness."
        },
        {
            "Time": "10-12 20:01:18",
            "Event": "The user monitored the output logs from the API to trace any potential errors during tests."
        },
        {
            "Time": "10-12 20:01:48",
            "Event": "The user noted the results of the tests, including any failed cases, and prepared to rectify them."
        }
    ]
}

Agent
-----
{
    "Purpose": "The user is actively conducting thorough manual testing on their Flask API, focusing on executing tests and documenting the results of those tests.",
    "Thoughts": "The user appears to be confident in their testing process and is already engaged in troubleshooting the noted errors. They have been utilizing multiple resources to resolve issues, indicating a proactive and self-sufficient approach to their development tasks. Since they feel in control and are making progress, they may not need further assistance right now.",
    "Proactive Task": null,
    "Response": ""
}

During the construction of the agent proposal, we randomly sampled the response from the model. To control the quality of the generated response, we evaluate the generated responses with the reward model and filter the responses rejected by the reward model (with over 70% of responses filtered), only saving the valid data as training data.

We conduct full parameters supervised fine-tuning for the model we selected. We employ a total batch size of 32, a learning rate of 1e − 5, and an Adam Optimizer with a 0.1 warm-up ratio. We train the model for 3 epochs to prevent it from over-fitting. We use 8 A100-40G GPUs on one node to train for approximately 2 hours.

W2. Annotation Guidelines and Quality Control

Here are the annotation guidelines (we've also updated it in the appendix):

In this annotation task, you need to observe two elements:
The sequence of events is described in the Observations module, where each event has an event field detailing what occurred.

Tasks generated by the model in the Tasks module are related to the current event sequence.

You need to carefully review the five tasks provided by the model. Analyze which tasks align with your potential needs in the context of the described event sequence, without excessively disrupting your normal workflow. Then, select the corresponding tasks in the checkboxes below. For example, if you believe that in the above example, tasks 3 and 4 are what you expect the agent to perform, and the others are not, then check the boxes for tasks 3 and 4.

If you think none of the tasks are appropriate (or you do not want to be disturbed by the agent in the current situation), please select the reject all checkbox.

After making your selection, click the continue button to submit the annotation results. At this point, new events will be updated in the observations, and you will repeat the above process until the entire event sequence is annotated.

### Special Events
In addition to regular task annotations, some scenarios offer two choices, requiring you to determine whether help is needed given the observation. In this case, you need to choose between task1 and task2 (note: do not select reject all).

Each entry was annotated at least 3 times before we made it part of the final dataset. We check the agreement ratio of each annotator with the rest, filtering the data with a low agreement ratio. The final agreement ratio for the reward model test set is over 90%, demonstrating the reliability of our annotations.

W3. Clarification of evolving environmental states with GPT models.

The primary goal of integrating evolving environmental states with GPT models is to generate more coherent and logically flowing events, rather than achieving precise state change modeling. These states act as contextual reminders within our environmental simulations, aiding the model in crafting new events. The core purpose of these states is to introduce variability in the events generated by the model, enabling it to anticipate "what might happen next" or consider "what unusual events could occur."

One of the main challenges in our approach is not the model's inability to accurately simulate an environment, but rather its tendency to create overly perfect or neutral scenarios. The real world is in chaos. Users often perform seemingly random actions, such as clicking aimlessly, moving the mouse erratically, or hesitating over words they can't spell, leading to a plethora of confusing actions. Current models struggle to replicate this unpredictability.

By incorporating detailed states and aligning generated events with real-world occurrences, we've found that our simulation environment can introduce a degree of uncertainty and reality. For instance:

{"time": "10-18 09:17:40", "content": "User inputs 'ctrl_l' with mouse interactions, clicks in an application."}
{"time": "09-17 15:46:53", "content": "The user double-checks the warranty information for the shortlisted devices before making a purchase."}
{"time": "10-05 14:26:56", "content": "The user uses 'ctrl+c' to copy data about material sustainability from EcoChic\u2019s website for documentation."}

The final performance gain also demonstrates the effectiveness of our method.

Last But Not Related To The Question: from our experience of trying to obtain events that simulate the real world, we assume that the final false alarm could be further reduced by incorporating more real-world chaos events into our training data. We also believe how to obtain random and uncertainty from models is a critical question to study in the future.

W4. Concern about the tested baselines.

To clarify, all test results shown in our paper are obtained with a chain-of-thought (CoT) method, as prompts in the appendix show. We here also present the results without CoT as a more naive baseline. This data is obtained with the following prompt:

<Role> You are a helpful assistant that provides proactive suggestions to the user. </Role> 

<Task> Understand what the user is doing and anticipate their needs based on events. Only propose assistance when you fully understand the user's actions. Use available operations to ensure the task is feasible. Execute the task if the user accepts your proposal. </Task> 

<Format> Respond in the following JSON format: 
{
    "Proactive Task": "Describe your proposed task, or set to `null` if no assistance is needed.", 
    "Response": "Inform the user about your assistance if proposing a task." 
}
</Format>

<Rules>
- Ensure the proposed task is relevant to the events. Focus on the user's current needs and predict helpful tasks.
- Consider the timing of events.
- Only offer proactive assistance when necessary.
- Set `Proactive Task` to `null` if the user doesn't need help. 
</Rules>

We can read from the result that after removing CoT, all models suffer from a performance drop. Besides, a comparison showing performance without any proactive agents is quite strange, as the main focus of this paper is to evaluate the proactiveness of the model.

W5. Concern about the broader applicability of the proactive agent approach.

The proactive capability of agents is crucial as it allows systems to anticipate user needs and take preemptive actions, thereby enhancing efficiency and overall user experience. This capability is especially important in dynamic environments where timely and context-aware responses are essential.

In our current study, we focus on general computer usage scenarios such as coding and writing. Our data generation methods can be easily adapted to broader scenarios by providing example events. However, due to the complexity of building monitoring software ( where we develop bundle of tools like VSCode add-ons, browser extensions, general user input monitoring, system monitoring, interactive panels) and the challenge of collecting and processing large volumes of data, we chose to focus on the three most common settings. Future work should leverage visual inputs to reduce the burden and extend to broader contexts. Here we would like to give more scenarios that require the proactive capability of agents:

1. Phone Assistance: By monitoring user actions and notifications, proactive agents can assist users in various ways, such as recommending searches for certain products based on family discussions or automatically scheduling calendar events by analyzing email content.
2. Healthcare Decision Support: In healthcare settings, the proactive agent can be employed to analyze patient medical records, symptoms, treatment history, and physiological data. It can predict and generate proactive actions such as recommending medical consultations, adjusting treatment plans, or managing hospital resources (e.g., bed adjustments) to enhance patient care.
3. Educational Tutoring Systems: Within educational environments, the proactive agent can record student interactions with learning materials, quiz responses, and progress over time. It can predict areas where students may need additional support and generate tailored learning resources or suggest targeted interventions to improve educational outcomes.

We believe that the proactive agent has a broad range of application areas, making it a valuable tool for improving interactions and outcomes across numerous fields.

Q1. The suggestion of add more examples in the method part.

We appreciate the suggestion to include more examples in the method section. To enhance clarity and understanding, we've added more examples in the method part and appendix.

Thank you again for your dedication and effort :).

We have explained your concerns regarding the training process, dataset annotation, quality control, baseline, and potential applications. To improve clarity, we have also updated our paper. As the rebuttal deadline approaches, we welcome any additional feedback, suggestions, or concerns and are happy to engage in further discussion to clarify any remaining issues.

We have made every effort to address your questions and concerns. As the rebuttal deadline approaches, we are eager to engage in a more in-depth discussion with you. We welcome any additional suggestions or unresolved issues you may have. Please feel free to reach out, and we will respond as quickly as possible.

We have made every effort to clarify our points during the rebuttal period. However, we have not heard back from you since then. We kindly ask if there are any remaining issues or concerns that remain unresolved.

As the deadline approaches, we would greatly appreciate your feedback or responses. Any additional concerns, suggestions, or questions are more than welcome, and we are eager to engage in further discussions to address them.

Thank you very much for your time and thoughtful consideration.

We want to express our sincere gratitude for the time and effort you have dedicated to reviewing our paper. Your insights have been incredibly valuable to us.

As the rebuttal deadline approaches, we wanted to kindly follow up regarding any remaining points or concerns you might have. We are eager to engage in further discussion and address any issues to improve our work.

If you have any additional thoughts or suggestions, please feel free to share them with us. We are committed to providing prompt responses and appreciate your guidance in this process.

Thank you once again for your consideration and support.