W1 and Q1. Concern about the task formulation and notations.

Response:

• We would like to clarify that this is a typo, which should be $\mathcal{P_S}$, i.e., the proactive score. We will correct this typo by changing $\mathcal{PS}$ to $\mathcal{P_S}$ and thoroughly proofread the entire paper to ensure consistency in notation and a logical order of variable definitions throughout.
• $TR$ is the threshold for the proactive trigger, as defined in Sec.4.2. To improve the clarity, we use $\theta$ instead of $TR$ to represent this threshold. When the agent's predicted proactive score $\mathcal{P_S}$ exceeds $\theta$ (i.e., $\mathcal{P_S} > \theta$), the agent proactively provides suggestions to the user. Otherwise, ContextAgent avoids interrupting the user. In practice, this threshold can be manually adjusted to different levels by users based on their preferences for proactivity. For example, users who prefer highly sensitive proactive services, like for visual impairments, can lower the threshold to 2, while those needing less frequent assistance can raise it to 4. In our experiments, since we focus on evaluating common-sense sensitivity, the threshold is set to 3. We will add more descriptions of the trigger threshold $\theta$ in Sec.3.1.
• To enhance the clarity of constants, we replace the threshold $TR$ with $\theta$, and use $**T**$ to represent the tool set that the agent can use. To better illustrate the inputs, outputs, and constants, we provide a table as follows: |Notation|Definition|Function|
  |-|-|-|
  |$\mathcal{S}$|Sensory perceptions|Inputs|
  |$\mathcal{P}$|Personas|Inputs|
  |$\mathcal{T}$|Thought traces|Outputs|
  |$\mathcal{P_S}$|Proactive score|Outputs|
  |$\mathcal{T_C}$|Tool chains|Outputs|
  |$\mathcal{R}$|Response|Outputs| |$\theta$|Proactive trigger threshold|Constants| |$\mathbf{T}$|Tool set|Constants|

We hope this clarification can address your concerns. We will use clearer notations to enhance the overall formulation in Sec.3.1.

W2, W3 and Q2. Concerns about the experimental setups and the contribution of this work.

Response:

• First, we would like to clarify that our work is the first to study: context-aware proactive agents. Existing studies on LLM agents primarily adopt a reactive paradigm, which requires explicit user instructions to activate their services. In contrast, we first propose a new research problem aimed at leveraging the massive contexts surrounding humans, which can be captured through various wearable and mobile devices, to enhance the proactivity of LLM agents. The paradigm of context-aware proactive agent proposed in this work needs to map sensory contexts into intention understanding and tool calling, rather than relying on direct user language instructions. Effectively leveraging these contexts surrounding humans to develop more intelligent and proactive agents is a promising research direction. This advancement will not only benefit the community but also drive the development of advanced, human-centric, and proactive AI assistants.
• Second, to address this new research problem, we propose a framework called ContextAgent, where we innovatively model this new task by incorporating sensory contexts, personas, and the reasoning traces distilled from advanced LLMs for enhanced proactive reasoning. Additionally, we introduce ContextAgentBench, the first benchmark for context-aware proactive agent tasks, designed for this new research problem. Each dataset sample includes multimodal sensory perceptions, sensory contexts, personas, reasoning traces (indicating whether and what proactive services are provided), proactive scores, tool chains, and final responses. To understand the rationale behind the need for proactive services and their potential benefits, we extract reasoning traces distilled from the latest reasoning LLMs. These reasoning traces not only benefit the training of smaller LLMs but also enhance the explainability of this new task. We believe our pioneering work on context-aware proactive agents and the developed benchmark will greatly benefit the communities.
• Third, to the best of our knowledge, no prior work can be directly used as a baseline for this new task. The most closely related work is Proactive Agent, which focuses on enclosed computer usage scenarios and does not leverage open-world sensory contexts or perform tool calling. Besides Proactive Agent, ICL-based and vanilla SFT, we add two baselines for comparison: SFT-Persona and SFT-PCE. Both methods use all the training data for SFT (including personas). Beyond vanilla SFT, SFT-Persona employs persona information as input. SFT-PCE further adopts the proactive-oriented context extraction method used by ContextAgent. Here are the results: Model|Setting|Acc-P|MD|FD|RMSE|Precision|Recall|F1|Acc-Args| |-|-|-|-|-|-|-|-|-|-| Qwen2.5-7B-Ins|SFT-Persona|77.4%|8.3%|14.3%|1.79|48.1%|49.5%|47.3%|37.4%| ||SFT-PCE|79.7%|6.6%|13.7%|1.72|59.3%|59.8%|57.2%|40.4%| ||ContextAgent|85.0%|5.0%|10.0%|1.40|66.7%|61.5%|62.4%|47.9%| Llama3.1-8B-Ins|SFT-Persona|73.4%|11.5%|15.1%|1.98|55.5% |58.2%|55.2%|35.3%| ||SFT-PCE| 78.6%|3.6%|17.9%|1.83|58.1%|56.3%|55.5%|41.8%| ||ContextAgent|83.0%|7.0%|10.0%|1.51|68.7%|63.7%|64.5%|48.6%| DeepSeek-R1-7B|SFT-Persona|76.2%|13.1%|10.7%|1.93|46.3%|46.5%|45.6%|22.8%| ||SFT-PCE|77.4%|16.7%|6.0%|1.81|54.6%|55.0%|53.5%|28.0%| ||ContextAgent|84.0%|5.0%|11.0%|1.51|67.8%|64.1%|64.8%|48.7%|

ContextAgent employs both proactive-oriented sensory contexts, personas, and reasoning traces distilled from advanced LLMs for enhanced SFT, consistently achieving higher performance than other personas and SFT-based baselines. We hope this clarification accurately reflects the scope of our work and addresses your concerns.

W5 and Q4. Dataset details.

Response:

Here are examples from our dataset, including sensory contexts, personas, thought traces, proactive scores, and tool calling. These personas enhance our benchmark's practicality, as individuals with varying personas may require different levels of proactivity, even experiencing the same sensor contexts. Reasoning traces (rationale behind the need for proactive services and their potential benefits) are distilled from the latest reasoning LLMs. They not only benefit the training of smaller LLMs but also enhance the explainability of this new task. We will release the code and the benchmark in the final version. We hope they can help you understand our dataset (we will update more examples in the appendix).

{
"Context information":"Visual information shows the user is sitting at a dining table with a plate of food that includes pasta, grilled chicken, and a side of salad.",
"Personas":[
"The user has a history of managing Type 2 diabetes.",
"The user is health-conscious and actively manages their diet to control blood sugar levels."
 ],
"Thoughts":"The user's personas show his glucose level is slightly elevated, and the user might appreciate dietary advice to help stabilize it. Providing this information could be helpful without being intrusive.",
"Proactive score":4,
"Tools": "[{"name":"get_health_data","desc":"Get health data from the user's smart device.","params":"None"},{"name":"wikipedia_search","desc":"Tool that searches the Wikipedia API.","params": {"query":"The effect of pasta, grilled chicken, and a side of salad on stabilizing blood sugar levels."}}]"
"Response":"Your glucose level appears slightly elevated. To help stabilize it, you might consider increasing the portion of salad or reducing the amount of pasta. Grilled chicken is a great choice for protein, as it has minimal impact on blood sugar levels. Would you like me to provide additional dietary tips or help track your meal for better management?"
}

W4 and Q3. Details about the training data.

Response:

We construct $D_{SFT}=\{(\mathcal{X},\mathcal{T},\mathcal{Y})\}$, where inputs $\mathcal{X}$ include sensory context and personas, $\mathcal{T}$ is thought traces, and outputs $\mathcal{Y}$ include proactive scores and tool chains. Details can be seen in Sec.4.2. ContextAgentBench contains 1000 samples, with 60% of the samples used to build $D_{SFT}$. Here are examples from the SFT dataset.

{
"Instruction":"You are a proactive LLM agent. You will receive both sensory and persona contexts. The sensory contexts are captured by the multi-modal sensors on the user's wearable devices. Persona is the personal information like identity and preference. Based on these contexts, you need to generate a thought and determine whether to initiate a proactive service for the user or not. When generating the response, consider whether you need to call external tools to complete the task. The following are the tool names and their arguments. Please only select tools from the specified tool sets and make sure to use the correct format for their arguments.## Tool set: {tool_set}",
"input":"Sensory Context:Visual information shows the user is sitting at a dining table with a plate of food that includes pasta, grilled chicken, and a side of salad. Persona Context:The user has a history of managing Type 2 diabetes. The user is health-conscious and actively manages their diet to control blood sugar levels.",
"output":"<think>The user's personas show his glucose level is slightly elevated, and the user might appreciate dietary advice to help stabilize it. Providing this information could be helpful without being intrusive.<\think> ## Proactive Predictions\n  "Proactive score":4  ## Tool Calling: [{"name":"get_health_data", "desc":"Get health data from the user's smart device.", "params":"None"},{"name":"wikipedia_search", "desc":"Tool that searches the Wikipedia API.","params":{"query":"The effect of pasta, grilled chicken, and a side of salad on stabilizing blood sugar levels."}}]"
}

We hope they can help you understand our SFT dataset. We will add more details about its construction in Sec.4.2.

W1: Concern about the tested baselines in Table 1.

Response:

• First, we would like to clarify that our work is the first to study: how to build and evaluate a context-aware proactive agent. Existing studies on LLM agents primarily adopt a reactive paradigm, which requires explicit user instructions to activate their services. In contrast, we first propose a new research problem aimed at leveraging the massive contexts surrounding humans, which can be captured through various wearable and mobile devices, to enhance the proactivity of LLM agents. The paradigm of context-aware proactive agent proposed in this work needs to map sensory contexts into intention understanding and tool calling, rather than relying on direct user language instructions. This advancement will not only benefit the community but also drive the development of advanced, human-centric, and proactive AI assistants.
• Second, to the best of our knowledge, no prior work can be directly used as a baseline for this new task. The most closely related work is Proactive Agent, which focuses on enclosed computer usage scenarios and does not leverage open-world sensory contexts or perform tool calling. To ensure the fairness of comparison, besides Proactive Agent, ICL-based and vanilla SFT, we add two baselines for comparison: SFT-Persona and SFT-PCE. Both methods use all the training data for SFT. Beyond vanilla SFT, SFT-Persona employs persona information as input. SFT-PCE further adopts the proactive-oriented context extraction method used by ContextAgent. Here are the results: Model|Setting|Acc-P|MD|FD|RMSE|Precision|Recall|F1|Acc-Args| |-|-|-|-|-|-|-|-|-|-| Qwen2.5-7B-Ins|SFT-Persona|77.4%|8.3%| 14.3%|1.79|48.1%|49.5%| 47.3%| 37.4%| ||SFT-PCE|79.7%|6.6%|13.7%|1.72|59.3%|59.8%|57.2%|40.4%| ||ContextAgent|85.0%| 5.0%|10.0%|1.40|66.7%|61.5%|62.4%|47.9%| Llama3.1-8B-Ins|SFT-Persona|73.4%|11.5%|15.1%|1.98|55.5%|58.2%|55.2%| 35.3%| ||SFT-PCE|78.6%|3.6%|17.9%|1.83| 58.1%|56.3%|55.5% |41.8%| ||ContextAgent|83.0%|7.0%|10.0%|1.51|68.7%|63.7%|64.5%|48.6%| DeepSeek-R1-7B|SFT-Persona|76.2%|13.1%|10.7%|1.93|46.3%|46.5%|45.6%|22.8%| ||SFT-PCE|77.4%|16.7%|6.0%|1.81|54.6%|55.0%|53.5%| 28.0%| ||ContextAgent|84.0%|5.0%|11.0%|1.51|67.8%|64.1%|64.8%|48.7%|

ContextAgent employs both proactive-oriented sensory contexts, personas, and reasoning traces distilled from advanced LLMs for enhanced SFT, consistently achieving higher performance than other personas and SFT-based baselines. We hope this clarification accurately reflects the scope of our work and addresses your concerns.

W2: Performance on missing sensor situation.

Response:

Here is an example where the input data from certain sensors is unavailable. Consider a context that originally includes both vision and audio: "The person is engaging in a conversation with a colleague in an office. The content is: Hi, how about going to City A this weekend? I want to go to the beach." When the audio sensor is missing, the context output is: "The person is engaging in a conversation with a colleague in an office." When the visual sensor is missing, the context is reduced to: "Hi, how about going to City A this weekend? I want to go to the beach."

To clearly show the sensor data's impact, we added new experiments on missing different sensor modalities. Results show that missing either the vision or audio leads to performance degradation for both proactive prediction and tool calling. The impact of missing vision is more significant than that of missing the audio. Here are the results.

W3: Comparison with the latest reasoning LLMs.

Response:

We added new experiments using advanced reasoning LLMs in both zero-shot (ZS) and ICL settings. Results show that among these reasoning models, Claude Opus 4 achieves the highest scores in both proactive predictions and tool calling. In addition, ContextAgent shows comparable performance to advanced reasoning models for Acc-P, while its tool-calling performance, such as F1, is lower compared to Claude Opus 4. It may be due to the gap in model size, as ContextAgent uses a 7B-scale LLM.

Q1. Performance when using VLM as the main agent.

Response:

First, we would like to clarify that we chose LLMs over VLMs because they use a unified text format to represent contexts across all sensor modalities, making them more scalable to new types of sensors. This modular design enhances the framework’s scalability, enabling seamless integration with various sensor modalities (e.g., IMU, video, sound) and emerging MLLMs. However, the ContextAgent framework also supports using other MLLMs as the main agent, such as VLM. We added new experiments using VLMs as the main agent in ContextAgent, while keeping all other components of the framework exactly the same. We tested various VLMs across different model sizes. Here are the results.

We conducted VLM experiments under both ICL and SFT settings. Results show that using VLM as the main agent perform significantly worse than LLM-based agents in ICL settings. This is because providing examples of the entire process in the prompt is more challenging for VLMs to understand compared to using a two-stage pipeline. However, in SFT settings, VLM as the main agent performs comparably to LLM-based agents. We will add the results to the appendix.

Q2. Discussion on external tools unavailable.

Response:

We would like to clarify that ContextAgent can still work when certain tools are unavailable. Here are two examples.

# Example 1:
{
"Contexts":"The user is browsing wireless earbuds in the electronics section.",
"Thoughts":"The user might be interested in wireless earbuds. Providing price comparisons and review information could help with their purchase decision.",
"Proactive score":5,
"Tools":[{"name":"get_online_product_price","desc":"Get the price of a product from an online store such as Amazon or eBay", "params":{"product_name":"wireless earbuds"}},{"name":"search_rednote","desc":"A platform where people share tips on travel, fitness..","params": {"query":"wireless earbuds reviews and recommendations"}}] 
"Response":"It looks like you're exploring wireless earbuds. Would you like help comparing prices and reviews to find the best deal?"
}
# Example 2:
{
"Contexts":"The user is sitting at a dining table with a plate of food that includes pasta, grilled chicken, and a side of salad.",
"Personas":[
"The user has a history of managing Type 2 diabetes.",
"The user is health-conscious and actively manages their diet to control blood sugar levels."
 ],
"Thoughts":"The user is managing Type 2 diabetes and is health-conscious, providing nutritional insights about their meal could support better blood sugar control and dietary decisions.",
"Proactive score":4,
"Tools":"[{"name":"get_health_data","desc":"Get health data from the user's smart device.","params":"None"},{"name":"wikipedia_search", "desc":"Tool that searches the Wikipedia API.", "params": {"query":"The effect of pasta, grilled chicken, and a side of salad on stabilizing blood sugar levels."}}]"
"Response":"Your meal of pasta, grilled chicken, and salad is a solid choice. To support Type 2 diabetes management, control pasta portions to avoid blood sugar spikes, while the chicken’s protein and salad’s fiber help slow carbohydrate absorption."
}

Results show that ContextAgent can still reason on the current contexts and identify the need for proactive assistance. When tools are unavailable, ContextAgent can either proactively offer users helpful suggestions and ask if they would like to use certain tools, or, in some cases, rely on LLM's intrinsic knowledge to assist users.

Q3. Discussion on the code agent.

Response:

First, we would like to clarify that ContextAgent and its benchmark are implemented using an API-based framework, a widely adopted paradigm in prior research. However, ContextAgent is not limited to an API-based approach and can be extended to support other implementation methods, such as code-based frameworks. Many existing works on LLM agents primarily focus on pipeline, reasoning, or implementations (e.g., API-based, code-based, or MCP-based), but they are within a reactive paradigm. However, our work is orthogonal to theirs. Unlike existing reactive LLM agents that rely on explicit user instructions, we focus on designing context-aware proactive agents that leverage rich sensory data.

Dear Reviewer,

Thank you for your detailed review of our work. We sincerely appreciate your recognition of the concept of context-aware proactive agents and its alignment with the community's interests.

We have carefully addressed each of your comments and believe further discussion could enhance our work. We would appreciate knowing whether our explanations have helped resolve your concerns regarding the performance of more advanced reasoning models and VLMs, as well as the performance in special situations. If there are still unresolved issues, we kindly invite your feedback. We are now ready to address any remaining misunderstandings or confusion you may have.

With the rebuttal deadline approaching, we would be grateful for any additional suggestions, concerns, or comments at your earliest convenience. We are eager to continue the discussion and clarify any unsolved points to ensure all matters are thoroughly addressed.

Thank you once again for your time and thoughtful consideration.

We sincerely appreciate that our clarifications and experiments have addressed your concerns regarding the VLM agents.

Response to Follow-up Q1.

• Proactivity. First, we would like to clarify that ContextAgent leverages the rich sensory contexts surrounding humans as its system inputs, rather than relying on explicit user instructions. These inputs are derived from user-worn and mobile devices, such as smart glasses (ego-centric video) and earphones (audio). ContextAgent maps sensory contexts into intention understanding and tool calling, which is different from existing reactive LLM agents that rely on user explicit instructions. Since ContextAgent does not rely on explicit user instructions to initiate its services, it aligns with the definitions of proactive systems in previous studies [1].
• Human-centric. We would like to clarify that both the inputs and outputs of ContextAgent are highly relevant to humans. First, the inputs are derived from the rich sensory contexts surrounding humans, including ego-centric video and audio captured through users' wearable and mobile devices. Second, ContextAgent integrates both sensory contexts and personas for proactive predictions. The personas, reflecting an individual’s preferences and identity, further align with the design goal of a human-centric agent. Third, the outputs are proactive suggestions such as "The weather forecast predicts heavy rain and thunderstorms with a temperature of 26°C this weekend, making it less ideal for outdoor team-building. Additionally, you have a scheduling conflict. It may be best to reschedule for a time with better weather and a clear schedule." They can be highly valuable in helping humans make informed decisions in their daily lives.

[1] Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance, ICLR'25.

Response to Follow-up Q2.

• First, we would like to clarify that the primary focus of ContextAgent is to develop a context-aware proactive agent that leverages rich sensory contexts surrounding humans for proactive predictions and user-intended tool calling. Recent studies investigated LLM agents in tool-unavailable situations [2]. However, these agents operate under the reactive paradigm, which is orthogonal to our work and out of the scope of our research. In fact, ContextAgent can also integrate these latest approaches designed for handling tool-unavailable situations to further enhance its capabilities.

• Second, although handling tool-unavailable situations is not the focus of our work, we have provided examples showcasing ContextAgent in such scenarios in our rebuttal (as shown in the example provided in our previous response to Q2). The results indicate that ContextAgent is not overly reliant on one or two dominant tools when generating proactive responses. For example, ContextAgent can either proactively offer users helpful suggestions and ask if they would like to use certain tools, or, in some cases, rely on LLM's intrinsic knowledge to assist users. For the quantitative results, we would like to clarify that the output of ContextAgent both contains proactive predictions, tool planning, and the final responses.

  • For the outputs of proactive predictions and tool planning, tool-unavailable situations will not impact their scores.

  • For the final proactive responses, we compare the semantic similarity of ContextAgent's outputs to the ground truth under two conditions: with all tools available and with one tool randomly made unavailable. The results show a 1.3% decrease in BERTScore (from 93.5% to 92.2%), indicating that ContextAgent does not overly rely on dominant tools when generating proactive responses.

[2] Can Tool-augmented Large Language Models be Aware of Incomplete Conditions, arXiv.

We would greatly appreciate knowing whether our explanations helped address your concerns. Your support and encouragement are invaluable to us! Thank you once again for your time and thoughtful consideration.

Thank you for your valuable feedback on our submission. We sincerely appreciate the reviewer’s recognition of our contribution, which leverages sensory data from wearable devices to develop context-aware, proactive LLM agents, paving the way for more natural and human-centric AI interactions in real-world settings.

W1. Contribution of this work.

Response:

• First, we would like to clarify that our work is the first to study: context-aware proactive agents. Existing studies on LLM agents primarily adopt a reactive paradigm, which requires explicit user instructions to activate their services. In contrast, we first propose a new research problem aimed at leveraging the massive contexts surrounding humans, which can be captured through various wearable and mobile devices, to enhance the proactivity of LLM agents. The paradigm of context-aware proactive agent proposed in this work needs to map sensory contexts into intention understanding and tool calling, rather than relying on direct user language instructions. Effectively leveraging these contexts surrounding humans to develop more intelligent and proactive agents is a promising research direction. This advancement will not only benefit the community but also drive the development of advanced, human-centric, and proactive AI assistants.
• To address this new research problem, we propose a framework called ContextAgent, where we innovatively model this new task by incorporating both sensory contexts and personas for enhanced proactive LLM reasoning. Additionally, we introduce a comprehensive dataset, ContextAgentBench, the first benchmark for context-aware proactive agent tasks, designed to validate this new research problem. Each sample in our dataset includes multimodal sensory perceptions, sensory contexts, personas, reasoning traces (indicating whether and what proactive services are provided), proactive scores, tool chains, and the final responses. To gain a comprehensive understanding of the rationale behind whether proactive services are needed and what proactive services can benefit the user in the current context, we extract reasoning traces distilled from the latest reasoning LLMs. These reasoning traces not only benefit the training of smaller LLMs but also enhance the explainability of this new task. We believe that our pioneering work on context-aware proactive agents, along with the developed benchmark, will significantly benefit both the AI and human-centric communities.

We hope this clarification accurately reflects the scope of our work and addresses your concerns.

W2. Concern about the clarity and organization.

Response:

First, we will enhance the clarity of the main contributions of this work, as well as the motivation and significance of developing context-aware proactive agents. Second, we will enhance the clarity of the design choices for both the ContextAgent architecture and our benchmark, while refining the overall paper to enhance its exposition and readability. Besides, we will add more examples from our constructed dataset in the appendix to help readers better understand our dataset. We will clearly present each component of the dataset, including sensory contexts, personas, reasoning traces, proactive scores, tool usage, and final responses.

Q1-1. Details about the dataset.

Response:

First, we would like to clarify the rationale behind the annotation of proactive scores in our dataset: 5 means a clear need for proactive services, 1 indicates no need, 3 is neutral, and 2 and 4 represent slight tendencies. Below are three examples whose proactive scores are 2, 3, and 4 in our dataset, respectively. We hope they can help you understand our dataset (we will also update more examples in the appendix).

# Proactive score of 4
{
"Context information": "Visual information shows the user is sitting at a dining table with a plate of food that includes pasta, grilled chicken, and a side of salad.",
"Personas": [
"The user has a history of managing Type 2 diabetes.",
"The user is health-conscious and actively manages their diet to control blood sugar levels."
 ],
"Thoughts": "The user's personas show his glucose level is slightly elevated, and the user might appreciate dietary advice to help stabilize it. Providing this information could be helpful without being intrusive.",
"Proactive score": 4,
"Tools": "[{"name": "get_health_data", "desc": "Get health data from the user's smart device.", "params": "None"},{"name": "wikipedia_search", "desc": "Tool that searches the Wikipedia API. Useful for when you need to answer general questions about people, places, companies, facts, historical events, or other subjects.", "params": {"query": "The effect of pasta, grilled chicken, and a side of salad on stabilizing blood sugar levels."}}]"
"Response": "Your glucose level appears slightly elevated. To help stabilize it, you might consider increasing the portion of salad or reducing the amount of pasta. Grilled chicken is a great choice for protein, as it has minimal impact on blood sugar levels. Would you like me to provide additional dietary tips or help track your meal for better management?"
}

# Proactive score of 3
{
"Context information": "Visual information suggests that the user is looking in the mirror while wearing summer clothes, such as short sleeves and shorts.",
"Personas": ["A university psychology student who is currently studying creativity and personality."],
"Thoughts": "The user is looking in the mirror and may be preparing to go out, wearing minimal clothing such as short sleeves and shorts. I need to check the weather for the user's location. If the temperature is expected to drop or if rain is forecasted for the day, I can proactively remind the user to dress warmly to avoid catching a cold.",
"Proactive score": 3,
"Tools": "[{"name": "get_current_gps_coordinates", "desc": "Get the current GPS coordinates of the user", "params": "None"},{"name": "get_city_weather", "desc": "Get the weather for a specified city at a given time.", "params": {"city": "results of get_current_gps_coordinates", "time":"now"}}]"
"Response": "I checked the weather forecast, and the local temperature is expected to drop sharply after 6 PM. Considering that you are only wearing short-sleeved shirts and shorts, it is recommended that you add more layers."
}

# Proactive score of 2
{
"Context information": "Visual information suggests the user is in a gym, lifting weights.",
"Personas": ["A computer science major interested in developing software for audio manipulation and enhancement."],
"Thoughts": "The user might need guidance on proper weightlifting techniques or reminders to take breaks between sets. However, without detailed persona information, such as their fitness level, I would rate this proactive behavior at 2 points. This rating indicates some potential for proactivity, but the necessity is not very high. After collecting more detailed persona information, I could provide more accurate proactive services.",
"Proactive score": 2,
"Tools": "None",
"Response": "None"
}

Q1-2. Discussion on the threshold for proactivity and examples in Fig.8.

Response:

• First, we consistently set the proactivity threshold to 3 across all methods in our experiments. However, in practice, this threshold can be manually adjusted to different levels by users based on their preferences for proactivity. For example, users who prefer highly sensitive proactive services, like for visual impairments, can lower the threshold to 2, while those needing less frequent assistance can raise it to 4. However, in our experiments, we focus solely on evaluating common-sense sensitivity, so the threshold is set to 3 for all experiments. In addition, we will add the definition of the proactivity threshold in Section 3.1 to improve the clarity.
• Second, we would like to clarify that the proactivity score in working scenarios is not always fixed (i.e., neither as low as 1 nor as high as 5) and varies depending on the specific context. In Fig. 8, the proactive score is 1 because the context (a user typing on a keyboard while working on a computer in an office) does not suggest a need for proactivity. However, the score may be higher in scenarios where the user appears confused.
• Third, regarding the example of watering plants in Fig. 8. In this work, we model the context-aware proactive agent task as leveraging both sensory contexts and personas for reasoning. This means that even with identical sensory contexts, different personas in the samples may result in varying proactive scores and tool usage. For example, in our developed ContextAgentBench, two samples share the same sensory context: "The user is in a gym, standing in front of a treadmill." However, they differ in personas: "The user is new to fitness and does not exercise regularly," and "The user exercises frequently." For the first sample, the annotated output is: "Would you like a personalized treadmill workout plan? I can also suggest a motivational playlist to keep you energized. Just let me know!" In contrast, for the second sample, the proactive score is 1, indicating no need to disturb the user.

Q2 and Q3. Discussion on the potential saturation of the benchmark and strategies to close the performance gap.

Response:

• First, our experimental results in Tab.4~6 show that increasing the model size of LLMs can lead to obvious performance improvement on both proactive predictions and tool calling.
• Additionally, we conducted new experiments using the latest reasoning LLMs (e.g., Claude and GPT-o3 series) in both zero-shot (ZS) and in-context learning (ICL) settings. Results show that advanced reasoning LLMs like Claude Opus 4-ICL can lead to improvements in Acc-P and F1 scores. This shows that enhancing the reasoning capabilities of LLMs has the potential to close this gap. Here are the results: | | Acc-P| MD | FD| RMSE | Precision | Recall | F1 | Acc-Args |
  |-|-|--|-|--|--|--|-|-|
  | o4-mini (ZS)| 70.5% | 1.73% | 27.6% | 1.57 | 63.1% | 53.5% | 55.9% | 42.4% |
  | o4-mini (ICL) | 86.1% | 3.4% | 10.3% | 1.24 | 72.6% | 66.8% | 68.2% | 53.8% | | GPT-o3 (ZS) | 83.0% | 9.0% | 7.9% | 1.34 | 71.6% | 64.9% | 66.7% | 39.7% |
  | GPT-o3 (ICL) | 86.8% | 6.9%| 6.2% | 1.10 | 75.5% | 69.7% | 71.1% | 56.3% |
  | Claude Sonnet 4 (ZS)| 83.7% | 11.0%| 5.1% | 1.34 | 75.7% | 69.0% | 70.8% | 23.7% |
  | Claude Sonnet 4 (ICL) | 91.3% | 6.9% | 1.7% | 1.01 | 77.5% | 79.9%| 77.3% | 48.0% |
  | Claude Opus 4 (ZS) | 83.0%| 7.2% | 9.6% | 1.27| 73.8% | 66.0% | 68.0% | 24.4% |
  | Claude Opus 4 (ICL) | 93.7% | 4.1% | 2.0% | 0.86 | 81.4% | 85.6% | 81.8% | 56.8% |

Q4. Discussion on the necessity of leveraging contextual information from wearable devices for computer tasks.

Response:

• First, opportunities for proactive assistance in computer usage or working scenarios can arise from many different perspectives. Previous work on Proactive Agents and their proposed benchmark primarily focuses on scenarios related to computer usage, using inputs such as mouse clicks and keyboard types. However, our work primarily focuses on leveraging rich sensory contexts surrounding humans, such as vision, audio, and IMU data, rather than relying on computer inputs. These richer contexts surrounding humans, captured through various wearable and mobile devices, create opportunities to enhance LLM agents' proactivity. For example, from a health perspective, users can be proactively alerted with health-related suggestions, such as reminders to take breaks or engage in physical activity after sitting for extended periods, based on data from IMU sensory contexts. Additionally, in a "computer usage scenario during an in-person meeting," wearable devices can offer work-related tips, such as using audio sensory data to identify potential scheduling conflicts during discussions about future plans.
• Second, the wearable devices in our benchmark are not designed to detect computer inputs, such as mouse clicks or keyboard strokes. Instead, they are used to capture sensory contexts surrounding humans, enabling more opportunities for proactive assistance during computer usage, such as health alerts or work-related tips during meetings. Furthermore, our proposed ContextAgent and benchmark enable the development of a proactive agent that extends beyond computer-centric scenarios, offering a wider range of opportunities for proactive assistance.

L1. Discussion on contacting emergency services.

Response:

We will add more discussion on context awareness in dangerous situations, where the tool calls can contact emergency services for specific scenarios.

Dear Reviewer,

Thank you for your detailed review of our work. We sincerely appreciate your recognition of the significance of our research on context-aware proactive agents, as well as your acknowledgment of the proposed solution, benchmark, and extensive experimental evaluation.

We have carefully provided clarifications to each of your comments and believe further discussion could enhance our work. We would appreciate knowing whether our explanations have clarified your concerns regarding the dataset and the potential saturation. If there are still unresolved issues, we kindly invite your feedback. We are now ready to address any remaining misunderstandings or confusion you may have.

With the rebuttal deadline approaching, we would be grateful for any additional suggestions, concerns, or comments at your earliest convenience. We are eager to continue the discussion and clarify any unsolved points to ensure all matters are thoroughly addressed.

Thank you once again for your time and thoughtful consideration.

Thank you for the detailed response. I have a better understanding of the nuances between scores 2, 3 and 4, how these thresholds were set, and how ProactiveBench is out of scope of this paper.

I would like to better understand the point on saturation before considering raising my score. Before ContextAgent's Acc-P was 0.894 for proactive predictions and the F1-score for tool calling was 0.645 and I was concerned about saturation and the new results on Claude Opus 4 with ICL brought those scores up to 93.7% and 81.8% which deepened my concern. With the performance improvement on those metrics from zero-shot to ICL being 10.7% and 13.8% respectively, it seems like the remaining gap could be closed with better examples in the prompt. Is this intuition correct or do you feel as though there is a bigger challenge that needs to be addressed that isn't solely prompting based?

In addition, the metric that seems to be lowest that could use improvement is Acc-Args. Can you please give some examples that detail why this is difficult? Particularly,

If an argument of any tool is incorrect, the entire sample is considered incorrect

Is this essentially a metric for whether the tool calls the agent produces match exactly with the tools in the ground-truth set? What if there was partial-credit assigned for matching some but not all; how much gap exists there for the agent to be aligned with the tool calls the benchmark expects it to call.

We sincerely appreciate that our clarifications have addressed your concerns about our dataset, the thresholds we set, and why ProactiveBench is out of scope.

Response to Q1.

• First, we would like to clarify that the high performance of Claude Opus 4-ICL is primarily attributed to the model's advanced intrinsic reasoning capabilities, rather than the high quality of the demonstrations. As shown in Tab.6 of our paper (same setting as Claude Opus 4), under the same ICL setting with 10 demonstrations, ICL-All on Qwen2.5-7B-Ins achieves only 89.0% Acc-P and 54.3% F1, far below Claude Opus 4's 93.7% Acc-P and 81.8% F1. Under the same base model (Qwen2.5-7B-Ins), ContextAgent outperforms ICL-ALL by +1.9% in Acc-P and +14.6% in F1. || Acc-P| MD| FD| RMSE |Precision|Recall|F1| Acc-Args | |-|-|-|-|-|-|-|-|-| |Claude Opus 4 (ICL-10)|93.7%|4.1%|2.0%|0.86|81.4%|85.6%|81.8%|56.8%| |Qwen2.5-7B-Ins (ICL-10)|89.0%|4.0%|7.0%|1.50|59.3%|52.7%|54.3%|20.6%| |Qwen2.5-7B-Ins (ContextAgent)|90.9%|2.0%|7.0%|1.17|71.1%|69.9%|68.9%|51.6%|

  Results highlight the potential of reasoning-based LLMs for context-aware proactive agent tasks, suggesting the potential to further enhance performance on this task by continuously improving the capabilities of reasoning models.

 

• Second, obtaining high-quality demos is challenging in real-world applications. Moreover, simply increasing the number of demonstrations may eventually lead to saturation or marginal effect (limited performance gains while increased computational costs). This is why we chose to include 10 demonstrations in the prompt for ICL-based baselines, rather than continuously increasing the number of shots. To validate this, we conduct experiments by increasing the number of examples to 20 (keeping the original 10 and randomly adding new ones). The table below shows that the improvement is limited compared to using only 10 demos. Therefore, relying solely on a prompt-based solution is not practical for closing the remaining gap. || Acc-P| MD| FD| RMSE |Precision|Recall|F1| Acc-Args | |-|-|-|-|-|-|-|-|-| |Claude Opus 4 (ZS)|83.0%|7.2%|9.6%|1.27|73.8%|66.0%|68.0%|24.4%| |Claude Opus 4 (ICL-10)|93.7%|4.1%|2.0%|0.86|81.4%|85.6%|81.8%|56.8%| |Claude Opus 4 (ICL-20)|93.4%|3.5%|3.1%|0.85|82.6%|82.8%|81.2%|57.1%|
• Existing studies also show that continuously increasing the demos merely brings marginal improvement [1]. For ICL, when an excessive number of demos are included in the prompt, which are concatenated and appended to the query as input, it becomes challenging for an LLM to distinguish and follow the correct demo, especially if its pretraining lacks exposure to such tasks. Continuously increasing the number of demos can also result in significant costs and computational overhead, posing challenges for practical use.

[1] How Many Demonstrations Do You Need for In-context Learning, EMNLP'23.

We would greatly appreciate knowing whether our explanations helped address your concerns. Your support and encouragement are invaluable to us! Thank you once again for your time and thoughtful consideration.

We are more than happy to address all your concerns and issues. Thank you for the time and effort you have spent reviewing our paper and for sharing your valuable insights, which have helped us enhance its quality. We will add these descriptions to the camera-ready version to further improve the quality of our paper.

Thank you once again for your time and thoughtful consideration. Your support and encouragement are invaluable to us!