DynaSaur: Large Language Agents Beyond Predefined Actions

We thank you for your time and the constructive feedback. We would like to address your questions below:

1. “Revision required, for example, improper citation formatting in L58-L60.”

Thank you for pointing this out. We have addressed the issue on our end and will include the fix in the revised version.

2. “The manuscript does not adequately discuss relevant related works, eg. Voyager, which similarly addresses autonomous skill creation and persistent retrieval. Including a direct comparison would significantly clarify the novelty of the current approach, such as its formal MDP framework and the broader range of task evaluations.”

While our method might share some high-level motivations with Voyager, Voyager relies on many Minecraft-specific assumptions, whereas our method is designed as a more general framework. Specifically, we differ in the following aspects:

1. Voyager assumes that tasks are organized into a well-structured hierarchy: In Minecraft, each item can be crafted from a set of lower-tier items, forming a well-defined hierarchical progression within the game. Voyager’s curriculum design heavily exploits this assumption. Given the current items in the inventory, the current position, and nearby items, it uses an LLM to suggest the next immediate task that can be performed (e.g., crafting higher-tier items). In contrast, we do not make such an assumption, as tasks in the real world are open-ended and often lack a structure that can be predetermined.

2. Tasks in Voyager follow well-defined templates: Voyager limits its task set to adhere to specific templates, such as “Mine [quantity] [block],” “Craft [quantity] [item],” “Smelt [quantity] [item],” “Kill [quantity] [mob],” and “Cook [quantity] [food].” In contrast, we make no assumptions about the task distribution. The tasks in GAIA, which we evaluated, exhibit significantly greater diversity and complexity, including web browsing, file processing, symbolic reasoning, commonsense reasoning, and more.

3. Voyager’s generated actions depend on low-level actions from the Mineflayer API: The actions generated by Voyager must conform to the Mineflayer API in order to interact with the Minecraft environment. In contrast, we impose minimal constraints on the generated actions (only basic formatting requirements for proper parsing). As a result, our method is more flexible and easier to adapt to different environments.

In summary, Voyager is specific to the Minecraft environment and cannot generalize to other environments without redesigning or heavily modifying the entire system. In contrast, DynaSaur is designed to be a general, domain-independent framework that can be applied to diverse environments. As such, a direct comparison with Voyager is not applicable in our case.

3. “The choice of retrieval parameter k=10 lacks justification. Could the authors explain this choice? Furthermore, varying k and using different embedding models to evaluate the robustness and efficiency of the cache retrieval mechanism would enhance understanding.”

We thank the reviewer for raising this point. We chose $k=10$ to strike a balance between ensuring that the retrieved toolset includes relevant actions for the current task while keeping the context length manageable. We will clarify this rationale in the revised manuscript. Additionally, we are currently conducting experiments with varying k values and different embedding models to better evaluate the robustness and efficiency of the retrieval mechanism, and will include these results shortly.

4. “There is a concern regarding memory scalability. While the retrieval method helps keep the prompt short, the action library (Ag) appears capable of unlimited growth. Could the authors provide empirical data on the growth rate and final size of Ag after extensive usage (e.g., thousands of episodes), and clarify how this impacts retrieval latency?”

We have addressed the majority of this question in our response to the common concerns above. Unfortunately, running the system for thousands of episodes is economically infeasible given our budget constraints. However, we conducted experiments over 500 episodes on the MATH dataset, which we believe is sufficiently significant to demonstrate that memory usage remains well within manageable bounds.

Regarding retrieval latency, we use Chroma, a production-grade vector database optimized for large-scale retrieval. According to its official benchmarks, Chroma achieves query latencies of approximately 30ms even with databases containing over 500,000 vectors [1], which is sufficient for our current scale and expected growth.

[1] https://docs.trychroma.com/production/administration/performance#latency-and-collection-size

Thank you for your insightful and constructive comments on our paper. We address your questions below:

1. “The action implementation capability relies heavily on the coding proficiency of the backend LLM, which may constrain the framework's applicability across diverse use cases.”

Please see our response to the common concerns above.

2. “Given that some initial actions are derived from AutoGen, it would be beneficial to include AutoGen as a baseline tool for comparison to better contextualize the proposed framework's contributions.”

We appreciate the reviewer’s suggestion to include AutoGen as a baseline. However, we would like to clarify that AutoGen is not itself an agent system, but rather a library for building custom multi-agent workflows. Therefore, a comparison with AutoGen as a baseline agent is unfortunately not feasible.

Dear Reviewer coxz,

Thank you once again for your valuable comments. Regarding generalization to the backend LLM, we have added additional experiments on open-source LLMs, including ones with weaker code generation abilities, in the “Common Concerns” section. These results demonstrate the robustness of our method across a broader range of models, which we believe addresses your concern about the reliance on the coding abilities of the underlying LLMs.

We would be grateful for your thoughts on these new findings and look forward to any further discussion you may have.

5. “The paper mostly tests with GPT-4o, so we don't know if it works with other LLMs, especially open-source ones”

Please see our response to common concerns above.

6. “How does the approach handle incorrect or malicious code generation? What safeguards would be necessary for real-world deployment?”

We addressed this in our Ethics Statement section, which we would like to reiterate here: to handle malicious code generated by LLMs, one can apply a safety filter or a world-model-based formal verifier to each action during creation or prior to execution. Furthermore, the agent should be deployed in an isolated environment with restricted inbound and outbound traffic. Limiting file system permissions, such as enforcing read-only access or encouraging minimal edits instead of overwriting files, can further reduce the risk of unintended or harmful behavior. Restricting the agent’s permissions also helps prevent the execution of malicious scripts.

7. “How well does the approach transfer to open-source LLMs with weaker code generation capabilities?”

Please see our response to common concerns above.

8. “Have you explored using this framework in truly interactive environments (dynamic) rather than static benchmarks?”

As noted in Section 4.1, most existing interactive environments do not permit arbitrary code execution, making it infeasible to run experiments using our framework in those settings.

Although not explicitly stated in the paper, our evaluation methodology inherently involves interactive processes between the agent and its environment. Specifically, our environment is an IPython kernel operating within a computer equipped with internet access and interacting directly with an underlying operating system, as illustrated in Figure 1.

9. “How would this architecture adapt to edge-device (phone, tablet, etc.) deployment where code execution is resource constrained?”

While adapting our architecture to resource-constrained edge devices is slightly beyond the current scope of our work, we envision potential strategies to address such constraints. For instance, explicit constraints on resources (e.g., "ensure code execution does not exceed 5 GB of CPU memory") could be specified beforehand. Alternatively, the framework could incorporate mechanisms to detect resource usage dynamically, alerting the agent when approaching or surpassing resource limits, enabling the agent to adjust its subsequent actions accordingly.

10. “Could this system generate incorrect but plausible functions that mislead the agent? Any checks for semantic validity beyond syntactic correctness?”

As with most code-generation systems, there is indeed a possibility of generating functions that are syntactically correct but semantically flawed or misleading. While our current system primarily checks for syntactic correctness, integrating comprehensive semantic validation remains an open problem and an active area of research. However, our design mitigates this issue somewhat through iterative interactions: incorrect code is usually quickly identified through execution outcomes (e.g., returning a zero or an empty list) and corrected by the agent in subsequent steps. We believe incorporating explicit semantic validation mechanisms or leveraging runtime feedback loops for immediate correction represents promising directions for future enhancement of our approach.

We thank reviewer P4Yg for their thorough and insightful comments on our paper, we would like to address them below:

1. “The fundamental idea of using code generation for LLM agents is not novel. Prior work already explores similar concepts”

Please see our response to common concerns above.

2. “The action retrieval and accumulation mechanisms raise context window and memory concerns as the function set grows. No real analysis of long term usability or degradation.”

For concerns about memory scalability, please refer to our response above, where we provide empirical measurements and discuss growth control strategies.

Regarding the context window, this is precisely the motivation for incorporating an action retrieval mechanism. Instead of appending the entire action set to the prompt, which would scale poorly, the agent has the autonomy to invoke a get_relevant_tools action that retrieves only the top-$k$ relevant actions. This keeps the prompt length effectively bounded, regardless of the size of the full action set.

As for long-term usability and degradation, we address this in Section 4.3.3 through the action coverage metric, which essentially measures how often previously generated actions are reused in successful trajectories. As shown in Figure 3, this action coverage metric increases over time, indicating that accumulated actions remain relevant and useful, with no observable signs of degradation in terms of action coverage.

3. “Limited analysis of the quality of generated functions. Many functions might be too task-specific to be truly reusable”

As mentioned above, in Section 4.3.3, we provide a detailed discussion on reusability, we also include examples of reusable actions in Figure 9. Beyond reusability, we also analyze the types and complexity of the generated actions. Additionally, we include a case study that highlights how our proposed framework offers more flexibility and effectiveness compared to standard agent frameworks with predefined action spaces. If the reviewer had specific aspects in mind that they felt were underexplored, we would greatly appreciate more detailed suggestions to help us strengthen the work further.

4. “No clear comparison with strong baseline methods that use a more comprehensive set of predefined actions”

We would like to emphasize that increasing the comprehensiveness of predefined actions does not address the fundamental limitations highlighted in our paper: (1) it significantly restricts the planning and acting capabilities of LLM agents by limiting them to predefined actions, and (2) it requires extensive human effort to enumerate and implement all possible actions, making this approach impractical for complex environments with a vast number of potential actions. These limitations are precisely what motivate our proposed framework.

Moreover, as described in Section 4.1, we ensure a fair comparison by initializing our action set with the same predefined actions used by the baselines. If a baseline with a more comprehensive action set were to be considered, we would initialize DynaSaur with the same set to maintain consistency.

Dear Reviewer P4Yg,

Thank you once again for your thorough and insightful feedback. In response, we have added additional experiments on open-source LLMs in the “Common Concerns” section. These results demonstrate the robustness of our method across a broader range of model backbones, including those with weaker code generation capabilities.

We would be grateful for your thoughts on these new findings and look forward to any further discussion you may have.

Thank you for the clarifications. I've updated my score.

We thank the reviewer for taking the time to write a positive assessment and express strong support for our paper. We are encouraged by the reviewer’s recognition of our contributions.

Regarding the novelty of our method, please refer to our response to the common concerns above.

We thank the reviewers again for their valuable suggestions. In response, we have conducted additional experiments using open-source LLMs to evaluate:

1. whether our method generalizes beyond proprietary models, and
2. whether it remains effective when paired with models that exhibit weaker code generation abilities.

To this end, we tested two open-source models:

• Qwen2.5-Coder-32B-Instruct: a code-specialized, instruction-tuned model.
• Qwen2.5-32B-Instruct: a general-purpose instruction-tuned model with weaker code generation capabilities.

Qwen2.5-Coder-32B-Instruct

Qwen2.5-32B-Instruct

These results highlight several findings:

1. Open-source compatibility: DynaSaur works effectively with open-source models, not just proprietary LLMs.
2. Robustness to weaker code generators: Even with a general-purpose model like Qwen2.5-32B-Instruct, which has weaker code generation abilities, DynaSaur significantly outperforms other pipelines.
3. Model-wise variability: Surprisingly, Qwen2.5-32B-Instruct even slightly outperforms its code-specialized variant. We hypothesize that its stronger commonsense reasoning and decision-making capabilities may compensate for its weaker coding skills, ultimately resulting in better overall performance.

Taken together, these results further support the robustness and versatility of DynaSaur. We hope they provide a solid basis for the reviewers to consider raising their scores.