Gödel Agent: A Self-Referential Framework Helps for Recursively Self-Improvement

Section 3 is notably small and lacks clarity on the method’s implementation. It would benefit from additional details, particularly on how self-improvement is enacted within the Godel Agent and how evaluations are conducted.

The core idea behind Gödel Agent is to create a self-referential system that can recursively modify its own components for continuous self-improvement. The initial setup focuses on enabling this recursive self-modification capability. Once the system is initialized, the LLM autonomously decides which modules to modify based on feedback from its current performance.

Regarding the self-improvement process, the LLM monitors the rewards from its current policy as well as its internal state information. Based on this feedback, the LLM generates action instructions that can modify various components of the agent, such as its policies, tools, or even its decision-making logic. This recursive process allows Gödel Agent to iteratively optimize itself over time.

As for evaluations, after Gödel Agent has optimized its policy, the resulting algorithm is tested on the target task’s test set. We use standard evaluation metrics such as accuracy or F1 score to assess the performance.

Lines 215-233 imply that evaluations rely on the environments provided. If this involves using the test set for evaluation, it represents a significant limitation of the method.

The agent does not have access to the test data. In fact, during the evolving process, we only utilize validation data, as detailed in Appendix B. To ensure a fair comparison with baseline methods, the selection of validation data follows the approach used in the paper[2]. We use the scores on the validation set as feedback from the environment to guide the Gödel Agent in optimizing its policies.

How the solutions in Godel Agent are evaluated (referenced in line 6 of the pseudocode) needs clearer explanation.

The evaluation of solutions in Gödel Agent is conducted using an environment-provided utility function. Specifically, after Gödel Agent optimizes its policy, the resulting policy (which represents a task-specific algorithm) is tested on the test set of the target task. The utility function assesses the policy’s effectiveness by calculating standard performance metrics such as accuracy or F1 score.

It would be beneficial to compare the cost of the Godel Agent with other methods, such as the CoT, both in terms of time and financial resources. The authors might want to reevaluate their claims and revise the paper to provide a clearer presentation of the proposed method.

Thank you for your suggestion. Our computational cost can be broken down into two parts:

1. Self-Improvement Phase: Gödel Agent incurs additional computational costs during the self-improvement phase, as detailed in Appendix D. Compared to approaches like ADAS, our method significantly reduces these costs by optimizing the agent's capabilities autonomously. While CoT-based methods do not require this self-improvement phase, they rely heavily on human experts for design, which involves a substantial time and financial investment. One of the main motivations for Gödel Agent is to replace this manual effort with computational self-optimization, thereby reducing the need for human intervention.
2. Testing Phase: The testing cost varies across different methods. For instance, methods like CoT-SC and debate require multiple queries and thus have significantly higher computational costs compared to standard CoT. In contrast, Gödel Agent's testing costs generally fall between those of CoT and CoT-SC. Notably, in some cases, Gödel Agent evolves strategies that reduce or even eliminate the need for LLM queries, achieving much lower computational costs.

[1] Goedel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements

[2] Automated Design of Agentic Systems

Thanks for your careful and valuable comments. We will explain your concerns point by point.

There are several ambitious claims within the paper; for example, the abstract suggests that the method can explore the entire space to find the optimal solution. This is a bold statement and does not seem to be substantiated by the findings presented.

In theory, Gödel Agent is designed to explore the solution space and find the optimal solution within the computational resources available, as demonstrated in the Gödel machine paper[1]. However, due to practical limitations, such as computational resource constraints and the inherent instability of LLM decision-making, it is possible that the method may not always achieve the theoretical optimal in real-world applications.

We appreciate your critique, and we will revise the paper to express this point more clearly and cautiously. Specifically, we will emphasize that while Gödel Agent can theoretically explore and optimize solutions within the available resources, practical challenges may affect the extent to which this optimality is achieved in real-world scenarios.

The central claim about the capability to create "different agents" from an initial agent mostly involves modifying the conditioning of the autoregressive order through prompt engineering.

It seems there might be a misunderstanding regarding the core mechanism of Gödel Agent. To clarify, we do not create "different agents" through prompt engineering. Rather, there is a single Gödel Agent that continuously refines and improves itself over multiple iterations.

The key difference is that Gödel Agent does not simply adjust its behavior through prompt modifications. Instead, it engages in a recursive process where the agent modifies its own underlying code to optimize its capabilities. For example, the initial agent $a_0$ uses its current state to improve itself, resulting in $a_1$. Then, $a_1$ uses its enhanced capabilities to further refine itself into $a_2$, and so on. This self-referential and recursive improvement process goes beyond trivial prompt adjustments—it fundamentally changes the agent’s optimization abilities over time.

Figure 1 shows some learnable components, yet the framework itself does not appear to incorporate learnable elements.

To clarify, Gödel Agent’s framework indeed incorporates learnable elements. The action space, tools, and policies are not static but are dynamically modifiable by the agent. Gödel Agent can autonomously refine its action space, create and adjust its tools, and optimize its policies based on feedback during its self-improvement process. These components are effectively learnable because they can be adapted and optimized over time as the agent evolves.

Line 50 claims "to eliminate the human design prior," which is challenging given that LLMs inherently contain human priors from being trained on human-generated text.

You are absolutely correct that LLMs inherently carry human priors because they are trained on human-generated text. However, in line 50, our intention was to convey that we aim to reduce the human priors specifically in the design of the agent’s logic, strategies, and decision-making processes. While LLMs themselves are inevitably influenced by their training data, our goal is to allow the agent to operate autonomously, minimizing manual human intervention in its policy design. In other words, while the underlying LLM contains human priors, the agent’s higher-level decision-making and optimization processes are designed to evolve independently, free from predefined human-designed rules or biases. We acknowledge that complete elimination of human influence is not feasible due to the nature of LLMs, but our focus is on reducing human biases in the agent’s self-improvement framework.

Discussions on self-awareness of LLMs are speculative and might be better left out of a scholarly paper focused on introducing a new method.

Thank you for your thoughtful comment and suggestion. We apologize for any confusion caused by our use of the term "self-awareness." In this context, we intended to convey that Gödel Agent has the capability to introspect and read its own code and files, not to imply any philosophical sense of consciousness or awareness. We have emphasized this point more clearly in the revised version of the paper. If you find that the current explanation is still not sufficiently clear, would using the term of self-introspection be more appropriate?

Thanks for your insightful comments. We are happy to discuss with you about your points.

1. Why LLM-based agent is a valid approach? No LLM is perfect. All LLMs make mistakes. What is reliable information? What is ground truth? Then why should we rely on LLMs to make decisions? For software engineering, no LLM can achieve a pass@1 of 100%. To make it worse, the popular evaluation method with several tests, is not perfect, so the evaluation results are not fully reliable.

2. How to make LLM-based agent a valid approach? LLMs may help generate plausible solutions. But there should be a mechanism to filter out or fix incorrect solutions.

How to improve an LLM-based agent? It seems this boils down to waiting for stronger LLMs, which is likely out of the scope of the paper, though.

1. There are papers discussing / criticizing the reasoning / planning capacity of LLMs, which is the foundation of LLM-based agent (including those based on the Gödel machine), e.g. 1) LLMs can’t plan, but can help planning in LLM-modulo frameworks, in ICML, 2024. 2) GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. arXiv 2024

Without sound capabilities for reasoning / planning, how to build LLM-based agents, in particular, the Gödel Agent?

It appears that empirical improvements are not enough to answer the above questions.

We fully acknowledge that LLMs have limitations, make mistakes, and have significant room for improvement. However, it is also undeniable that LLMs have achieved remarkable results across many tasks and continue to improve. In our work, we propose utilizing LLMs to enable Gödel Agent’s self-improvement, focusing on leveraging their strengths rather than dwelling on their imperfections.

Gödel Agent’s approach doesn’t rely solely on LLMs for decision-making. Instead, it incorporates LLMs as part of a recursive, self-improvement process where the agent can autonomously refine its policies and improve its performance over time. This self-referential mechanism allows Gödel to evolve beyond the limitations of the initial LLM and address its shortcomings through continuous self-modification.

We also want to emphasize that our framework is not tied exclusively to LLMs as decision tools. If alternative methods for decision-making exist, our framework is flexible enough to incorporate them, which makes it adaptable to different settings and tools.

We ask that you focus on evaluating the innovative contributions of our framework and the novel approach it offers for agent self-improvement. While we are open to further discussion of LLMs' limitations, we kindly request that you reconsider the evaluation of our paper based on the broader contributions of our approach, as it provides a unique perspective on agent self-improvement. Thank you again for your feedback.

• Reliable Signals: Gödel Agent uses performance on the validation set as a reliable optimization signal. This signal is quantitative, task-specific, and derived directly from the agent’s actions, ensuring relevance to the optimization process. Experimental results in Section 5 demonstrate that Gödel Agent can effectively improve its strategies based on this feedback.
• Cost of Interactions: Gödel Agent minimizes the cost of interactions by requiring only one interaction per task. Once a policy is optimized, it can be reused for repeated task executions without additional interactions. As shown in Appendix D, the experimental costs are manageable and not prohibitive.
• Dependency on LLMs: Gödel Agent’s reliance on LLMs reflects the current state of the field, where LLMs are central to many state-of-the-art AI systems. This dependency is not unique to Gödel Agent but is shared by a majority of cutting-edge methods.

We believe Gödel Agent represents a novel and significant step forward, particularly in its ability to self-introspect, adapt, and optimize autonomously. We encourage the reviewer to consider these contributions and their potential implications. A fair and thoughtful assessment of the work’s value would be beneficial.

If further clarification or additional evidence is needed, we are more than happy to provide it. We sincerely appreciate your effort in reviewing our work.

From the Abstract: "Gödel Agent leverages LLMs to dynamically modify its own logic and behavior, guided solely by high-level objectives through prompting." My understanding is that LLMs are essential to Gödel Agent.

I adjust the score a little bit, to appreciate the authors' hard work. However, I'm concerned with how the LLM community is approaching AI agents, as discussed earlier.

Thank you for adjusting the score! This work represents our first step in exploring the framework of self-referential agents, and we are committed to further advancing this direction in future research. We greatly appreciate your insights regarding reasoning and planning with LLMs, and we will carefully consider the role of LLMs in agent frameworks as we move forward. Thank you again for your valuable input.

Thanks for your careful and valuable comments. We will explain your concerns point by point.

The motivation for adding four additional tools is not well-justified; there is no evidence supporting that these are the correct tools to add or that they comprehensively cover all needs. It's counterintuitive because if we claim the model has the ability to refine its policy and develop new policies, why do we need to provide tool guidance? Additionally, it's notable in Appendix C that although there is variability in the best policy, this policy seems can be decomposed into a combination of the listed tools, which suggests this work is not fully distinguished from meta-learning optimized agents where humans manually define the high-level policy, and the agent learn a strategy to combine or rank the high-level strategy.

The motivation for initially providing Gödel Agent with four tools (Chain-of-Thought, error handling, code execution, and API calls) stems from the current limitations of existing LLMs. While Gödel Agent is designed to autonomously refine its strategies and develop new tools, the capabilities of current LLMs are not yet sufficient to handle more complex tasks without additional support. The inclusion of these tools serves to enhance the agent’s initial performance and practicality, given these constraints.

It’s important to note that these tools are not hard-coded limitations. Gödel Agent has the flexibility to modify, extend, or even replace these tools as it optimizes itself. The tools provided are simply starting points to bridge current limitations, allowing Gödel to function effectively in its current environment. Over time, Gödel can autonomously evolve beyond the initial toolset as its capabilities improve.

Regarding the concern that our approach is similar to meta-learning agents where humans define high-level strategies, we would like to emphasize a key distinction: Gödel Agent is not limited to combining predefined strategies. For instance, in the "Game of 24" task (Section 5.3), Gödel Agent developed a completely new search-based algorithm that achieved 100% accuracy. This represents a qualitative leap beyond what Meta Agent Search can achieve, as the latter is constrained by predefined search spaces and prompt-based strategies.

The current framework heavily relies on the model's capability to generate and refine policies, as well as generate responses given a policy and query. However, this would not improve the model's ability to acquire new knowledge if such kind of off-line dataset available.

The performance of Gödel Agent is indeed dependent on the capabilities of the underlying LLM to recognize that modifying its own code and structure will directly impact its performance.

However, Gödel Agent does have the potential to surpass the limitations of the base LLM. If the LLM is advanced enough to understand how its own parameters influence the agent’s overall performance, Gödel could theoretically go further by generating code to fine-tune or even replace the underlying LLM. This self-modification could enable the agent to evolve beyond the capabilities of the original LLM, effectively enhancing its performance beyond its initial constraints.

The current refinement requires an oracle to progress the performance (U in line 6 from Algorithm 1). For the current Table 1, is the number of oracle utility function calls the same across all methods, and is the number of model queries the same for different settings?

Regarding the use of the oracle utility function, it is true that our method relies on an environment-provided utility function for optimization. During this phase, Gödel Agent autonomously decides when to call the utility function to obtain rewards on the validation set, similar to how human experts adjust their methods based on validation feedback. However, once the policy optimization is complete, the final policy uses the utility function in the same way as other methods, ensuring consistency across approaches in the evaluation phase.

Regarding the number of model queries, it is indeed different across various methods. This variation is inherent due to the nature of each approach. For instance, CoT only requires a single query, while methods like CoT-SC and debate require multiple queries to achieve their results. As the framework we propose focuses on self-improvement, our primary concern is evaluating the effectiveness of each method rather than strictly equalizing the number of queries.

Based on Appendix C, the best policy found by Godel Agent for MMLU is the same as CoT-SC, why are the two numbers reported in Table 1 different? Similarly for the best policy for GPQA, why is it different from CoT? If this is due to extended few-shot examples or specific instructions, then it seems more like a prompt engineering problem, suggesting we're not using optimal prompts.

You are correct that the best policies found by Gödel Agent for MMLU and GPQA are similar in methodology to CoT-SC and CoT, respectively. The differences in the reported performance in Table 1 indeed stem from Gödel Agent’s use of extended few-shot examples and specific instructions.

However, we do not view this as a limitation. On the contrary, this highlights Gödel Agent’s capability to autonomously optimize prompts, which includes fine-tuning few-shot examples and refining specific instructions to achieve better performance. If a human were to manually adjust prompts to optimize results, it would be a time-consuming and labor-intensive process. Gödel Agent, on the other hand, integrates this prompt optimization seamlessly into its self-improvement process. Thus, Gödel Agent is not only optimizing policies from a methodological standpoint but also refining prompts in a way that enhances performance, as demonstrated by our experimental results.

The reason why the method is evaluated on MGSM instead GSM8K is not well-justified, for my understanding, multi-lingual is not a target of the framework. Moreover, all current benchmark, even regular CoT has high accuracy, this together with limitation, I am questioning the generalizability of the framework on harder tasks, like MATH, MATH500,omni-MATH and SWEBench.

From a theoretical perspective, our framework is task-agnostic—provided that the environment can provide feedback, Gödel Agent can adaptively improve itself, making it applicable to both controlled and more complex real-world tasks, including embodied environments. However, we fully agree that experiments on more complex, real-world tasks would provide stronger evidence of the framework’s scalability. However, we chose the current set of tasks for the following reasons:

1. The baseline methods we are comparing against have only been tested on these controlled tasks, so using the same tasks ensures a fair and consistent comparison.
2. As this is the first proposal of such a framework, we wanted to demonstrate its feasibility and core capabilities on representative tasks before extending it to more complex scenarios. More complex tasks might distract from highlighting the novel aspects of our framework at this stage as they need more engineering work.

For the related work, some recent recursive self-improvement work is not discussed.

Thank you for pointing out these recent works. Below, we compare these approaches to our Gödel Agent and we have added them in our current version:

1. Glore proposes SORMs to improve LLM reasoning through global and local refinements, using synthetic data to predict correctness of solutions. It focuses on identifying when and where to refine reasoning steps but does not involve modifying the model's architecture or self-improvement mechanisms like Gödel Agent.
2. V-star trains a verifier to evaluate both correct and incorrect self-generated solutions, iteratively improving reasoning accuracy. Unlike Gödel Agent, which autonomously modifies its own architecture, V-star uses external verifiers to guide the refinement of reasoning.
3. RISE enables recursive self-improvement by fine-tuning models to introspect and correct previous mistakes in multiple iterations. Gödel Agent differs by incorporating a recursive mechanism that includes altering the architecture, not just refining responses.
4. SCoRe uses reinforcement learning to improve self-correction in LLMs by learning from self-generated correction traces, avoiding issues like behavior collapse.

How to safeguard the current workflow if the model is allowed to modify the policy and interaction code directly, and the policy will be used to generate further responses. How can we prevent runtime memory issues and ensure that both the code implementation and the final generation are not harmful?

Ensuring the safety of Gödel Agent during self-modification is indeed a top priority. To prevent potential misuse of resources, we recommend running Gödel Agent within a sandboxed environment where system permissions and resource access are tightly controlled. We have added an Ethics Statement at the end of the paper. Thank you for your suggestion!

We hope that our responses have addressed your questions and concerns. We also hope that you find our approach exciting, as it represents a fundamental departure from previous methods. If you have any further questions, please feel free to reach out, and we will be happy to address them. Thank you once again for your valuable feedback.

It's important to note that these tools are not hard-coded limitations. The Gödel Agent has the flexibility to modify, extend, or even replace these tools as it optimizes itself.

While this potential flexibility is indeed intriguing, the current experimental results do not fully demonstrate this capability. With the exception of the "Game of 24" (which I will address separately), the observed behaviors primarily consist of combinations of the predefined tools detailed in Appendix C, rather than novel extensions or modifications.

Regarding the "Game of 24" example, I have concerns about the claim that it "develops new strategies" beyond the predefined tools. Upon investigation, I found striking similarities between the generated algorithm and existing solutions. Specifically, when inputting the same problem from LeetCode (https://leetcode.com/problems/24-game/description/) into ChatGPT, the returned solution (https://chatgpt.com/share/67390c11-c9d0-8009-bd2d-ee221cc372fc) is remarkably similar, sharing even specific implementation details such as the 1e-6 threshold. This suggests that the model may be leveraging pre-trained knowledge rather than developing novel strategies through the proposed mechanism.

These observations lead me to question whether the results presented in the paper fully support the authors' claims about the system's capabilities.

If the LLM is advanced enough to understand how its own parameters influence the agent's overall performance, Gödel could theoretically go further by generating code to fine-tune or even replace the underlying LLM.

This theoretical possibility faces significant practical challenges. Given that even human experts currently struggle to fully understand how model parameters influence agent performance, it may not be prudent to base the method's validity on capabilities that require such advanced developments.

Regarding the use of the oracle utility function, it is true that our method relies on an environment-provided utility function for optimization.

There appears to be an important distinction between two different applications of the oracle utility function that warrants clarification. While both the final policy generated by the Gödel agent and alternative strategies require oracle utility function calls during evaluation, the Gödel approach uniquely requires multiple calls during the policy development phase itself. This additional requirement during development could present significant practical limitations for general task applications, especially when compared to approaches like Chain of Thought (CoT) that do not need oracle utility function calls during their development phase.

The differences in the reported performance in Table 1 indeed stem from Gödel Agent's use of extended few-shot examples and specific instructions.

The baseline methods we are comparing against have only been tested on these controlled tasks, so using the same tasks ensures a fair and consistent comparison.

I would like to seek clarification on two aspects of the experimental setup and baseline comparisons:

Regarding dataset selection: The baseline methods cited in this paper (COT-SC, Self-Refine, LLM debate, Step-back-Abs) were actually evaluated on datasets GSM8K instead of MGSM. Could the authors explain the rationale behind this dataset selection?

Concerning baseline performance: I notice that the baseline performance metrics reported in Table 1 are lower than those published in their respective original papers. This brings me to question about prompt optimization - could the authors clarify whether they used the optimal prompts from the original papers for these baseline implementations?

The clarification on related work and safeguards makes sense to me, thanks!

More detailed comparison to ADAS is needed in related work (Automated Design of Agentic Systems) What is Meta Agent Search and why does this Godel agent perform better?

Thank you for the opportunity to clarify the comparison between our method and Meta Agent Search (MAS). In MAS, a fixed meta-agent is manually designed to create task-specific agents by interacting with the environment. However, this meta-agent remains static throughout the process, meaning the final optimized results are inherently constrained by the initial design and capabilities of the meta-agent.

In contrast, our approach, inspired by the Gödel machine, moves beyond the meta-learning paradigm by endowing the agent with self-referential capabilities. This allows the agent to evolve not only its task-specific part but also its own optimization part. As a result, both the agent's ability to optimize and its performance on downstream tasks improve simultaneously, leading to more efficient and effective solutions.

A clear example of this qualitative improvement can be seen in the "Game of 24" task as shown in Appendix C.2. Gödel Agent was able to design a search-based algorithm that solves the task with 100% accuracy, whereas MAS is limited to searching within predefined spaces and designing prompt-based methods. This shows that Gödel Agent can generate solutions that are not only more flexible but also fundamentally more capable than those produced by a fixed meta-agent.

The method is somewhat vague and not clear what’s going on, and what this has to do with self reference/recursion. Is this different than just tasking an LLM with modifying its own agent code?

In Gödel Agent, the concepts of self-reference and recursion refer to the agent's ability to autonomously modify its own code to improve both its optimization process and task-solving capabilities. The key difference lies in the fact that after each self-modification, Gödel Agent's optimization abilities themselves are enhanced. This creates a loop where improved optimization leads to more effective self-modification, which is recursively applied to the agent itself rather than an external entity.

This process is fundamentally different from simply tasking an LLM with modifying its own agent code, as seen in methods like ADAS. In those cases, the LLM’s optimization capability remains static, meaning the changes are limited to task-specific adjustments without fundamentally enhancing the LLM's underlying optimization skills.

How does Gödel Agent ensure safety and prevent harmful behaviors during self-modification? How do you put constraints on it such that it doesn't blow all of your GPT credits or hog all the GPUs on your system?

Ensuring the safety of Gödel Agent during self-modification is indeed a top priority. To prevent potential misuse of resources, we recommend running Gödel Agent within a sandboxed environment where system permissions and resource access are tightly controlled. We have added an Ethics Statement at the end of the paper. Thank you for your suggestion!

What mechanisms are in place to handle errors or prevent the agent from degrading its performance over time?

Currently, Gödel Agent is equipped with an error feedback mechanism: when it performs an action that is invalid or non-functional, the system returns an error message. Gödel Agent uses this feedback to adjust its behavior and refine its strategies.

Regarding performance degradation, we have chosen not to implement additional manual corrections. Instead, we rely on Gödel Agent’s ability to reflect on its actions and autonomously adjust its strategies over time. While it is technically possible to manually evaluate the strategy at each iteration and prevent modifications if performance declines, we have opted against this approach to minimize human intervention.

Moreover, we believe that allowing the agent to occasionally make mistakes can be beneficial. By experiencing negative outcomes, Gödel Agent can learn which modifications are counterproductive, which can inform better decision-making in the future.

Can this approach scale to more complex tasks that require long-term planning or interaction with unpredictable environments?

We believe that Gödel Agent’s approach can indeed scale to more complex tasks that require long-term planning and interaction with unpredictable environments. The key factor is the availability of feedback or rewards from the environment. As long as the environment can provide such feedback, Gödel Agent is capable of optimizing its strategies to adapt to the given task.

How does the agent's performance depend on the underlying LLM's capabilities? Can it surpass those limitations?

The performance of Gödel Agent is indeed dependent on the capabilities of the underlying LLM, particularly in terms of its decision-making, planning, and its ability to understand that it is part of Gödel Agent. The agent relies on the LLM to recognize that modifying its own code and structure will directly impact its performance.

Regarding the second part of your question, Gödel Agent does have the potential to surpass the limitations of the base LLM. If the LLM is advanced enough to understand how its own parameters influence the agent’s overall performance, it could theoretically go further by generating code to fine-tune or even replace the underlying LLM. This self-modification could enable the agent to evolve beyond the capabilities of the original LLM, effectively enhancing its performance beyond its initial constraints.

We hope that our responses have addressed your questions and concerns. If you have any further questions, please feel free to reach out, and we will be happy to address them. Thank you once again for your valuable feedback.

Thanks for your careful and valuable comments. We will explain your concerns point by point.

Experiments seem limited in scope—mostly on controlled tasks; might not scale to complex real-world applications or embodied tasks

Thank you for your valuable feedback. From a theoretical perspective, our framework is task-agnostic—provided that the environment can provide feedback, Gödel Agent can adaptively improve itself, making it applicable to both controlled and more complex real-world tasks, including embodied environments. However, we fully agree that experiments on more complex, real-world tasks would provide stronger evidence of the framework’s scalability. However, we chose the current set of tasks for the following reasons:

1. The baseline methods we are comparing against have only been tested on these controlled tasks, so using the same tasks ensures a fair and consistent comparison.
2. As this is the first proposal of such a framework, we wanted to demonstrate its feasibility and core capabilities on representative tasks before extending it to more complex scenarios. More complex tasks might distract from highlighting the novel aspects of our framework at this stage as they need more engineering work.

The final policies that are returned by the method don't seem very complicated or different from a basic human designed template. The agent designs discovered in the ADAS paper seem much more complex and creative. What explains this difference?

ADAS operates within a predefined search space, limited by human-designed methods, such as CoT and Debate. These algorithms are combined to create a solution, but the underlying structure is based on existing human knowledge. In contrast, Gödel Agent is not bound by such human priors. It is free to explore and design strategies that go beyond predefined templates, as demonstrated in the "Game of 24" task in Appendix C.2. In this task, Gödel Agent independently chose to use a search-based algorithm, breaking away from the limitations of prompt-based methods, as detailed in Section 5.3. While the policies Gödel Agent generates may not always seem highly complex in every instance, its full autonomy dictates that it has a higher ceiling and will get stronger as LLM grows.

Given the limited evaluations, the room for self-improvement seems limited, can we really get much better than the base model for mathematical reasoning with fancy agent pipelines and prompts? A better showcase of the method would be on openended or embodied tasks vs text based reasoning

We agree that the full potential of Gödel Agent will be best demonstrated in more complex, real-world scenarios, such as open-ended or embodied tasks. We are excited about the possibility of applying our framework to these types of tasks in the future. Currently, our experiments have focused on text-based reasoning tasks to establish the feasibility and advanced capabilities of Gödel Agent in a controlled setting. While we recognize that this scope is limited, it was an essential first step in proving the underlying framework's effectiveness.

What’s the upper limit of self improvement, does it saturate? *Whats the improvement in the 6 iterations

Thank you for your thoughtful question. The concept of recursive self-improvement does suggest that the process could continue indefinitely, as each iteration improves the optimizer, which in turn enhances the agent's ability to improve itself. However, in practice, we believe that there is an upper limit to this process. On the one hand, the core of Gödel Agent is based on a LLM, and the LLM’s understanding is inherently limited. If the agent becomes too complex, the LLM may no longer be able to comprehend or process its own changes, at which point the self-improvement process would stagnate. On the other hand, LLMs have some degree of randomness and are prone to errors, meaning that the optimization process may encounter hurdles that prevent indefinite improvement.

Figure 4 in our paper shows an example of how the strategy changes as the number of iteration steps increases.

Thanks for your careful and valuable comments. We will explain your concerns point by point.

Does the learning agent ever see the data the final policy will be evaluated on?

The agent does not have access to the test data. In fact, during the evolving process, we only utilize validation data, as detailed in Appendix B. To ensure a fair comparison with baseline methods, the selection of valid data follows the approach used in the paper[1]. We use the scores on the valid set as feedback from the environment to guide the Gödel Agent in optimizing its policies.

For how many iterations do you run the Godel agent and how did you determine this hyperparameter?

As stated in line 310 of the paper, we conducted 6 independent self-improvement cycles for each task, with a maximum of 30 iterations per cycle. However, based on our experimental observations, Gödel Agent often stops iterating before reaching this limit. The choice of 30 iterations as a hyperparameter was determined empirically through multiple self-improvement trials. Specifically, in 50 experiments conducted on MGSM, we observed that in 42 cases, Gödel Agent terminated early, deeming its current policy sufficiently optimized before reaching 30 iterations.

In general, how did you tune prompt variations and hyperparameters for this agent? How did you prevent overfitting to the test data?

Thank you for this question. Due to Gödel Agent's capability for unrestricted self-modification, there was no need for manual tuning of prompts or other hyperparameters. During the self-improvement process, the agent autonomously optimizes these aspects. This self-optimization ability is a key advantage of our approach compared to others. Consequently, the agent does not overfit the test data, as all adjustments are driven internally by the agent based solely on its interactions with the validation set.

Could you please clarify the action space of the agent and how the agent learns about the possible interactions? Is it only given the prompt in Appendix A or is it also given few-shot samples for example?

The action space of Gödel Agent evolves continuously as it undergoes self-optimization. Initially, as described in Section 3 of the paper, the action space includes: (1) reading its own code, (2) modifying its own code, (3) evaluating its current policy using a utility function, and (4) recursively entering the next level of self-improvement. These actions were deliberately designed to align with the core capabilities of Gödel Agent.

During the self-improvement process, Gödel Agent is free to create new actions and tools, which means its action space is not fixed but rather expands dynamically based on its needs. For example, in our experiments, Gödel Agent autonomously developed tools such as numerical parsers, calculators, and even assistant agents.

Regarding how the agent learns about possible interactions, this is facilitated through the initial prompt provided in Appendix A. The prompt sets up the foundational understanding, encouraging the agent to analyze, explore, and construct new actions. However, we deliberately avoided using few-shot because they could introduce human priors and influence agent’s behavior. Our approach is to minimize human intervention.

How does the agent execute the policy? Is this happening automatically each iteration, or does the agent manually chose to exit the policy? Is this different during iteration and during evaluation?

Thank you for this insightful question. The execution of the policy is not automatic at every iteration. Instead, testing the policy using the utility function is treated as one of the actions available to the agent. During the iterative self-improvement phase, Gödel Agent autonomously decides whether or not to test its current policy based on its assessment. It does not execute the policy at every step but rather chooses when it is optimal to do so.

In contrast, during the final evaluation phase, the process is different: we manually invoke the utility function to test the agent’s final policy. This ensures that the evaluation is consistent and unbiased. Thus, the approach during iteration—where the agent makes autonomous decisions—is distinct from the controlled, manual process used during the final evaluation.

Clarify the evaluation protocol

As mentioned earlier, each task involves a utility function that helps evaluate the agent’s policy. During the iterative process, Gödel Agent decides autonomously when to invoke this utility function to test its current policy. In the final evaluation phase, we manually test the final policy using the same utility function.

A key distinction between iteration and evaluation is the data used: during the iterative process, the utility function is applied to the valida data to guide the agent's self-improvement. In contrast, during the evaluation phase, we use test data to assess the performance of the agent’s final policy.

Justify the choice of having baselines that use less inference compute. Provide a best-of-n baseline or argue why this is not a good comparison.

Thank you for your question. It seems there may be a misunderstanding about the computation cost of our method. During the optimization iterations, Gödel Agent does not generate a new strategy or test it at every step. Instead, it selectively optimizes its policy at certain steps and evaluates it based on feedback from the validation set. The specific steps where this happens are determined autonomously by the agent.

In terms of inference computation, the cost is similar to that of the baselines because we only test the final optimized policy once. Therefore, the additional computational cost is primarily associated with the optimization process, where Gödel Agent uses this compute to replace the human labor typically involved in manually designing and refining agent strategies. Moreover, for each task, only a one-time optimization process is required, and the generated policy can always be applied on that task, so we consider this consumption acceptable.

The baseline methods, on the other hand, are manually designed by human experts, which consumes significant time and resources. Thus, while Gödel Agent uses more compute during optimization, it eliminates the need for manual design, which is far more resource-intensive.

Clarify the comparison to Meta Agent Search and wheather your method produces qualitatively better solutions.

Thank you for the opportunity to clarify the comparison between our method and Meta Agent Search (MAS). In MAS, a fixed meta-agent is manually designed to create task-specific agents by interacting with the environment. However, this meta-agent remains static throughout the process, meaning the final optimized results are inherently constrained by the initial design and capabilities of the meta-agent.

In contrast, our approach, inspired by the Gödel machine, moves beyond the meta-learning paradigm by endowing the agent with self-referential capabilities. This allows the agent to evolve not only its task-specific part but also its own optimization part. As a result, both the agent's ability to optimize and its performance on downstream tasks improve simultaneously, leading to more efficient and effective solutions.

A clear example of this qualitative improvement can be seen in the "Game of 24" task as shown in Appendix C.2. Gödel Agent was able to design a search-based algorithm that solves the task with 100% accuracy, whereas MAS is limited to searching within predefined spaces and designing prompt-based methods. This shows that it can generate solutions that are not only more flexible but also fundamentally more capable than those produced by a fixed meta-agent.

Justify the choice to use GPT-3.5 for the policy models. If possible, provide some results using GPT-4 for the policies.

We chose to use GPT-4 only during the learning phase because it offers superior reasoning capabilities, which are essential for the self-improvement process where the agent generates and refines its strategies. This phase involves replacing human labor with autonomous optimization, and GPT-4's enhanced reasoning helps Gödel Agent improve its policy efficiently.

For task execution, we opted for GPT-3.5 for all methods because, in real-world applications, executing policies often requires frequent API calls, and GPT-3.5 is significantly more cost-effective for this purpose.

The policy generated by Gödel Agent is fixed after the learning phase, much like CoT, where the learned policy remains unchanged during execution. Therefore, the decision to use GPT-3.5 for execution maintains fairness in comparison.

The paper often uses the term "self-awareness" without qualification. I'm concerned that anthropomorphization leads to a less scientific discussions and I would recommend the authors try to rewrite the paper to remove phrases like "Our Godel Agent achieves self-awareness".

Thank you for your thoughtful comment and suggestion. We apologize for any confusion caused by our use of the term "self-awareness." In this context, we intended to convey that Gödel Agent has the capability to introspect and read its own code and files, not to imply any philosophical sense of consciousness or awareness. We have emphasized this point more clearly in the revised version of the paper. If you find that the current explanation is still not sufficiently clear, would using the term of self-introspection be more appropriate?

We hope that our responses have addressed your questions. We also hope that you find our approach exciting, as it represents a fundamental departure from previous methods. If you have any further questions, please feel free to reach out, and we will be happy to address them. Thank you once again for your valuable feedback.

[1] Automated Design of Agentic Systems