Dear reviewer BsWX,

Thank you for engaging in depth with this position paper, and for your many constructive points of feedback! We think that we can address many of them, but are happy to engage in a discussion that can bring the paper closer to the one you would have wanted to read!

As you you said very accurately about a position paper like ours, it’s appropriate to:

judge its thinking relative to prior work: in what sense does this paper enhance the reader's ability to think clearly about issues of superintelligence and recursive self-improvement, relative to the body of literature that already exists

However, we would want to clarify our intended scope up-front: the paper is about recursive self-improvement, yes, but not about superintelligence, AI safety, or the alignment problem more broadly.

Also about scope, we explicitly stated (L34) that we are not aiming for this to be a survey paper (but to stimulate new thinking). That said, with currently 50+ citations drawn from a wide spectrum, the paper clearly does not live in a vacuum, with particularly dense connections to the broader landscape of research in language and RL.

We think this is an appropriate scope, but from your perspective, is this narrower scope valid and useful? And deserving of ICLR visibility?

On Closedness of the system [...]

There seems to be a misunderstanding here. Our aim is not to argue in favour of the closed system scenario, but to assume it and see what can be said about that more restricted setting. Never do we go so far as to claim that Socratic learning (recursive self-improvement in a closed system) is sufficient for ASI. Instead, we try to characterize the ways in which it is limited, fundamentally (and to a lesser extent, pragmatically). One key claim is that “the potential of Socratic learning is high”, which challenges the potential pessimism of whether a “closed system [can] produce open-ended improvement”.

We vividly agree that an "everything framework" defeats the purpose, and we have tried to be disciplined and clear about the restricted types of system we consider. The exception is the caveated statement you highlight: if this was confusing, would it sit better in a footnote?

The definition in the paper seems to be this sentence [...] However, the idea of recursive self-improvement is of course that the system improves itself, which is not part of this definition.

The definition of self-improvement is earlier, in Section 2, which itself is using the definition of “improvement process” in Section 1. The sentence you quote zooms in on an instance of self-improvement that is also recursive.

In general, we think clear definitions are extremely important for a paper like this (and we iterated them many times), so we appreciate the scrutiny!

we can consider ways of recursive self-improvement that do not feed outputs into future inputs (e.g., if the AI rewrites its own training code, then the training code is not necessarily an input of the AI; yet, this would still arguably be recursive self-improvement in the correct philosophical sense.)

People use terms like “recursive” in many ways. For the purpose of our paper, we use one crisp and narrow definition, in order to make some precise follow-up statements. We discuss the broader space of related terms in its dedicated Section 6.

assumes superintelligence will be developed

According to this survey, this is now a consensus view, even if timelines vary. We have added this citation.

"We can motivate this simplification by taking the long view: assuming that compute and memory keep growing exponentially, scale constraints are but a temporary obstacle."

We have revised this to clarify that we need not assume exponential growth: “We motivate this simplification by taking the long view: if compute and memory keep growing exponentially, scale constraints are but a temporary obstacle. If not, considering a resource-constrained scenario for Socratic learning (akin to studying bounded rationality) may still produce valid high-level insights.

If you have additional points that could help reach the community "where it's at", please let us know, we are happy to iterate!

Reviewer BsWX,

Thank you for your engagement and patience!

On the inclusion of non-scholarly materials, we will have to agree to disagree: if the ideas are valid and important, and then someone should put them through the scrutiny of peer-review, and publish them, for everyone's benefit. The specific podcast you linked is interesting, and touches on a number of related themes (AGI, scale, self-improvement), not on others (closed systems, coverage, language games), and makes none of the same claims or arguments as our paper.

I am not aware of any known technique for designing a proxy that can help to achieve broadly superhuman capabilities without human feedback. You have not proposed a design for such a proxy, or even a paradigm for doing so. [...] In what ways are language games a way toward recursive self-improvement? You must have in mind a certain amount of breadth for this claim to become meaningful, and this breadth should be clarified.

Same here, we are also not aware of any existing proxy feedback that could drive broad superhuman capabilities. But what we are aware of (and you have acknowledged as much) are a lot of proxies for narrow capabilities. And the central thing about language games (plural) is that that may be enough: a single agent playing many games that each have narrow but precise and optimizable objectives could attain broad capabilities. This idea is what we tried to convey in the "if you have enough of them" section.

Dear reviewer t9Dm,

Thank you for your constructive feedback, and for the many positive things you had to say about the paper! We think that the criticisms you raise can be addressed, and are eager to engage with you on how to improve the paper in these dimensions.

are definitions in Section 1 all derived from prior work? [...] If not, which are novel ones and which are not? [...] you do not provide any reference in any of the two sections

The definitions are our own, and novel as far as we are aware, but of course closely related to how others are using these terms: unfortunately, as argued here, there are few canonical agent definitions. For our purposes, we wanted to have concise, self-contained definitions that permit the exposition of the remaining arguments, without imposing familiarity with additional formalisms on the reader. But we have now added in a number of additional references in those sections (Abel et al, Colas et al, Fernando et al, Ladosz et al, Lu et al, Pislar et al, Wang et al).

Is this the first work to propose the use of language games for recursive self-improvement?

Yes, we think ours is a novel proposal: if you are aware of this kind of usage of language games, we’d be eager to see!

you could include [...] LLM agent self-playing with the goal of improving in some aspect, e.g., negotiation, debate, fact checking, etc.

We already had pointers to LLM debate examples (Du et al, Liang et al) and included two references to negotiation papers too (Lewis et al, FAIR et al).

I missed a concrete example that connects the described methodology to real-world applications

Our paper is by its nature far removed from practical applications. But we think there are very real problems that Socratic learning can be applicable to. Here is a brief example, which we’ll integrate into the next revision of the paper. Consider a math-centric system:

• The agent reads and writes mathematical statements and proofs.
• The observer’s performance metric is binary: did the agent find a proof for the Riemann hypothesis.
• The system outside the agent contains a proof verifier (eg Lean), and a collection C of theorems and conjectures.
• The proxy reward available to the agent is +1 for each verified new proof of any of the statements in collection C.
• The system allows the agent to formulate new statements, verify its proofs about them, and add them to a second collection L (lemmas, or subgoals).
• Over time, the agent may learn to simplify and decompose existing theorems, accumulate lemmas in L, learn to formulate lemmas that are more and more reusable, and increase the fraction of theorems for which it can produce valid proofs. At some point, the expanding frontier of verified mathematical knowledge (L plus proven statements in C) reaches the Riemann hypothesis, and the observer, satisfied, stops the system.

This example of Socratic learning matches our definition, is open-ended within a closed system, could be non-trivially useful, and it seems not a priori implausible (especially since the AlphaProof results).

What compatible inputs and outputs exactly mean in line 138?

Inputs and outputs are compatible when they live in the same space, and when (at least some) outputs of the agent can be fed back as inputs. For example if both are English text, or both are Python code, or both are formal math statements. As we pointed out in a footnote, input and output spaces are not necessarily identical, but they intersect. For example, the agent could be generating code, but perceive a broader set of things, like natural language, (partly self-generated) code, and execution traces.

How should Socratic learning and the framework you are proposing be modified when the Observer dynamically changes, instead of being stationary as implicitly assumed by the paper?

Intriguing question! Indeed, our framing of how the observer’s performance metric is measured and can improve assumes an observer that does not change their mind about what “good” means. If the observer’s preferences are non-stationary, the problem may be ill-defined, or intractable for an agent in a closed system (as it cannot access any information about how the observer changes). It seems to us that the non-stationary scenario would be better served by an entangled setup with direct feedback, instead of a closed system.

We will expand on all these thoughts in the next revision of the paper, but are happy to discuss any further questions or suggestions you may have!

Dear reviewer t9Dm,

In your original review you stated two concerns. The "lack of a concrete example" has been very directly addressed. The "insufficient covering of prior work" is more debatable, but we have made significant changes in this direction, as well as clarified that not all the terms we define have standard comparable definitions elsewhere. For example about the agent definition we responded that, no, there are not many canonical ones:

The definitions are our own, and novel as far as we are aware, but of course closely related to how others are using these terms: unfortunately, as argued here, there are few canonical agent definitions. For our purposes, we wanted to have concise, self-contained definitions that permit the exposition of the remaining arguments, without imposing familiarity with additional formalisms on the reader. But we have now added in a number of additional references in those sections (Abel et al, Colas et al, Fernando et al, Ladosz et al, Lu et al, Pislar et al, Wang et al).

Also, to be clear, while some of our definitions are novel to the best of our knowledge, we do not claim that the definitions are a contribution in and of themselves: they are only spelled out to make the arguments of the paper as clear and self-contained as possible.

But as always, if you have specific pointers to related definitions or important comparisons, please tell us and we will happily integrate them!

(b contd) Does the fact that coverage might not be satisfied in this example means that recursive self-improvement might just not lead to the agent learning the desired proving skill you describe? If yes, shouldn't this then imply that your Socratic learning approach might not be effective in this example, bc the conditions under which it would be are simply impractical?

Yes, exactly, if the coverage condition is not satisfied, then the system will not reach its goal. Socratic learning can be effective, in this example and in general, when the three conditions hold. For some systems we may be able to make strong statements about those, and hence predict success. For other systems, we may need to fall back on empirical evidence. But even in the empirical scenario, awareness of the three conditions can benefit progress, as compared to only hill-climbing on the external success metric.

I found the response [...] at least inappropriate

We have high respect for the time our reviewers put into this process, and have aimed to respond in an open, collaborative spirit. We believe it is possible to be both direct and respectful (even if a deferential style is common elsewhere).

Dear reviewer 3DHg,

Thank you for your thoughtful and constructive feedback. For a moment, let us suspend the question of what is the right venue for such a position paper (and maybe leave that to the AC – in any case, ICLR can accept position papers, and has in the past, as pointed out by reviewer BsWX), and engage on its substance:

in the absence of such as a known reward function

Yes this precisely what we highlight as one of the two key challenges to recursive self-improvement, in section 2.1. We emphasise the critical importance of setting the system up with a feedback proxy (introduced in L48) that is sufficiently aligned: as you state correctly, without it the agent cannot improve. One type of proxy is a classic (programmatic) reward function coded up by the observer/designer, in which case the system internally does “just” RL, and the agent may or may not learn the behaviour that was intended (cf the large literature on reward hacking). Another form of proxy could be a formal verifier (like the Lean prover in AlphaProof) that can provide grounded feedback on the agent’s progress.

In section 5, we discuss another mechanism for feedback: not one language game, but many language games, each endowed with a scoring function that can drive the agent’s learning. On top of this space of language games sits then another less easily grounded “meta-game” that informs which games the agent plays (based on what playing them and learning from them produces in terms of validation performance, for example). Some of this is already revised for clarity in the latest version (using Carse’s notion of finite and infinite games), but we will iterate further on this.

or human feedback

Our emphasis on a closed system deliberately excludes this kind of learning from the discussion, and zooms in on what is doable without human feedback. In no way do we want to say that human feedback is not useful, or not desirable, but we think the question of what can be attained without it is an interesting one.

Some concrete examples of the concepts put forth here would have been more useful.

Luckily, as this is a non-technical paper, it is not too late for that (and the paper is not currently at its page limit), so we will add one or two worked examples to the next revision.

For a brief example right now, consider a math-centric system:

• The agent reads and writes mathematical statements and proofs.
• The observer’s performance metric is binary: did the agent find a proof for the Riemann hypothesis.
• The system outside the agent contains a proof verifier (eg Lean), and a collection C of theorems and conjectures.
• The proxy reward available to the agent is +1 for each verified new proof of any of the statements in collection C (and 0 otherwise).
• The system allows the agent to formulate new statements, verify its proofs about them, and add them to a second collection L (lemmas, or subgoals).
• Over time, the agent may learn to simplify and decompose existing theorems, accumulate lemmas in L, learn to formulate lemmas that are more and more reusable, and increase the fraction of theorems for which it can produce valid proofs. At some point, the expanding frontier of verified mathematical knowledge (L plus proven statements in C) reaches the Riemann hypothesis, and the observer, satisfied, stops the system.

This example of Socratic learning matches our definition, is open-ended within a closed system, safe and aligned, useful and non-trivial, and it seems not a priori implausible (especially since the AlphaProof results).

Are there more questions that the paper has raised that you’d like to spell out or discuss?

Dear reviewer 3DHg,

I still fail to see the novelty of the contribution if theorem proving (and search?) can be viewed as a valid instantiation of your framework. As I said the paper contains some thought-provoking ideas, but to me a position paper must point out something original that is missed by the community and needs attention.

Whether an idea "needs attention" is inherently somewhat subjective, but if you consider that recursive self-improvement could be or soon become a driver of progress in the community, then ideas closely related to that should be of high interest?

Do we state things that have been "missed by the community"? Unambiguously, yes. We are not aware of any prior work (nor has any been pointed out in this review process) that discusses the limitations of learning within closed systems specifically, or that proposes to use language games for the purpose of recursive self-improvement.

Also, from our earlier response to reviewer BsWX:

• Some readers might think: “scale is enough”, they should learn: “no, you need coverage and feedback too!”
• Some readers might think: “human feedback is necessary”, they should learn: “no, not necessarily, it hinges on the reliability of the proxy: a lot can be achieved within a closed system!”
• Some readers might think: “ok, recursive self-improvement works in principle, but we have no path toward it”, they should learn: “oh, maybe language games are that path!”

Dear reviewer zfgR,

Thank you for your positive and constructive review; it is great to see that our ideas got you thinking, hopefully into new directions!

its suitability for a technical conference like ICLR may require further evaluation

We acknowledge this is a reason for being cautious, but ICLR can accept position papers (and has in the past, as helpfully pointed out by reviewer BsWX), but maybe we can leave that dimension to the AC, and in the meantime work on the substance of the paper?

Socratic learning presupposes that the game’s agent/player is already somewhat intelligent, meaning it can independently derive insights, learn from experiences, and comprehend new knowledge without external input. However, today’s agents, even the most advanced LLMs, lack true intelligence; they generate responses probabilistically rather than through logical understanding. Therefore, should the design of an intelligent agent itself also be considered a prerequisite for Socratic learning?

This is a very insightful suggestion! Indeed, there are conditions on the agent: most obviously, the agent must be capable of learning (otherwise its performance cannot improve). And it must have the capacity to learn a lot, potentially, depending on how demanding the observer is. In broad terms, we included this under the umbrella term of “scale” in section 2.3, and the questions of “which algorithms work” (L125).

While capability to learn is definitely a prerequisite, your second point about “the design of an intelligent agent” is more nuanced: the last decade has produced a lot of evidence that learning from scratch, with zero initial competence, can lead to excellent performance (eg DQN, AlphaZero). But for language domains, the situation is less clear-cut: learning to communicate, let alone full language, is challenging to do from scratch (eg here), and even when it does not start from scratch language can drift away from anything humans can understand (eg here). Hence the temptation to use LLMs as very strong starting points for intelligent agents. We agree with you that LLMs are not sufficient unless they keep learning, and would even venture that it is not obvious that they can learn enough, nor that the community knows how to train them in an open-ended way. But a research program that targets ambitious Socratic learning (instead of pleasing chatbot users) may by necessity lead to the development of such algorithms.

We will expand on all these thoughts in the next revision of the paper, but are happy to discuss any further questions or suggestions you may have!

Dear reviewer 8rsu,

it seems that your primary concern with the paper is that it is a position paper, not a technical paper. Please note that ICLR can accept position papers (and has in the past, as pointed out by reviewer BsWX).

Engaging with it on its own terms: Do you think the ideas we propose are novel, useful, precise, “thought-provoking” as the other reviewers state? Do you think the position is an interesting one, and that our arguments are valid? If not, could you provide constructive feedback? If yes, would you mind adjusting your score?

Dear Authors,

Thank you for your thoughtful response. After careful consideration, I have decided to maintain my original score for this paper. While the work is valuable, I did not find it particularly "thought-provoking" or feel that it addresses a specific gap that is currently missed by the community when evaluated as a position paper.

We have now updated a draft with numerous improvements suggested by the reviewers. The three main changes are improvements to clarity throughout, a worked example of Socratic learning in the domain of maths (end of Section 3), and ~20 additional references to related literature. We're happy to address another round of comments, if needed!

P.S: Just for fun, here is an AI-generated podcast about the paper that we just came across.