Self-Adapting Language Models

We thank the reviewer for their constructive feedback! We will address the proposed weaknesses and questions. 

Response to Weaknesses:

1. Smaller Models Might Not Have Enough Knowledge. We agree that SEAL’s ability to generate useful self-edits is tied to the underlying capabilities of the base model. However, our results show that even small 7B models can substantially benefit, outperforming using data generated by GPT-4.1, despite starting with much lower baseline knowledge. We agree though that starting with a stronger base model would likely greatly improve the ability to self-edit, as scaling increases generation length and the ability to synthesize "new" information in context, given the original piece of data. 

2. Scaling Model Size. To further explore this, we conducted an additional scaling experiment across different Qwen model sizes (3B and 7B). We observed that the difference in performance between the base model and the SEAL model increased with the larger model, indicating that stronger base models are indeed better at generating synthetic data for self-adaptation, and that the benefits of RL increase as well. We will include these results in the revised paper and have provided them at the bottom of this response.

3. Chain-of-Thought. We’ve run experiments incorporating CoT in two different ways. First, in producing a self-edit, we experiment with having the model write a CoT before the generation of synthetic data (see table under response to reviewer nQa9). We found little difference in performance in our document QA domain (39.7% vs 38.7% with base model), likely due to the fact that generating implications of our relatively simple documents require little in-context “reasoning.” In addition, we ran an experiment where given the questions during evaluation, we only do CoT with no training on the original document, to see if CoT can “pull out relevant pieces of information from the LLM,” as you suggested. We believe this intuition makes sense, but did not find any substantial difference in our setting (37.8%).

Response to Questions: 

1. Scaling Model Size. See response to weakness 2.

2. Chain-of-Thought. See response to weakness 3.

3. Qualitative Difference in Self-Edits. We appreciate the suggestion to include more qualitative analysis of self-edits. Our initial inspection of edits did not reveal clear differences beyond length, specificity, and perhaps diversity (as noted in Figure 5), but we agree that more illustrative examples would be helpful. As we note in our response to reviewer nQa9, length clearly does not fully explain the difference. We will expand the appendix with additional qualitative examples and will provide a link to the repo containing all of the data. Furthermore, we show in our response to reviewer hhPg that a proxy reward based on these attributes can achieve good results with much less computation.

Response to Minor Remarks:

1. Clarify Paper Text. We will revise the paper to clearly state that the base LLM used in the QA experiments is Qwen2.5-7B.

Additional Experiment: Scaling Model Size

We rerun our experiments with the 3B-parameter Qwen variant. The results are given below.

We can calculate the relative improvement of SEAL over the base model compared to using self-edits produced by the base model over the base model:

$\$ &\text{3B: }\frac{37.0−25.1}{31.9-25.1}=1.75\\ &\text{7B: }\frac{47.0−32.7}{39.7−32.7}=2.04.
$$

The relative improvement is greater for the 7B model, which provides some evidence that not only are stronger base models more effective at leveraging synthetic data for self-adaptation, but reinforcement learning may have compounding benefits as model capacity increases. We acknowledge that it is hard to draw conclusions though without actually scaling up.

We thank the reviewer for their thought-out response! We address the proposed weaknesses and questions below.

Response to Weaknesses:

1. Generalization. The QA task is meant to simulate general training data. We focused on document QA for its ease of verification, but concede that RL training on only this likely would not generalize to many other types of LM training data. While we don't have experiments specifically on math/code, we believe the method would still work.

2. Larger-Scale Experiments. We have to start somewhere! We've added a larger scale CPT result (all 2067 passages of the validation set, with full finetuning rather than LoRA) to the paper, discussed in the response to reviewer nQa9. This shows the self-edit policy scales well as we increase the amount of ingested data, so long as they are applied in a batch. As we point out in our limitations section though, sequential self-edits (without maintaining optimizer states, replay, etc.) remain prone to catastrophic forgetting.

3. Reward Hacking. We believe rewarding the ability to answer questions about a passage, as well as applying a few-shot rule to a novel input, are fairly robust to reward hacking. Note that these questions are not seen by the model while generating the self-edit. Is there a specific failure case you imagine?

4. Prompting. We've added a much more in-depth analysis across different prompts (see response to review nQa9).

5. Computational Cost. Computational cost is indeed high, but this is a fixed cost. After training the model to perform self-edits (or training a teacher model to generate strong synthetic data), we can then use this model for data generation indefinitely without further expensive meta-learning.

6. Scaling Model Size. We didn't have the capacity to run experiments on 70B models (especially with our meta-learning RL setup). However, we hypothesize that scaling will greatly improve self-editing ability (both before and after RL). In particular, the ability to generate synthetic data given a document is governed by the model's in-context learning ability, and its ability to generate long sequences of data conditioned on the original passage. Scaling increases context/generation length and the ability to synthesize "new" information in context, given the original piece of data and prior knowledge. To provide some evidence that SEAL might show greater benefits over base-model synthetic data, we've run an additional experiment scaling down to 3B. See our results given in the response to reviewer oBBv. The key point is that the the relative increase in performance with SEAL over using the base model for synthetic data generation increases with model size.

7. Self-Edit Formats Predefined. We've added results with no predefined self-edit format, as well as a variety of other prompting formats (see our rebuttal to reviewer nQa9). Predefined prompting formats won out in our experiments, but SEAL improved performance by similar margins no matter the prompt (or lack thereof).

Response to Questions:

1. What Makes a Good Self-Edit. We’ve noted that self-edits in knowledge-incorporation are longer. However, as we explain in the new results and response to reviewer nQa9, this does not account for all of the gains. Qualitatively, in the case of the “implications” prompt, good self-edits seem to contain a long but diverse list of short atomic statements that follow from the base text. We will also expand the appendix with additional qualitative examples and will provide a link to the repo containing all of the data for any future analysis.

2. Proxy Reward. Note that RL would not be done throughout pretraining. We see it more as a warmup such that the model can then self-edit effectively (or produce synthetic data for a student model). However, we think that using a proxy reward models is a good idea! As with any RL setup, we could train a reward model and then use that instead, or we could try to use a faster-to-compute proxy metric for what is a good self-edit. To explore this idea, we've run another experiment using a rubric-based reward, where we query gpt-4.1 to score the length, diversity, quality, and correctness (whether it follows from the contexts) of the response, and use this as a reward rather than our current inner-loop. This is significantly faster than our original inner loop, but performance did not reach the level of the full meta-learning inner loop.

Additional Experiment: Proxy Reward

We experiment with replacing the inner loop with a proxy reward based on a human-crafted and tuned rubric with 4 categories: length, diversity, quality, and correctness. A gpt-4.1 grader model ranked each category on a 1-5 scale, and the sum of these scores was used as the RL reward. The final results are shown below, as well as the overall RL training times.

We believe further rubric/metric tuning could lead to a stronger reward signal. However, the benefit of the full SEAL loop is that this is not necessary, and we directly learn what edits are useful for the base model. We believe both approaches are promising when scaling to larger model sizes and greater compute budgets.

We thank the reviewer for their constructive feedback! We address the reviewers weaknesses and questions below.

Response to Weaknesses/Questions:

1. Prompting With Both Context and Questions. Prompting the model on both C (context) and Q (downstream task) might be useful in some settings, but we wish to be able to self-edit in response to any document C without knowing ahead of time what downstream tasks Q we will encounter. We also agree that our current setup uses task-specific self-edit strategies, while further work might allow for the model to employ an even larger pool of strategies in response to different types of data. One potential extension, as you suggest, is to train across a diverse set of tasks, each with its bespoke self-editing method, and allow the model to learn to select an appropriate self-editing strategy for new tasks.

2. Typo in Formula. Could you clarify which formula you are referring to?

3. Comparison with Related Work. Thank you for the feedback. We’ve added comparisons to a number of recent baselines and prompting methods for generating synthetic data for a task. See the table in our response to reviewer nQa9 for these new results. The key point is that some prompting methods do better than "implications," but RL on top of these prompts improves accuracy further, by similar margins. Entigraph, a human-crafted synthetic data generation pipeline, does comparably to SEAL in our continued pretraining setup. Generative-Adapter, which aims to directly generate the LoRA adapter weights, does better in the single-document regime, but has many other limitations. See our discussion at the bottom of our response to reviewer nQa9.

4. Contribution of Components of Self-Edits. Without training for self-edit generation, performance in the few-shot domain drops to 20%. For each of the parameters such as learning rate, number of epochs, etc., setting them far off the “sweet-spot” leads to a substantial decrease in performance. In this sense, all of these parameters are important. It’s unclear what would be a “fair” point to fix one while letting the model generate the other, or what this would tell us, so we did not run any additional experiments here. These experiments are meant to show that signal from transiently adapted models can reinforce better self-generated hyperparameter selection through RL, which we believe was shown.

5. Theoretical Analysis. While we do not currently provide theoretical contributions, we agree that this represents an interesting direction. Prior work ([1], Yang et al., Synthetic Continued Pretraining) demonstrates from a theoretical perspective how strategically rewriting data can greatly improve the scaling behavior of learning on a set of data. Here, we do not enforce any particular structure such as in [1], so it’s hard to give additional theoretical insight. Instead, we see our method as towards automating the discovery of good synthetic data formats such as that identified in [1].

We thank the reviewer for their attention and helpful feedback! We will address each weakness and question.

Response to weaknesses:

1. Generalization. Our evaluation still uses an unseen test set from the distribution, and our model’s ability to self-edit generalizes within that scope. The general knowledge experiments were meant to simulate learning from arbitrary training data. We focused on document QA for its ease of verification, but concede that RL training on only this likely would not generalize to many other types of LM training data. We also don’t claim that self-editing on ARC tasks would transfer to entirely different few-shot learning tasks. However, we believe that even within these datasets, generalization is a challenge.

2. Prompting & Heuristic Pipelines. This is a great point. We have experimented with a variety of prompts and structured methods for synthetic data generation—ranging from different prompting strategies for SEAL or GPT‑4.1 to using entigraph [1] for continued pretraining—within the general knowledge incorporation domain. The table below includes the additional experimental results, showing that SEAL consistently delivers performance gains across each base prompting strategy, and matches or exceeds [1] for continued pretraining in our setting. This table will be included in the updated version of the paper. 

3. Algorithm Ablations. We chose ReST-EM because it was simple and worked well, and we had a difficult time stabilizing GRPO in our meta-learning setting. However, to scale up SEAL, we agree that moving to more sophisticated RL methods would be beneficial. In the few-shot domain, we already know that augmentation strategies and the optimization parameters make a difference (e.g. moving any optimization parameters off of the sweet spot would lead to worse performance). It’s unclear what would be a “fair” point to fix one while letting the model generate the other, so we did not run any additional experiments here.

4. Hypernetworks. Thank you for pointing out the related work. We will include additional discussion on hypernetworks. We ran Generative Adapter [2] with our evaluation setup and have discussed the results at the bottom of this response.

Response to Questions:

1. Model Collapse. Our experiment in the Limitations section shows that continual gradient steps (without maintaining optimizer states or batching) leads to model collapse. However, batching the updates recovers much of the performance, at least on the task at hand (see table 2 on continual pretraining experiments). Additionally, we’ve run larger continual pretraining experiments with full finetuning rather than LoRA and provided them in the table below. Notably, accuracy does not degrade but rather increases to 58.2% for n=200 and remains high at 46.4% for n=2067 when updates are batched, compared to training and evaluating a separate LoRA adapter for each document (47.0%).

Main Additional Experiments

Since there is no option for a global response, we provide the bulk of our additional results and explanations below, and reference them throughout our individual responses.

Prompting

A central critique was that our self-adaptation procedure is not sufficiently general, or that we did not compare with other methods for generating synthetic data. In the knowledge QA task, we’ve added results showing that SEAL remains effective across a variety of self-editing instructions. This demonstrates that within the general knowledge incorporation domain, SEAL is not tied to a single self-edit format. 

The “implications-long” and “implications-very-long” columns prompt for longer generations.  While none of the reviewers pointed this out explicitly, we wanted to test whether prompting could match the gains from RL, since we saw that RL training naturally increased the lengths of generations (for example, see Figure 5). We find that many prompts achieve higher scores than our main “implications” prompt, but that RL on top of them increased results by similar margins, meaning that the improvement could not simply be attributed to length. 

Here, “chain-of-thought-self-edit” refers to having the model reason before providing the synthetic data, while “chain-of-thought-eval” refers to having the model reason before answering the questions (letting the model “pull out” information from its weights). Both are done with prompting for "implications." The "gpt-4.1 synthetic" column refers to finetuning the base Qwen2.5-7B model using data generated by gpt-4.1 using the corresponding prompt.

Further analysis is given in specific responses to reviewers.

Synthetic Continued Pretraining

Here, we run additional continued pretraining experiments. These experiments are done with the same methodology as the ones in the original paper, except we do full finetuning rather than LoRA. We experiment with n=2067 documents, which is the full validation set. Entigraph [1] has two settings—the second (pairs+triples) creates 10 generations per document, while the first (pairs) and all other methods create 5. 

Surprisingly to us, with full finetuning, we find that performance improved over the n=1 regime in which we RL-trained our model to self-edit. Performance was highest using a batch of n=200 documents, but remained strong at n=2067. We conjecture that this is due to issues with gradient updates over small amounts of data that are encountered with n=1 document, even with strong hyperparameters.

Comparison to Generative Adapter

We ran Generative Adapter [2], a hypernetwork approach to generating a LoRA adapter based on context, with our evaluation setup. All 6 values in the table are on the same evaluation set, but CPT batches updates over all documents while TTT trains an adapter and evaluates each one separately. We used the Mistral-7B-based model for Generative-Adapter, and it achieved a performance of 66.8% on n=1 document, outperforming SEAL, and 28.0% on n=200 continued pretraining, greatly underperforming SEAL. 

However, we would like to highlight some key differences between the methods. SEAL parameterizes weight updates in the generation of training data, which (1) allows for reusing data for continued pretraining or on arbitrary models and (2) allows models to leverage reasoning as documents scale in size and complexity. SEAL is also not restricted to LoRA finetuning, and is much more general in that it potentially allows for many different types of updates given a base document beyond just memorizing a piece of data. For example, perhaps we wish to learn and do a weight-update from an environment/user interaction. It is unclear how hypernetworks could scale to such tasks, while generating data for next-token prediction could leverage the in-context learning abilities of the model.

Thanks for your clear response. You have addressed all my questions, and I will increase my assessment to 5.