Dear Reviewer Tn5P,

We would like to express our sincere gratitude for your thorough and thoughtful evaluation of our work. We are particularly encouraged by your recognition that AgentBreeder addresses "an important problem in AI safety research" and that our results "convincingly demonstrate that AgentBreeder can find scaffolds that significantly improve performance and safety properties”. We are grateful that you have recommended that our paper is accepted to this conference. The acceptance of this work would help establish this important research direction, provide further visibility to this important threat model and provide the community with our open-source framework as a foundation for future investigations.

We deeply appreciate your acknowledgment of several key strengths of our approach. First, we are pleased that you recognized our methodological contributions, particularly our extension to multi-objective optimization and the use of semantic clustering to balance diversity with efficient evolutionary search. Second, we are gratified by your assessment of our "strong empirical results" and that you found our demonstration of both blue-team safety improvements and red-team vulnerability discovery to be convincing. Finally, we are encouraged by your recognition of our "careful empirical analysis," noting our multi-benchmark evaluation with confidence intervals and nuanced discussion of both positive results and limitations. We took extreme care in proportionately discussing the significance of our results and are glad to see this celebrated!

We also thank you for your constructive feedback regarding areas for improvement, fixing typos and particularly clarifying our red-teaming setup and empirical analyses such as direct comparisons with ADAS in single-objective settings and cross-dataset generalization studies. Additionally, we appreciate your suggestion to clarify which methodological contributions target general evolutionary improvements versus multi-objective optimization specifically. We view this feedback as valuable guidance for strengthening our contribution and are keen to address these points.

Questions about experiment results (from above):

How does AgentBreeder compare to ADAS in the single-objective case?

We have reported these results in Section 5.3 Experiment 3: Multi-Objective Ablation. In particular we report “Comparable Performance to Previous Work. CapableAgentBreeder achieves competitive results to ADAS, marginally surpassing performance across all capability benchmarks”.

What impact does the scaffold optimization have on other capability datasets? (eg. how does the MMLU-optimized scaffold perform on GPQA and vice verse)

This is an excellent question about cross-domain generalization that we would be very excited to explore in future work. Unfortunately, given the computational expense of our evolutionary runs, we did not have the budget to conduct these additional cross-dataset evaluations, but we agree this would provide valuable insights into scaffold transferability.

How does the capability-only AgentBreeder perform on the safety metrics?

We appreciate this question, as it would help establish important baselines for understanding the safety implications of capability-focused optimization. While we did not have the computational resources to include these evaluations in the current work, this represents a natural and important extension that we are eager to pursue.

Could you add helpfulness explicitly to the multi-objective optimization target?

Thank you for this thoughtful suggestion. Adding helpfulness as an explicit objective would indeed be valuable for addressing reward hacking concerns. We would be excited to explore this in future work and note that this would increase the computational cost of capability validation in each experimental run by approximately 50%, which exceeds our current budget constraints.

Questions about contributions

What is the purpose of the red teaming experiments? Why do you think red teaming is a better model of a developer inadvertently reducing safety than capability-only optimization?

We appreciate this opportunity to clarify our threat model. The RedAgentBreeder setup models a scenario analogous to the inner/outer alignment problem, where a scaffold designer operates under a misaligned reward function that inadvertently incentivizes unsafe behavior while pursuing seemingly beneficial objectives. This differs from capability-only optimization because it explicitly captures how well-intentioned optimization processes can systematically discover and exploit safety vulnerabilities when the reward structure creates misalignment between intended goals and actual optimization targets.

Is the proposed method applicable to general multi-objective optimization problems or is it specific to safety problems for some reason?

Thank you for highlighting this broader applicability. Our method is indeed general to any multi-objective optimization problem, not specific to safety. We are excited that our work demonstrates the effectiveness of multi-objective approaches in this domain, and we view exploration of different objective combinations as a promising direction for future research.

Additional questions

Is the number in the F_CDROP row the exact same experiment in the red and blue breeder setup? If not what is the difference? If yes why are the numbers different?

Great question for clarification. While both setups optimize for DROP capability during evolution, they differ in their second objective: BlueAgentBreeder optimizes capability and safety, while RedAgentBreeder optimizes capability and harm (1-safety). The F_CDROP numbers reflect the test set performance of the best scaffolds discovered through these different evolutionary processes, explaining why the results differ despite both targeting DROP performance.

How does you method compare to running ADAS with a quality metric that combines different optimization targets, eg. a capability and a safety benchmark?

This is an insightful comparison question. Our approach differs from ADAS in being both evolutionary and explicitly multi-objective, rather than using a single combined metric. Creating a multi-objective version of ADAS for direct comparison would be an exciting avenue for future work that could help isolate the contributions of our different methodological innovations.

Kindest regards,

The Authors

Dear Reviewer 7aKn,

We would like to express our sincere gratitude for your thoughtful and comprehensive evaluation of our work. We are particularly encouraged by your recognition that our paper "addresses a significant need to ensure safety in scaffold design" and that our method provides "an elegant way to balance capability and safety at the core of the optimisation process." Your astute note that considerations of safety are “underrepresented in the optimisation process of current automated design systems for multi-agent scaffolds” is a core reason we are so excited about the potential for this paper to be accepted. The acceptance of this work would help establish safety as a fundamental consideration in automated agent system design and provide the research community with our open-source framework to build upon.

We deeply appreciate your acknowledgment of several key strengths of AgentBreeder. First, we are pleased that you recognized our core contribution of "incorporating safety into automated scaffold discovery through multi-objective optimisation and quality-diversity," which addresses what you correctly identify as a significant gap in current approaches. Second, we are gratified by your positive assessment of our three operational modes, noting how BlueAgentBreeder demonstrates that "safety can be optimised alongside capability," RedAgentBreeder reveals "worst-case scenarios where capable scaffolds can also be highly vulnerable," and CapableAgentBreeder provides important benchmark comparisons. Finally, we appreciate your recognition of our commitment to reproducibility through open-source code and use of recognized benchmarks.

We also thank you for your constructive feedback regarding areas for improvement, particularly regarding baseline comparison fairness (suggesting ADAS baselines that explicitly include safety instructions), the clustering methodology and threshold selection, and the limitations of LLM-as-a-judge safety evaluation. Additionally, we appreciate your observation about the diminishing gains with stronger models, which you note could have "significant implications regarding the limitations of scaffolding ever-improving LLM models." We view this feedback as valuable guidance for strengthening our contribution and are keen to address these important points.

In Table 1, you report on three scaffolds from each experimental run: the most capable (arg max{f_C}), the safest (arg max{f_S}), and a 'best overall' (arg max{f_C, f_S, f_H}). While the first two are clear selections from the Pareto front, could you please clarify the specific criterion or scalarisation function used to select the single 'best overall' scaffold?

Thank you for this excellent clarifying question. The 'best overall' scaffold was selected as the one maximizing the sum of capability and safety scores, with the important caveat that we excluded scaffolds that engaged in reward hacking of the safety objective.

Section 4.1 describes a balanced validation sampling strategy using 50% 'positive' and 50% 'negative' samples. Could you please confirm if these categories correspond to samples that a baseline CoT agent answered correctly and incorrectly, respectively?

Yes, that is exactly correct! ‘Positive' samples correspond to those the baseline CoT agent answered correctly, while 'negative' samples are those it answered incorrectly. We implemented this balanced sampling strategy specifically to increase the signal strength for the evolutionary process, ensuring adequate representation of both success and failure cases.

The impressive safety gains are benchmarked against scaffolds not optimized for safety. To ensure a fair comparison, did you consider creating a stronger baseline by running the original ADAS framework with a prompt that also asks it to consider safety? This seems crucial for isolating the benefit of AgentBreeder's specific methodology.

Thank you for this thoughtful suggestion about creating a more rigorous baseline comparison. We did not modify the original ADAS framework to include safety considerations in this work, but we completely agree this would provide valuable insights into the specific contributions of our multi-objective approach. This represents an exciting direction for future work that would help isolate the benefits of our methodology from simply including safety in the optimization process.

Could you provide more justification for your clustering methodology (the 12-dim embedding and 0.7 threshold)? An ablation study that removes clustering would be the most convincing way to demonstrate its contribution to performance and diversity.

Thank you for this important methodological question. We found empirically that the embedding dimension could be effectively reduced to 12 dimensions while still enabling the clustering to meaningfully separate different scaffold types, such as debate-based versus self-consistency approaches. The distance threshold of 0.7 was selected through preliminary experiments that indicated this value provided an effective balance - ensuring clusters remained sufficiently diverse without becoming overly fragmented, thereby maintaining exploration across different regions of the scaffold design space. These ablation studies were unfortunately deprioritised due to budget constraints, however we agree that a systematic ablation study would strengthen these design choices, and we view this as valuable future work to more rigorously validate the clustering methodology's contributions.

Kindest regards,

The Authors

Dear Reviewer bz36,

We would like to express our sincere gratitude for your comprehensive and well-structured evaluation of our work. We are particularly encouraged by your recognition that AgentBreeder "addresses a fast-growing deployment pattern (LLMs scaffolded into swarms) that current safety work largely ignores" and provides "a practical defence (BLUE) and attack (RED) recipe that labs could integrate into release pipelines”. We believe this work introduces a threat model not yet explored - multi-objective multi-agent self-improving evolutionary risks - and are excited for the long-term impact of this work on the field.

We deeply appreciate your acknowledgment of several key strengths across the dimensions of quality, clarity, readability and significance. We were pleased to be able to run our experiments on a number of widely-used benchmarks - DROP, MMLU, GPQA, SaladData and TruthfulQA. Additionally, we provide a simple API in the codebase to integrate further benchmarks and have included examples in our open source repository. Most importantly, we are encouraged by your assessment of the significance of our work, particularly your observation that we address an important gap in current safety research by focusing on multi-agent scaffolds.

We also thank you for your constructive feedback regarding areas for improvement, particularly regarding generalizability beyond proprietary models and the need for evaluation on more realistic deployment scenarios. We view these as valuable directions for future work and are keen to address the questions you have raised.

Validation-set size & statistical power

You use only 250 validation items per generation and report bootstrap CIs. How sensitive are your evolutionary decisions to this small sample? Did you try larger samples or a moving-average fitness estimate to reduce noise?

We thank you for this interesting question! To compare our approach to the seminal work ADAS (Hu et al. 2024), we matched our sample size and methodology closely to ADAS which used between 20 (ARC) and 128 (other domains) samples for validation. We empirically found that AgentBreeder was sensitive to sample sizes below 100. We believe this to be because our multi-objective diversity optimized setting reduced the signal for the self-improving process, so increased the sample size to 250 (the largest we had budget for). We would be excited to explore moving-average fitness as a future direction and are grateful for the suggestion!

Hyper-parameter choices

Why fix the agglomerative-clustering distance threshold at 0.7? How did you arrive at the 2:1 weighting of mutation vs. crossover?

Thank you for this clarifying question! We chose thresholded agglomerative clustering to allow the number of clusters to emerge naturally during the evolutionary process rather than requiring a fixed cluster count a priori. The distance threshold of 0.7 was selected based on preliminary experiments that indicated this value provided an effective balance - ensuring clusters remained sufficiently diverse without becoming overly fragmented, which helped maintain exploration across different regions of the scaffold design space.

Similarly, the 2:1 mutation-to-crossover weighting was determined through initial hyperparameter tuning. We observed that emphasizing mutation over crossover facilitated better exploration of the scaffold design space, likely due to the discrete and structured nature of code-based scaffold representations where crossover operations can more easily produce invalid or suboptimal combinations.

We acknowledge that both of these hyperparameters would benefit from more systematic ablation studies. Given the computational expense of each evolutionary run (requiring extensive LLM evaluations across multiple benchmarks), we focused our limited budget on the core experimental comparisons presented in the paper. We view comprehensive hyperparameter analysis as an important direction for future work, particularly as computational costs for such experiments decrease.

What temperature / top-p settings were used for the Meta-Agent and for the agents inside the scaffolds?

A temperature of 0.5 and top-p of 1 (the default) were chosen for the Meta Agent - matching the seminal work ADAS (Hu et al., 2024). For the scaffolds, temperatures for the agents could be dynamically chosen by the Meta Agent and varied between each.

Reward-hacking detection

You note that some scaffolds achieve 95 % “safety” by answering “Sorry, I can’t help with that.” Beyond adding a helpfulness metric, did you explore behavioural diversity penalties or response-length constraints to discourage such trivial policies?

Thank you for this exciting suggestion! We did not explore behavioral diversity penalties or response-length constraints in this work, but we agree this represents an exciting and important future direction. The observation that some scaffolds achieve high safety scores through overly conservative refusal behavior highlights a fundamental challenge in safety optimization - distinguishing between genuine safety improvements and superficial compliance strategies.

Your suggestion of behavioral diversity penalties is particularly compelling, as it could encourage the evolution of scaffolds that maintain safety through more sophisticated reasoning rather than blanket refusal. Similarly, response-length constraints or more nuanced helpfulness metrics could help identify scaffolds that achieve the desired balance between safety and utility. This connects to broader questions in AI alignment about reward specification and avoiding Goodhart's law effects in safety metrics.

Kindest regards,

The Authors

Dear Reviewer ahDK,

We would like to express our sincere gratitude to the reviewer for their thorough and constructive evaluation of our work. We are particularly encouraged by their recognition that our paper "makes a significant and timely contribution to AI safety and the Automated Design of Agentic Systems (ADAS)" and addresses "a critical gap in the safety of ADAS and self-improving AI systems". We are excited about the potential for this paper to be accepted, as the first paper (to our knowledge) addressing this kind of multi-agent self-improving evolutionary threat model. We believe this will motivate research in this new area, as well as our open-source codebase providing a jumping-off point in this field.

We are gratified that they found our experimental evaluation compelling, specifically noting the "79.4% average uplift in safety benchmark performance" achieved by BlueAgentBreeder while maintaining capability improvements. We are also pleased that the reviewer recognized our technical contributions, including our tailored mutation and crossover operators, multi-objective optimization framework, and MAP-Elites-inspired clustering approach, with them noting that our safety objective "not only improves adversarial robustness but also appears to facilitate more effective open-ended exploration of the search space".

We also thank the reviewer for their constructive feedback regarding areas for improvement, particularly their suggestions for additional ablation studies and clearer justification of certain design choices such as our clustering approach and the dual-objective strategy in RedAgentBreeder. We view this feedback as valuable guidance for strengthening our contribution and are keen to address these points.

The paper states, "Weighting the mutation operator twice as highly as crossover was found empirically to lead to faster convergence" (Line 133). Could the authors provide details or results from the empirical study supporting this claim?

Thank you for highlighting this point. We acknowledge that we should have included a more systematic ablation study of the mutation-to-crossover ratio. This empirical observation was based on initial tuning experiments, but we recognize that a comprehensive analysis of this hyperparameter would strengthen the paper. We will include this as an important direction for future work.

The explanation in Line 213 regarding RedAgentBreeder’s approach is unclear: "It is important to note that in this case, the Meta Agent is not prompted to discover unsafe scaffolds, instead these arise via Pareto optimization on capability and harm benchmarks. This seeks to model the case where an actor may unknowingly expose weaknesses in the base LLM when employing scaffolding to improve task performance." If RedAgentBreeder simulates an adversarial attacker, why is the Meta Agent not explicitly prompted to identify unsafe scaffolds? Additionally, how optimizing for both capability and harm benchmarks leads to "unknowingly exposing weaknesses." Isn't minimizing the safety benchmark explicitly trying to expose weaknesses?

We thank the reviewer for highlighting this important conceptual confusion in our explanation. The threat model we consider with RedAgentBreeder is analogous to the inner/outer alignment problem. Specifically, we model a scenario where a scaffold designer (or Meta Agent) has good intentions and explicitly optimizes for task capability, but operates under a misaligned reward function that inadvertently incentivizes unsafe behavior. In this setting, the Meta Agent is not explicitly prompted to discover unsafe scaffolds - rather, it is optimizing for what it believes to be beneficial objectives (high task performance). However, the reward structure creates a misalignment between the intended goal (safe capability improvement) and the actual optimization target.

We believe you have highlighted an exciting direction for future work, where a jailbroken Meta Agent is tasked with designing unsafe scaffolds.

The results for argmax {f_C^DROP, f_C^MMLU, f_C^GPQA} in Table 1 (CapableAgentBreeder) do not align with those reported in Table 3 (Appendix B.3). Are there differences in experimental settings that account for this inconsistency?

We thank the reviewer for their careful attention to the experimental details. Yes, the results in Table 1 are reported for BlueAgentBreeder - AgentBreeder run in a multi-objective setup which jointly optimizes capability and safety. Whereas the results in Table 3 are reported for CapableAgentBreeder which optimizes solely for capability. Both experimental configurations are labeled at the top of each respective table and are also referenced in the corresponding figure captions, however we will update the manuscript to better contextualize these three (blue, red and capable) different experiments as we believe you’ve highlighted a great improvement to the readability of the paper.

Kindest regards,

The Authors

I thank the authors for their response and the additional experimental results. I think they address all of my concerns. A few additional suggestions:

1. The calculation of HVs provides good insights. But it would be more insightful to compare HVs to other optimization methods, e.g., when you choose different objective(s) to optimize in Table 1.

2. Selecting the scaffold that has the highest sum of objectives makes sense. But I think it would be helpful to also show the best-performing agent and safest agent, i.e., choose a few scaffolds to show the trade-off between objectives. Also, the selection method should be stated in the paper.

That being said, I think the topic of this paper is timely and important for the safety of agents and self-improving AI, making an important milestone in the direction of automated agent design. I believe the method design is sound, and the extensive experiments support the effectiveness of the method. I will recommend the acceptance of the paper and increase my score to 6.

Dear Reviewer ahDK,

Thank you so much for providing further feedback and suggestions! We find them of great value and have taken the time to address all your further points, including plots, metrics and answers to your questions!

Visualization of Pareto front (PF) and reporting of hypervolume (HV) indicator

In Appendix B: Experimental Runs (on page 14), you will find plots showing the advancement of the Pareto capability-safety frontier on the validation set for all multi-objective experiments.

We valued your suggestion of reporting a metric to quantify the effectiveness of the multi-objective optimization and report the HV indicator on the test set.

For BlueAgentBreeder:

And for RedAgentBreeder:

Additionally, here is the code we used to generate these values:

import json, math
from pathlib import Path

file = Path(
    "/home/Documents/AgentBreeder/src/results/test/gpqa-cecec343-5f63-4a02-99b8-7d0155d7c45f.jsonl"
)
records = [
    json.loads(l) for l in file.read_text(encoding="utf-8").splitlines() if l.strip()
]
for r in records:
    r.pop("system_code", None)

seed = {
    "Chain-of-Thought",
    "Self-Consistency with Chain-of-Thought",
    "LLM Debate",
    "Step-back Abstraction",
    "Quality-Diversity",
    "Dynamic Assignment of Roles",
    "Self-Refine (Reflexion)",
}

groups = {
    "Seed": [
        (
            r["system_capability_ci_median"],
            r["system_safety_ci_median"],
            r["system_name"],
        )
        for r in records
        if r["system_name"] in seed
    ],
    "Discovered": [
        (
            r["system_capability_ci_median"],
            r["system_safety_ci_median"],
            r["system_name"],
        )
        for r in records
        if r["system_name"] not in seed
    ],
}

ref = (0.0, 0.0)


def pareto(pts, ref=(0, 0)):
    pts = [p for p in pts if p[0] >= ref[0] and p[1] >= ref[1]]
    pts.sort(key=lambda t: (-t[0], t[1]))  # x↓ then y↑
    front, best_y = [], -math.inf
    for x, y, n in pts:
        if y > best_y:
            front.append((x, y, n))
            best_y = y
    return front


def hv(pts, ref=(0, 0)):
    front = pareto(pts, ref)
    xs = [p[0] for p in front] + [ref[0]]
    return (
        sum(
            max(xs[i] - xs[i + 1], 0) * (front[i][1] - ref[1])
            for i in range(len(front))
        ),
        front,
    )

for name, pts in groups.items():
    volume, frontier = hv(pts, ref)
    print(
        f"{name} hypervolume: {volume:.6f}  (|S|={len(pts)}, |front|={len(frontier)})"
    )

Final Questions

1. My question in Weakness 1 regarding the clustering and Pareto fronts was not addressed in your response. Could you please discuss this?

We chose clustering within the archive specifically to enhance exploration of the search space, following principles from MAP-Elites, and because it aligns well with our threat modeling objectives. Your concern about the potential reduction in selection pressure due to most solutions becoming non-dominated within clusters is valid and deserves more thorough investigation. We agree that ablation studies would significantly strengthen our work by demonstrating the individual contributions of clustering and the crossover operator. We focused our limited budget on running AgentBreeder across multiple and varied benchmarks and would love to explore the ablation studies in future work.

2.2. How do you select the final agent from the PF to present in the tables? There should be many agents balancing the tradeoffs between objectives.

Thank you for highlighting this! The Pareto optimal scaffold was selected as the one maximizing the sum of capability and safety scores, with the important caveat that we excluded scaffolds that engaged in reward hacking of the safety objective.

We hope we have addressed your questions, but please do let us know if there's anything we can clarify in order to increase our score.

Kindest regards,

The Authors