1. Reliability of Language Loss and Gradients: The idea of using evaluator networks to improve performance is not new, especially for complex reasoning tasks. However, the reliance on LLMs for computing language loss and gradients raises concerns of reliability, especially when the authors want to "back-propagate" the loss. Deviations may accumulate and make the overall update unstable. Could the authors conduct experiments with complex tasks where the analogous symbolic network has extended node numbers and investigate how the language loss transits under such settings?
2. Data-Centric Claims and Unsupervised Role of Data: While the authors claim the framework to be data-centric and can be applied in an unsupervised way, the exact role of data under such setting is not thoroughly addressed. The appendix only provides templates for language loss function under the supervised setting. Could the authors provide the template for the unsupervised setting, and the corresponding experiment results?
3. Convergence: Convergence is a critical aspect of neural network training, yet the paper does not discuss when or how the analogous "training" process stabilizes for the framework. Could the authors elucidate specific metrics or criteria they use to determine convergence in their framework and potential challenges in reaching it?
4. Expanded Baselines and Benchmarks: While the framework shows promising results on selected tasks over several methods, additional baselines and benchmarks are needed to comprehensively evaluate the performance. Specifically, for the "standard LLM benchmark", the paper does not clearly state the benchmarking method for GPTs. Is it zero-shot or few-shot based? Given that the method uses few-shot examples, few-shot prompting for GPTs would be necessitated for a fair comparison. Also comparisons with other baselines, such as CoT/ToT at least, are important for validating the effectiveness of the method. A wider choice of tasks would also be favorable since creative writing and software development alone might not be sufficient for representing complex agent tasks.
5. Ablation Studies for Component Analysis: An ablation study on the components, such as the workflow optimizer and prompt optimizer, would provide more granular insights into each component’s contribution and help understand how the proposed method achieves its reported performance improvements.
6. Open-Sourcing the Framework: The authors mentioned that the framework would be open-sourced. I wonder if this could be done in the rebuttal period.

• The improvements in the experimental results of this paper are not very significant.
• The paper is more like a multi-agent cooperation framework, describing operations such as reflection and discussion as forward and backward propagation, and combining them into a framework. The paper does not obtain better agents through training or similar methods, so describing it as "evolve" is a bit far-fetched.
• There is a lack of evaluation on complex sequential decision-making agent tasks, such as those in AgentBench [1], AgentTuning [2], AgentGym [3], and τ-bench [4], and the paper lacks comparison with these frameworks.
• The backbone models used are mainly from the GPT series, and it is necessary to include more models, especially open-source models.

[1] AgentBench: Evaluating LLMs as Agents

[2] AgentTuning: Enabling Generalized Agent Abilities for LLMs

[3] AgentGym: Evolving Large Language Model-based Agents across Diverse Environments

[4] τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

Weaknesses:

Quality:

WQ1: Section 3.1 offers an analogy between connectionist learning and agent symbolic learning, but it fails to mention and discuss where does the analogy breaks, such as for instance:

• in the fact that the loss function requires comparison to a groundtruth label which is not instantiated here in the agent symbolic learning ;

• in the fact that the lack of groundtruth labels makes the whole approach more akin to an unsupervised learning approach rather than a supervised learning approach (like connectionist learning) ;

• in the fact that the connectionist gradient is effective due to mathematical guarantees that ascertain that updating numerical weights in the gradient direction will lead to an optimization of the corresponding loss ; but there is no such guarantee when considering language gradient as they are only the result of an LLM output, which we know to be riddle with issues, such as hallucinations[2,3] and unfaithfulness[4,5] ;

• …

Adding this discussion would improve the quality and significance of the paper, by detailing to the reader possible limitations that could be addressed down the line, and it would greatly improve my appreciation of the paper towards proposing honest and thought-through research that does not build unnecessary hype.

WQ2: I would like to invite the authors to present results with open-source LMs, as it would be valuable for overall replicability purposes and also to understand better how the approach performs with different LMs.

WQ3: I would like to invite the authors to consider include an experiment that studies how the performance scale with the size of both the forward pass LM and the symbolic optimizer LM. I think it would provide great insights into the potential of the propose approach for the democratisation of LMs, for instance.

Clarity:

WC1: In Section 3.2, in the paragraph explaining equation (1), ‘task requirements’ are mentioned without much details, and I believe it is the first time it is mentioned in the paper. I would like to invite the authors to define it as precisely as possible, possible in section 3.1 for problem formulation, as I now realise how important of an element it must be (Please clarify if I am mistaken?). I also suspect that it is an element that cannot be optimised in a data-centric way but is still dependent on manual engineering effort, along with the prompt $\mathcal{P}_\text{loss}$, right? If so, I mean to invite the authors to collate all of those elements that are not yet data-centric-ly optimisable within the propose agent symbolic learning framework and propose a discussion of how future works may try to address them?

WC2: In section 3.2, the language loss computation paragraph actually details supervised agent learning vs unsupervised agent learning scenarios. I think it would make better sense to introduce this in Problem Formulation Section 3.1, and clearly highlight in which scenario will the main paper present experiments. I was actually assuming until this paragraph that only the unsupervised agent learning scenario was performed given the mention of using LLM-as-a-judge framework. It might be valuable to nuance and explicitly detail what is meant by LLM-as-a-judge when it is used in Section 3.1 actually.

WC3: As much as possible, it would be really valuable to integrate the different prompt templates for loss and gradient, and gradient update computations into the main text of the paper, in a concise form, or at least point specifically at them and discuss some of their guiding principles further in depth, in order to give a more concrete understanding of their impact to the reader.

Originality:

WO1: Optionally adding a comparison of the contemporary approach of [1] could improve the impact of the paper.

Significance:

WS1: Following up on my concerns in SS1, I find that the paper does not sufficiently discusses the limitations of the proposed framework, which result in an impression of building up hype for a product rather than doing thought-through research. I would like to emphasise again that my appreciatetion of the work would be greatly improve if the authors could revise the paper by including a lot more insights about limitations and what does not work fully (yet) and/or how some critical elements may impact the performances of their proposed work. For instance, could it be possible to perform an ablation study on the quality of the task description or the quality of the different engineering-centric prompt templates for loss and gradient and gradient update computation, please?

1. The second paragraph of the introduction needs references.
2. The claim that "these optimization methods are prone to a local optimum of isolated prompts, tools, and nodes" in the paragraph from lines 079 to 092 is incorrect. GPTSwarm includes an edge optimization method, which optimizes the connection of nodes globally.
3. Moelding the nodes or components with a sequential structure, as defined in line 213, is not making a full analogy with computational graphs in neural nets, as claimed in line 211. A computational graph of neural nets can be arbitrary directed acyclic graphs.
4. It is better to have a citation for dynamic neural nets at the end of page 4.
5. There is an inconsistency of variable use. Variable $n$. is used for the number of nodes in line 213. However, it is used as an arbitrary index of the node in equation 2. The same inconsistency also applies to lines 12, 13, and 17 of algorithm 1.
6. It is unclear to me how the gradient of the last node is defined.
7. There are typos, such as the word "optimizer" in line 351 should be "optimize."
8. Line 13 of algorithm 1 appends a gradient to the trajectory. This is not mentioned in the main text. Which strategy should I follow?
9. The abstract claims that the proposed methods outperform others on real-world tasks, which I do not think is the case.
10. It is better to give a reference about how OpenAI implements GPTs in line 423.
11. There is a lack of detail about how the initial agents are implemented.
12. Many experimental details are missing. For example, what are the learning rate and batch size used for experiments? Is the optimization done in a supervised or unsupervised setting?
13. Many of the references are missing the publisher.