MAC-CAFE: Multi-actor, Centralized Critic Architecture for Feedback-driven Editing

1. Limited methodological contribution: Besides the splitting of the problem into multiple documents, the paper seems like a fairly direct application of PromptAgent with limited technical novelty besides the application to a new domain.

2. Assumption of error knowledge: The method uses LLMs to generate code and then use feedback from generated code to update the docs. In lines 194-199, the authors correctly point out that there could be multiple sources of error, including sources that do not stem from incorrect docs. However, the authors then say that they assume errors result from only from incorrect docs. It's not clear at all from the writing how -- or whether -- this is enforced, i.e. how the authors ensure that the errors are in fact from errors in the docs rather than the generator's shortcomings. This is especially troubling given that they evaluate on lower-resource coding languages where the model might be worse at generating even with correct docs. If this assumption is enforced, the authors should explain how. If it's not enforced, it's the authors responsibility to convince readers that their benefits come from the system in fact improving the docs in some interpretable way, as opposed to addressing simpler kinds of errors.

3. Limited results: the "extensive experiments" mentioned in fact boil down to 2 short paragraphs on the last page of the paper. There are no ablations for the design choices made (e.g. action space, state representation) and no ablations showing the necessity of splitting the task into a multi-actor setup. The authors only evaluate a single LLM. There is no analysis of the resulting KBs after editing.

4. Unclear baselines and metrics: It's not clear what the baseline PromptAgent-E is/how it is implemented, why it was chosen, and why it is a fair/relevant baseline. This kind of detail should be brought up in the main paper (which the paper fails to do) and can then be elaborated in supplementary material (which the paper lacks). The authors also do not convincingly argue for the completeness metric measures completeness. The coherence metric is not clearly explained. The metric section is split across 5 and 6.3 in a way that is very hard to follow

5. Unclear writing and organization: In addition to the clarity issues above, the rest of the paper also omits a large number of details. To give a few examples:

• The method hinges almost entirely on prompting but the authors do not provide their prompts.
• In Table 3, it's not clear what any of the numbers refer to. They aren't percentages, but they also don't add up to the number of documents.
• Much of the writing could be compressed. There are several sections that conceptually should be subsections/compressed together. At the same time, other parts are overcompressed, e.g. the results section, where Table 3 is unreadably small.

6. Method described as using gradients: The method (which is in fact gradient-free) is described in terms of gradients. My feeling is that the authors should make clear that the "gradient" part here is purely a helpful metaphor, as the method does not actually compute any gradients at all. However, from the writing, this point is not brought through clearly.

• Section 4.1 that introduces the problem formulation can be written more clearly, especially regarding the following points:
  • Introduce tau(q_i, K) before equation-1 in page 4, to ensure the equation is understandable - currently tau is first introduced in the next subsection.
  • Explain the function g - is g supposed to give a higher score when o_i=c_i?
  • Define o_i=B(qi, tau(qi, K)) [is this the definition?]
• I'd suggest to add a short note describing the ReACT agent (perhaps near line 348) - even though it has been cited, not everyone might have a working knowledge of it.
• Line 348, Actors: I assume a real-life knowledge base would have a huge number of documents - how feasible is it to have a distinct actor model for each document?
• What are the models used for (1) B the LLM, and (2) the actors? I was unable to find it in Section 6. I suggest to add a subsection or paragraph on the same in Section 6.
• I also suggest the authors to add a paragraph each about the baseline PromptAgent and about Monte Carlo Tree Search for more accessibility of the paper.

My major concern lies in two aspects:

1. The rationale behind the design of the proposed method is insufficiently detailed. Although the authors thoroughly describe their implementation, particularly in Section 4, the intuition underlying each component of the design is not clearly described.

2. The result analysis section is quite limited. The authors mainly emphasize the effectiveness of the proposed approach but miss a variety of experiments, such as ablation studies or error analysis, to offer a deeper understanding of its characteristics. For example, which components of the approach are more influential than others? Are there any identifiable patterns in the prediction errors?

Additionally, the paper writing could be improved. For example, I’m a bit confused about the purpose of section 3. If the illustrative example is intended to motivate MAC-CAFE, it might be more effective to condense this description and incorporate it into the Introduction. Doing so would allow for a more detailed and thorough result analysis in the corresponding section.

W1. The centralized feedback analysis should be compared with self-reflection / self-refine methodologies.

W2. Managing each document with an agent seems quite computationally expensive-- can you provide some analysis of cost/benefit?

W3. Not a factor in my score-- the contributions listed in the intro don't tie back clearly to sections in the paper; it would be stronger to identify which section each of these contributions is described within. This particular subsection also feels like it may have been LLM-generated; I'm not concerned about that for the rest of the paper, but that section reads a bit poorly.