Enhancing Multi-Agent Learning in Real-World Interactive Environments through Process Reward Decomposition

1. The organization of the paper needs some improvement, especially MARL with Edge Updates Section.
2. The main concern is that the proposed Process Reward Decomposition is not well-justified: I am skeptical about using a critic agent to directly generate temporal and structural credit assignments.
3. Lack of comparison with related work on multi-agent LLM[a].

a. Zhuge, Mingchen, et al. "Language agents as optimizable graphs." arXiv preprint arXiv:2402.16823 (2024).

1. In Task Formulation, it is necessary to clarify the concepts of agent and policy. Furthermore, there needs to be consistency between a_t in Eq(3) and Eq(1). It is unclear whether a_t represents an action by a single agent, a joint action by multiagent, or an aggregation action, which should be specified.

2. In stage 1, why is the curriculum divided into three parts? or what is the insight? It's better to explain the rationale behind choosing these three particular parts for the curriculum, and how each part contributes to the overall learning process.

3. How to comprehend ''The rationale is that, for the critic agent, it might be more simple to identify whether a single decision is wrong, than to judge the reward of long decision chains between multiple agents''? It seems to contradict RL research. Instead, single-step rewards are usually not as accurate as long-term rewards, From the internal logic, judging a single-step decision is more difficult than multi-step. It's better to provide evidence or reasoning to support their claim, especially in light of existing RL literature that suggests otherwise.

4. Multi-agent reinforcement learning methods are usually based on mathematical models such as Markov Game and Dec-POMDP. In stage 2, what is the multi-agent model, which needs a detailed explanation? How does it relate to or differ from standard models like Markov Games or Dec-POMDPs?

5. The nature of their reward decomposition process in stage 2 should be explained clearly. Is it artificial prior or adaptive learning? If it is artificial prior, it is no different from role allocation in form, and it is difficult to reflect the advantage of adaptive learning. How does it differ from or improve upon traditional role allocation methods?

6. In stage 2, it seems that each agent learns independently. How to ensure multi-agent collaboration? What are the mechanisms or techniques to promote collaboration between agents during the learning process in stage 2?

7. The paper also has several unclear character definitions and grammatical problems, such as：

   (1) It should be "each agent \pi_i” instead of "all agents \pi_i” in line 271

   (2) It should be "the number of agents” instead of all "the number agents" in line 132

   (3) In Task Formulation, transition function and the maximum step are both represented by the character `T’.

   The reviewer suggests that the authors conduct a thorough proofreading of the paper, paying particular attention to consistency in mathematical notation and grammatical correctness.

1. In the majority voting described in equation (5), how do you process the situation where two or more actions are tied to have the same counts? If the strategy is random selection, could you please show me if this kind of strategy will incur some fluctuation in performance?
2. In line 171-172, the sentence that "The proper size of local memory enhances the diversity of decision making and avoids introducing too long contexts," is not easy to comprehend. I cannot have a clear picture to link the local memory size with the diversity of decision makings. Could you give more explanation about it?
3. I scanned the paper, and have not noticed any discussion on the faithfulness of automatically generated data to the realistic situations. Without this guarantee, I cannot foresee the benefit of automatic generation of data for training LLMs, as the resulting harm to the society could be more severe than the improvement measured in quantitative metrics and paid human labours. How do you guarantee that the generated data is faithful to realistic situations, with no hallucinations?
4. During processing preference knowledge, you proposed to use SFT followed by DPO, but with no reason to explain this strategy. Could you please give more insights into this strategy?

The experimental setup in this paper lacks clarity, particularly regarding the configuration and evaluation of different agent setups, which impacts the transparency and reproducibility of the experiments. Additionally, the framework is demonstrated with a relatively small number of agents, raising scalability concerns. As the number of agents grows, managing an undirected communication graph may become computationally expensive, potentially affecting performance. Furthermore, some figures could benefit from clearer annotations and layout to better convey the framework’s structure and processes.