G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems

We sincerely thank you for your careful comments and thorough understanding of our paper! Here we give point-by-point responses to your comments and describe the revisions we made to address them.

Weakness 1: The paper admits that tasks are limited to five benchmarks, mainly QA, embodied action, and games. Generality to other domains is unproven.

Thank you for your valuable suggestion. To further demonstrate the generalizability and effectiveness of G-Memory across diverse task settings, we add the GAIA benchmark, which includes tasks like file reading, web browsing, coding, and multimodal understanding. We integrate G-Memory into the SmolAgent framework and evaluate its performance, API cost, and latency:

These results suggest that G-Memory consistently improves agent performance across all task levels, with only marginal latency and API cost increase. We will incorporate this analysis into the revised manuscript to further clarify G-Memory’s scalability and practical utility.

Weakness 2: Some lower-level details, such as precise prompt engineering and insight validation, rely on in-LLM processing. The robustness of these steps is not deeply analyzed.

Thank you for your insightful question! To address your concern, we elaborate on both the intuitive design and empirical effectiveness of our prompt engineering and insight validation mechanisms.

(1) prompt settings

Our work incorporates three main components of prompt engineering, which we detail below:

• LLM-based Graph Sparsifier (Eq. 7): The operator $\mathcal{S}_\text{LLM}$ is designed to distill a verbose interaction graph $\mathcal{G}\_\mathsf{inter} = (\mathcal{U}, \mathcal{E}\_\mathsf{u})$ into a minimal yet sufficient reasoning path tailored to a new query $Q$. This is achieved in two stages: (1) Node Summarization, where an LLM function condenses the textual content $m\_i$ of each utterance node $u\_i = (\mathcal{A}\_i, m\_i)$, producing a summarized node set $\mathcal{U}'$ while preserving the original graph structure. This yields an abstracted graph $\mathcal{G}\_\text{sum} = (\mathcal{U}', \mathcal{E}\_\mathsf{u})$, where $\mathcal{U}' = \\{ (\mathcal{A}\_i, \text{LLM}\_\text{sum}(m\_i)) \mid (\mathcal{A}\_i, m\_i) \in \mathcal{U} \\}$; (2) Graph Extraction Subsequently, the abstracted graph $\mathcal{G}\_\text{sum}$ is then provided to $\mathcal{S}\_\text{LLM}$ and produces $\hat{\mathcal{G}}\_\mathsf{inter}(\hat{\mathcal{U}}, \hat{\mathcal{E}}) = \mathcal{S}\_\text{LLM}(\mathcal{G}\_\text{sum}, Q)$.
  To further validate the necessity of this component, we conduct an ablation study comparing systems with and without $\mathcal{S}\_\text{LLM}$:

These results empirically confirm that the LLM-based interaction summarization and filtering mechanism is both effective and essential.

• LLM-based Filtration (Eq. 7): As referenced in Eq. (7), the filtration process is guided by an LLM, and we provide the full implementation details in Appendix C for transparency and reproducibility.
• Insight Extraction (Eq. 10): This is detailed below:

(2) insight validation

G-Memory is equipped with at least two mechanisms to ensure that accurate and helpful insights are increasingly emphasized over time:

• G-Memory periodically performs consolidation over accumulated insights to merge redundant ones and discard irrelevant or obsolete content. First, recent queries are grouped into semantically related clusters $\{\mathcal{C}\_l\}$. For each cluster, all corresponding insights are retrieved from the graph: $\mathcal{I}\_l = \Pi_{\mathcal{Q} \to \mathcal{I}}(\mathcal{C}\_l)$, which is then synthesized into a set of more general insight, $\iota_l^\text{merged}$, using the function $\mathcal{J}$ in Eq. (10) that merges their textual content and supporting query sets: $\iota\_l^\text{merged} = ( \mathcal{J}(\{\kappa\_k \mid (\kappa\_k, \Omega\_k) \in \mathcal{I}\_l\}), \bigcup\_{\iota\_k \in \mathcal{I}\_l} \Omega\_k )$. This consolidation ensures that the insight graph is regularly updated to maintain concise and representative high-level knowledge.
• As the query and insight graphs evolve over time, correct and helpful insights/interactions naturally lead to more successful task executions, thereby accumulating additional edge connections. This process positions them as denser and more central nodes in the graph, making them more likely to be retrieved, as illustrated by nodes 45 and 55 in Figure 10. In contrast, misleading elements tend to have limited impact and remain on the periphery of the graph (e.g., nodes 36 and 37 in Figure 10). This update dynamically implicitly prioritizes reliable knowledge while suppressing noise.

Question 1: How does the memory graph scale with increasing agents/queries/tasks

We would like to respectfully point out that all major operations within G-Memory are designed to scale efficiently without significantly increasing token cost as the queries/tasks grow:

In addition to being robust to the number of queries/tasks, G-Memory is also largely insensitive to the number of agents, as almost all processes and operators within G-Memory are independent of the agent team size. The only exception is Eq. (8), where the computational cost scales linearly with the number of agents. Nevertheless, this cost remains manageable in practice and does not significantly affect overall efficiency.

In summary, G-Memory can scale its memory pool without incurring excessive token cost, making it suitable for long-horizon multi-agent tasks.

Question 2: Are there mechanisms for automatic pruning or compression of obsolete information?

Thank you for this insightful inquiry! Please refer to our response to Weakness 2, where we introduce the consolidation mechanism where insights are periodically reconstructed or merged.

Hi Reviewer adrQ,

Can you check the author response and see if you would like to follow up on any remaining concern?

Thanks, AC

We would like to express our deepest respect for your meticulous review! We can genuinely sense that you have dedicated considerable time to thoroughly reviewing our manuscript and providing very specific insights and feedback. We must acknowledge that this is one of the most enlightening and helpful reviews we have received in recent years! In response to your efforts, we have carefully prepared a point-by-point reply:

Weakness 1: What about the cold-start problem?

Thank you for this important question! We respectfully detail the cold-start process for each memory layer as follows:

• Query Graph: Each node represents a task query, and a new node is added upon the completion of each task. Retrieval in this graph does not involve a cold-start process—each query retrieves similar past ones via Eq. (4). To reduce early-stage noise caused by sparse or weakly correlated data, we apply a hard similarity threshold; when no historical query exceeds this threshold, no retrieval occurs, effectively suppressing spurious early matches.
• Insight Graph: To ensure meaningful insight-level retrieval, we delay G-Memory’s insight-based querying until 20 queries have been processed. Nevertheless, the accumulation of insight-level data begins from the very start. Nevertheless, G-Memory is not sensitive to the choice of cold-start threshold. We present a sensitivity analysis below, showing that varying the minimum delay (i.e., the number of queries before retrieval is enabled) has a limited impact on performance:

Table A. Sensitivity study of delay queries using GPT-4o-mini.

• Interaction Graph: This graph evolves in sync with the query graph. If relevant past queries are retained (i.e., not filtered out due to low similarity), the corresponding sparsified interaction graphs are available for retrieval. If none are deemed relevant, no retrieval is triggered.

Overall, G-Memory is designed to be robust to cold-start issues and can efficiently operate in new domains with minimal prior data. We sincerely thank you again for your thoughtful feedback and will include these clarifications in the revised manuscript.

Weakness 2: How exactly are token costs calculated? And does the size of memory affect this?

Please allow us to respectfully respond to the questions one by one:

(1) How exactly are token costs calculated? The token cost demonstrated in Figures 3 and 7 is the sum of both prompt token and completion token.

(2) Does the size of memory affect this? The memory pool becomes larger gradually, what's the impact on the cost?

We would like to clarify that all major operations within G-Memory are designed to scale efficiently without significantly increasing token cost as the memory pool grows:

In summary, G-Memory can scale its memory pool without incurring excessive token cost, making it suitable for long-horizon multi-agent tasks.

Weakness 3: Why skip the 0-hop baseline in Figure 4?

To address your concern, we have supplemented the sensitivity analysis in Figure 4(a) with additional results for 0-hop:

Table B. Hop expansion experiment using qwen-2.5-14b.

These results demonstrate a trade-off between the receptive field and noise as the hop size increases. With fewer hops (e.g., 0-hop), the retrieved queries may be too sparse or uninformative. Conversely, expanding to larger hops may introduce excessive noise. We observe that 1–2 hops strike a reasonable balance.

Therefore, we agree that “the similarity may be only superficial or noisy” applies both when the hop expansion is too limited or too broad. Thank you again for your insightful comment! We will clarify this point in Section 5.4 of the revised manuscript.

Weakness 4: The ranking/sparsification part feels handwavy.

To address your concern, we will humbly (1) provide a more fine-grained explanation of the sparsifier, and (2) empirically validate its effectiveness.

（1）Detailed Process

The $\mathcal{S}\_\text{LLM}$ operator distills a verbose interaction graph $\mathcal{G}\_\mathsf{inter}=(\mathcal{U}, \mathcal{E}\_\mathsf{u})$ into its essential reasoning path relative to a new query $Q$. This is performed in two main stages: (1) Node Summarization, where an LLM function condenses the textual content $m_i$ of each utterance node $u\_i = (\mathcal{A}\_i, m\_i)$, producing a summarized node set $\mathcal{U}'$ while preserving the original graph structure. This yields an abstracted graph $\mathcal{G}\_\text{sum} = (\mathcal{U}', \mathcal{E}_\mathsf{u})$, where $\mathcal{U}' = \\{ (\mathcal{A}\_i, \text{LLM}\_\text{sum}(m\_i)) \mid (\mathcal{A}\_i, m\_i) \in \mathcal{U} \\}$; (2) Graph Extraction Subsequently, the abstracted graph $\mathcal{G}\_\text{sum}$ is then provided to $\mathcal{S}\_\text{LLM}$ and produces $\hat{\mathcal{G}}\_\mathsf{inter}(\hat{\mathcal{U}}, \hat{\mathcal{E}}) = \mathcal{S}\_\text{LLM}(\mathcal{G}\_\text{sum}, Q)$.

(2) Empirical Evalution

We further conducted an ablation study on this component, and the results are presented below:

Table C. Ablation study of sparsifier. The backbone LLM is gpt-4o-mini.

These results confirm that, providing interaction memory in a more concise manner is conducive for multi-agent systems to better leverage previous experience.

Hi Reviewer o31T,

Can you check the author response with new results and see if all your questions were addressed?

Thanks, AC

Thank you immensely for your time and efforts, as well as the helpful and constructive feedback! Here, we give point-by-point responses to your comments.

Weakness & Question 1: Some notations are not very clear. What is one epoch in MAS? In debate case, one round of agent speaking is one epoch; but in solely task solving case, what does one epoch mean?

Thank you for pointing out this issue. We would like to respectfully clarify the following:

• In general, an epoch refers to a single iteration of the agent interaction process, typically defined by one full cycle of decision making or action execution within the multi-agent workflow. More specifically:

  • In frameworks such as LLM-debate and DyLAN, an epoch corresponds to one complete round of agent responses, as you rightly noted.
  • In coding frameworks, such as MetaGPT and EvoMac, an epoch denotes one round of code generation, which often concludes with a reviewer or tester evaluating the output and feeding back signals for the next round.
  • In AutoGen, where a single round of interaction typically produces a final action, the number of epochs is naturally set to 1.

• In Eq. (5), $\mathcal{N}^+(Q_j)$ and $\mathcal{N}^-(Q_j)$ denote the out-neighborhood and in-neighborhood of node $Q_j$, respectively.

We greatly appreciate your careful reading and will incorporate these clarifications into the revised manuscript.

Weakness & Question 2: In the experiment section, it is expected to adopt more tasks to validate the effectiveness of the proposed method; ... such as on WebArena dataset or GAIA dataset?

Thank you for your valuable suggestion. To further demonstrate the generalizability and effectiveness of G-Memory across diverse task settings, we integrate G-Memory into the SmolAgent framework and evaluate its performance, API cost, and latency on the GAIA benchmark across multiple levels:

These results suggest that G-Memory consistently improves agent performance across all task levels, with only marginal increases in latency and API cost. We will supplement the experiment into the revised manuscript to further clarify G-Memory’s scalability and practical utility.

Question 3: How about the latency of the G-memory; In the paper, the authors present the token cost, but the latency is not mentioned.

To address your concern, we provide additional latency measurements below:

As shown, (1) G-Memory incurs only moderate inference overhead even when delivering the most significant performance gains. For instance, on AutoGen+ALFWorld, it increases latency by merely 9% compared to the no-memory baseline; (2) in some cases, G-Memory even improves efficiency, e.g., on DyLAN+ALFWorld, it leads to a 20% speed-up, as helpful memory cues enable the multi-agent system to reach correct actions more quickly and terminate the interaction earlier.

We sincerely thank you for the thoughtful and constructive reviews of our manuscript! Based on your questions and recommendations, we give point-by-point responses to your comments and describe the revisions we made to address them.

Weakness & Question 1: Cold-Start Graph Initialization

Thank you for this important question! We respectfully detail the cold-start process for each memory layer as follows:

• Query Graph: Each node represents a task query, and a new node is added upon the completion of each task. Retrieval in this graph does not involve a cold-start process—each query retrieves similar past ones via Eq. (4). To reduce early-stage noise caused by sparse or weakly correlated data, we apply a hard similarity threshold; when no historical query exceeds this threshold, no retrieval occurs, effectively suppressing spurious early matches.
• Insight Graph: To ensure meaningful insight-level retrieval, we delay G-Memory’s insight-based querying until 20 queries have been processed. Nevertheless, the accumulation of insight-level data begins from the very start.
• Interaction Graph: This graph evolves in sync with the query graph. If relevant past queries are retained (i.e., not filtered out due to low similarity), the corresponding sparsified interaction graphs are available for retrieval. If none are deemed relevant, no retrieval is triggered.

Overall, G-Memory is designed to be robust to cold-start issues and can efficiently operate in new domains with minimal prior data. We sincerely thank you again for your thoughtful feedback and will include these clarifications in the revised manuscript.

Weakness 2: Ambiguous Graph Update Mechanisms: The update process for assimilating new experiences requires clarification

To further address your concern, we provide a more intuitive and detailed explanation of how each layer of the graph is updated:

• Query Graph: As described on Line 214, each encountered query is directly added as a node in the query graph. Additionally, previous queries that contributed to solving the current one (e.g., by providing helpful insights) are linked via directed edges. This construction captures not only semantic similarity but also temporal and causal relationships, representing how past queries inspire future ones.
• Insight Graph: Newly distilled insights from each incoming query are added to the insight graph. Edges are drawn from previously leveraged insights that contributed to solving the current query. Compared to query-level connections, this high-level topology provides a more precise mapping of transferable collaborative knowledge, enabling MAS to quickly locate relevant prior experience. G-Memory also performs periodic consolidation of the insight graph: First, recent queries are grouped into semantically related clusters $\{\mathcal{C}\_l\}$. For each cluster, all corresponding insights are retrieved from the graph: $\mathcal{I}\_l = \Pi\_{\mathcal{Q} \to \mathcal{I}}(\mathcal{C}\_l)$, which is then synthesized into a set of more general insight, $\iota_l^\text{merged}$, using the function $\mathcal{J}$ in Eq. (10) that merges their textual content and supporting query sets: $\iota\_l^\text{merged} = ( \mathcal{J}(\{\kappa\_k \mid (\kappa\_k, \Omega\_k) \in \mathcal{I}\_l\}), \bigcup\_{\iota\_k \in \mathcal{I}\_l} \Omega\_k )$. Finally, the insight graph is atomically updated by replacing the original set of insights $\mathcal{I}\_l$ with $\iota\_l^\text{merged}$. This is explicitly implemented in the merge_insight function of /mas/memory/mas_memory/GMemory.py in our provided codebase. This iterative refinement prevents knowledge fragmentation and ensures the long-term coherence of the system's memory.
• Interaction Graph: The interaction history associated with each query is stored in the interaction graph for future reference.

Weakness 3 & Question 4: Safety and Robustness Gaps: Potential risks from retrieving misleading insights/interactions are underexplored, despite the impact statement noting amplification of incorrect reasoning.

Thank you for your insightful observation! We would like to respectfully highlight that the current design of G-Memory includes at least two mechanisms to mitigate the potential negative impact of misleading insights/interactions:

• The consolidation process of insight graph will filter the useless insights and emphasize the impactful ones, as described in our response to Weakness 2.
• As the query and insight graphs evolve over time, correct and helpful insights/interactions naturally lead to more successful task executions, thereby accumulating additional edge connections. This process positions them as denser and more central nodes in the graph, making them more likely to be retrieved, as illustrated by nodes 45 and 55 in Figure 10. In contrast, misleading elements tend to have limited impact and remain on the periphery of the graph (e.g., nodes 36 and 37 in Figure 10). This update dynamic implicitly prioritizes reliable knowledge while suppressing noise.

Question 2: Are all new queries incorporated into the memory? If not, what criteria filter queries for updates?

Both the query and its associated interaction history are incorporated into the memory. However, the insights distilled from these interactions may be dynamically merged or pruned, as described in our response to Weakness 2.

Question 3: How is the correctness of insights verified, and does insight quality impact G-Memory's performance?

Thank you for your thoughtful question! We agree that the quality of insights can indeed influence the performance of G-Memory. While G-Memory does not explicitly evaluate the correctness of each insight (which is often impractical, as even human-generated insights may appear correct in the short term but prove inaccurate over time), our graph update mechanism implicitly addresses this issue. Specifically, helpful and positively contributing insights naturally accumulate more edge connections and are thus positioned as denser and more central nodes in the graph, while misleading ones are gradually marginalized. Additionally, the consolidation process of the insight graph further prunes or merges low-quality or redundant insights.

Hi Reviewer PY4E,

Can you please check if the author response has addressed your questions and provided the details you were looking for?

Thank you! AC