SeCom: On Memory Construction and Retrieval for Personalized Conversational Agents

• Dialogue Scope Limitations: Conversational agent should encompass a broader range of dialogue types including task-oriented conversations, chit-chat, role-playing dialogues and conversational question answering. Currently, the evaluation is restricted to question-answering tasks, specifically single-turn interactions. This narrow focus undermines the conversational agent's versatility and applicability in real-world scenarios. To provide a more comprehensive assessment, additional experimental results across various dialogue types should be included.Task-oriented datasets like MultiWOZ, open-ended conversation datasets like PersonaChat or multi-turn question answering datasets like CoQA are needed.
• Evaluation Methodology: The evaluation methods primarily rely on auto-machine metrics such as bleu, roughl, bertscore and gpt4score based on large language model(LLM), which has significant limitations and may not fully capture the nuances of human interactions. The absence of human evaluations is a significant. Incorporating human assessments such as response relevance, coherence and factual accuracy would provide a richer perspective on the conversational agent's performance. Moreover, establishing a consistency measure between human evaluations and auto-machine metrics is crucial. Given the large language model often generate lengthy responses, some of which may be valid yet not included in the standard answers, human insights could enhance the credibility of the results.
• Inclusion of Recent Methods: The evaluation baselines currently lack consideration of the latest memory-enhanced methods, such as MPC and COMEDY discussed in the related work. These methods represent significant advancements in the memory with the compression of long term conversation and should be included in comparative analyses to ensure a robust evaluation framework.

1. Using segment-level content as the dialogue unit is not novel.
2. The dialogue segmentation method employed is an integration of existing methods, with no specific improvements or optimizations tailored for dialogue memory purposes.

• Based on my understanding, SeCom is only tested to work with GPT-4 as the segmentation model, which is massive, costly, and thus unrealistic for many real-world applications. I would like to see a discussion on how well the framework can work without GPT-4-level models, even with some fine-tuning like in [1].

• While adding a denoising stage is an innovative proposal, there is no technical novelty in the denoising methodology itself as this work directly uses a previous prompt compression method.

• Other design aspects of the memory retrieval are ignored. For instance, it seems that this work always uses the value itself as the key, while a more advanced indexing approach could possibly reduce the need for a more “correct” value granularity [2].

[1] Dense x retrieval: What retrieval granularity should we use? Chen et al., 2023.

[2] HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. Gutiérrez et al., 2024.

1. While the paper proposes segment-level memory construction, it lacks in-depth analysis of how to determine the optimal segment granularity, and the specific impact of different granularities on performance is not fully explored.
2. The paper does not thoroughly explore the performance of the SECOM method in dialogue systems of different scales or across different domains. Specifically, the scalability and generalization of the model in resource-constrained environments are not analyzed in detail.