S3 human study

In terms of the analysis, we were indeed limited by the budget. For the setting we presented in Figure 4a, the annotators are required to manually interact with the model round-by-round for multiple sessions. Therefore, analyzing one question for a single model could take more than 10 minutes on average. We further elaborate this challenge in the ethics statement below. In terms of reproducibility, we argue that our result represents best effort under the current LLM landscape with non-transparent API-based (or UI-based) accesses. We also argue that including the results is beneficial because they authentically represent the systems’ performance in a certain time period and sets up an informative reference for future open-source reproductions of the memory systems in the style of ChatGPT and Coze. Our comparison with offline reading also highlights the degree of challenges of building robust online memory systems, even for commercial products.

Based on your suggestion, we also extended the analysis to five smaller LLMs and present the results in the table below. We mainly evaluate the either long-context (LC) generation or generation with top-10 memory retrieval.

Overall, we observe that the performance is consistent with the results reported in the paper. For instance, long-context performance on LongMemEval_S is significantly lower than the oracle retrieval performance. In addition, the recommended fact-based key expansion and CoN consistently improves the performance.

S5 Question type definition

As defined in section 3.1, each instance is a 4-tuple (S, q, t_q, a). As such, we differentiate the task types not only based on S, but also based on their (q, a) distribution. Our final ontology ensures that all the instances of the same type have the same (S, q, t_q, a) distribution and format.

Specifically, for “knowledge-update”, the sessions either explicitly define the user’s old life state, or explicitly verbalize the new life state, or both. On the other hand, the “multi-session” questions are mainly aggregation or comparison questions that require the model to collect supporting facts at multiple places.

For “temporal-reasoning” questions, the (q, a) pairs are constructed such that the answers strongly depend on the timestamps of both t_q and the timestamps of the evidence sessions. For all the tasks that are not “temporal-reasoning”, we ensure that the answer is time-independent, i.e., the question could always be reliably answered with any set of arbitrary timestamps.

In our final revision, we will add a table to clarify how the task types are defined. This will also be clarified more when the readers are able to see the full released dataset.

Contribution

C1 Human-AI vs. Human-Human

Thank you for raising this question. We argue that there is a fundamental difference between building systems for dialogue modeling (human-human chat) and chat assistants (human-AI chat). Specifically, two stylistic aspects are different: topic distribution and information density.

Topic Distribution. For human-human chat, each side of the conversation has a personality. The conversation is driven by social interaction, information exchange, or relationship building. On the other hand, user-assistant chats are more likely to be information-seeking. In terms of the style, the information in human-human chats is also likely to be presented in a more colloquial style. Therefore, the two types of conversation represent significantly different distributions, which pose challenges even for language modeling.

Information Density. More importantly, human-AI chats challenge long-term memory with a lower density of personal information. For various chat assistant tasks, the user may provide long-context inputs, and the assistant can provide long-formed responses. In such scenarios, it is nontrivial for the memory system to decide what to memorize and how to organize the memory structure for more effective retrieval. This problem is more salient for agentic tasks, where the assistant is often provided with large observation contexts.

C2 Prior work on task-oriented dialogues

Thank you for the suggestions. We have revised the introduction and the related work to clarify this and cite previous works on task-oriented dialogues you have mentioned.

We would like to note that the major prior works are the papers on long-term conversational memory, and our previous draft assumed it in a number of locations. Datasets such as MultiWoZ, AirDialogue, CoQA, and SituatedQA do not evaluate chat assistants on the long-term memorization abilities with long-term dependencies between the sessions, which is the core of LongMemEval.

Specifically, SituatedQA evaluates question answering within a given temporal or situational context. Its challenges come from pinpointing the correct version of public knowledge, instead of organizing the user information during user-AI interaction. For CoQA, it evaluates dialogue understanding across multiple turns in the same session, and its history length is also very short by the current standard. Therefore, we argue that these datasets are less relevant than the related works discussed in the paper such as LoCoMo and PerLTQA.

In comparison, LongMemEval designs a unique method to embed personal details in extensible task-oriented dialogue histories without causing conflicts. We encourage the readers to envision LongMemEval as the counterpart to the “needle-in-a-haystack” test for long-term memory abilities, which is unique among the current literature.

Other Questions

Q1 Ethics statement

We have added an ethics statement section at the end of the paper. In the ethics statement, we discuss the human annotation standards we uphold and potential societal effects of the LongMemEval.

Q2 Limitations

Beyond the robustness issues mentioned in the newly added ethics statement, we also recognize limitations in both evaluation and modeling. On the evaluation side, further work could benefit from (1) introducing more complex multi-hop temporal reasoning questions, such as querying the contents in some session using another session and the relative time as the reference; (2) consider long-term evolution of the user’s relationships between other users; and (3) directly evaluate user satisfaction in the personalized chat. These scenarios pose a harder challenge than LongMemEval and would require significant additional efforts to accomplish. On the modeling side, future work would benefit from a fully online memory system that automatically resolves the knowledge conflicts during interaction. For these systems that read and write for every session, efficiency is also a significant concern, as the system would benefit from automatically deciding when and what to read/write.

We have incorporated this discussion into the updated draft (Appendix Section F).

Q3 Temperature and model version

We set the temperature to 0 for all the QA experiments. The GPT-3.5 version is gpt-3.5-turbo-0125 for Figure 4a.

Q4 Figure 4a

Yes, we have done a number of manual analyses after the human study. We find that the system tends to aggressively rewrite and remove memory items when GPT-4o is used as the model. After several sessions, a number of important details are likely to be wiped out. By contrast, we found GPT-4o-mini rewrites and merges fewer memory items, which causes more detail to be retained. We also found that for the human study we conducted, the performance bottleneck is the memory write instead of reading. In other words, as long as the model is able to retain the evidence statement in the memory, it is likely to answer the question correctly.

Q5 NDCG

Overall, we have found a high correlation between NDCG@k and Recall@k, with a few disagreements when the two metrics are used to evaluate query expansion methods in Table 3. This is mainly because the introduced time-based query manipulates the order using the timestamp metadata. In our implementation, the values with timestamps outside of the specified range will be moved to the last position, causing a low NDCG score. In this case, recall is the more informative metric. We will add a more detailed discussion in the final revision.

Q6 Timestamp sorting

We sort the sessions by the ascending order of their time (i.e., older sessions come first). We have done preliminary analyses on Llama-3.1 8B Instruct and found that sorting by the time order outperforms using random timestamp order or retrieval order. We thus stick with the practice for the entire paper as this method also presents the contexts in a logical order. This design is crucial when round is used as the granularity as it ensures that items from the same round are placed together.

Q7 Table 2 K = summary setting

No, we did not include K = summary under Value = Round in Table 2 because the user message in each round is succinct enough such that further summarizing is not required. We have fixed the wrong boldface in the table.

Q8 Confusing rounding

Thank you for identifying the ambiguous rounding issues.

11.4% is obtained from [(0.451-0.421)/0.421 + (0.495-0.499)/0.499 + (0.526-0.489)/0.489 + (0.722-0.550)/0.550] / 4. This is the gain in recall for value = round across different key settings and top-k settings. We were using Google Sheets for the average and it might produce some rounding discrepancies. We have modified it to 11.3% in the latest version.

Similarly, 6.7% is obtained from [(0.654-0.639)/0.639 + (0.707-0.651)/0.651 + (0.722-0.684)/0.684 + (0.797/0.721)/0.721] / 4. We have modified it to 6.8% in the latest version.

For the 4% improvement in retrieval metrics, we apologize that we used a wrong row to calculate the averaged result. The actual number should be [(0.644-0.582)/0.582 +(0.784-0.692)/0.692 + (0.732-0.706)/0.706 + (0.862-0.783)/0.783 / 4 = 9.4%

For the 5% improvement in QA accuracy, the more precise result should be [(0.657-0.615)/0.615 + (0.720-0.670)/0.670 + (0.638-0.6)/0.6 + (0.682-0.624)/0.624 + (0.566-0.518)/0.518 + (0.572-0.534)/0.534 + (0.714-0.67)/0.67 + (0.7-0.676)/0.676 + (0.588-0.592)/0.592 + (0.584-0.57)/0.57 + (0.53-0.524)/0.524 + (0.49-0.464)/0.464] / 12 = 5.4%.

We have revised the paper PDF accordingly.

Q9 Figure 4b and Figure 7

The results in Figure 4b and Figure 7 are not directly comparable. Specifically, Figure 4b uses LongMemEval_S while Figure 7 uses oracle retrieval (only the evidence sessions; explained in the previous replies and clarified in the revised paper).

For Llama-3.1 8B, we observe that CoN is in fact helpful when the memory items are provided in JSON. When the memory items are in natural language, CoN does not improve its performance. We hypothesize that the 8B model has limited ability and confuses the history with the current chat context when NL is used, thus failing to extract the notes effectively. This further suggests the importance of using JSON format to present the memory items.

Q10 Other typos

Thank you for these editorial suggestions. We have reflected them in the revised PDF. For the requested example, we will instead provide a discussion and combine Table 7 with Figure 5 in the final version (page 7).

We sincerely appreciate your comprehensive comments and are committed to improving the quality of the paper. Please kindly review the updated draft PDF along with the accompanying responses below.

As your weaknesses section simply summarizes the detailed comments, we directly address the detailed comments point by point.

Presentation

We have resolved your paper writing comments in the updated PDF (highlighted in orange) and we respond to the other questions and concerns below.

P1 Context reorganization

Moving more analysis to the main content

Balancing the sections was indeed a hard trade-off. In the submitted version, we decided to emphasize on the benchmark’s quality and its challenging nature. In the revised paper, we have moved Table 7 into the main text such that it can serve as an informative complement to Figure 5.

What is NL in Figure 7? What is the result reported in Figure 7?

NL in Figure 7 is natural language. We clarify that Figure 7 is measured with “oracle retrieval” with only the evidence sessions. This experimental design isolates the effects of imperfect retrieval. We have revised section 5.5 to further clarify these issues.

P2 Figures and tables

We have improved the content and placement of the figures and tables based on your feedback.

Why #Q instead of #data in Table 1?

For Table 1, we choose to use #Q because several datasets such as MemoryBank and LoCoMo construct a number of questions from each session. Therefore, it is hard to define a “datapoint”.

“session” in line 83 has a different meaning

In this paper, “session” always has the same meaning. The “session” in line 83 means that the evidence sessions for each problem in LongMemEval_m are embedded in chat histories that contain 500 sessions. It is a coincidence that LongMemEval also has 500 problems.

Figure 5 presentation and further examples

The goal of Figure 5 is to illustrate the concept of the memory framework in the most general case. Taking the indexing stage as an example, there are multiple specific designs (facts, entities, time, etc.) for keys and an exponential number of combinations. As a result, providing oversimplified examples would give the readers wrong impressions. In our revision, we have moved the original Table 7 to the main text to further clarify concepts in Figure 5.

P3 Emphasis and compound word overuse

We have significantly reduced the use of \textit and compound words throughout the paper, especially in the introduction. We hope these modifications improve the paper’s clarity and readability.

Soundness

S1 Formulation of the unified framework

We have provided a formulation of the system in section 4.1. Below, we formulate the system with more detailed explanations.

We formulate long-term memory as a massive key-value datastore $D = [(k_1, v_1), (k_2, v_2), ...]$, where the keys $k_i$ can be heterogeneous and the values $v_i$ might repeat. More concretely, a key could be discrete or continuous. In the discrete case, the key could be a sentence, a paragraph, a fact, or an entity, etc. In the continuous case, the key could be e.g., the model’s internal representation under some inputs.

For the three stages, we formulate them with four functions: $\mathcal{I}$, $\mathcal{Q}$, $\mathcal{S}$, $\mathcal{R}$. The offline index function $\mathcal{I}(S, D)$ converts a session $S$ into a series of key-value tuples. Alternatively, the online index function $\mathcal{I}_{online}(S, D)$ updates D with the key-value tuples extracted from S, potentially removing or editing the existing items in D. The query formulation function $\mathcal{Q}(q)$ converts a natural language user query $q$ into a representation $q’$ that function $\mathcal{S}$ could leverage. The salience scoring function $\mathcal{S}(q’, D)$ orders D by their relevance to $q’$. Potential initializations of $\mathcal{S}$ include ranking dense text embedding, graph-based ranking, and so on. Finally, the reading function $\mathcal{R}(M, D’)$ combines the model M with D’, the most relevant part as ranked by the salience scoring function $\mathcal{S}$ and generates a response. The methods outlined in the columns of Table 7 could be seen as different ways to instantiate the four functions $\mathcal{I}$, $\mathcal{Q}$, $\mathcal{S}$, $\mathcal{R}$.

S2.1 Baseline comparison

We first observe that a number of baselines are conceptually captured by the unified framework (Table 7). For example, the standard in-context retrieval augmentation is the case with K = V = session or K = V = round. For another example, the MemoryBank method is the case with K = V = user facts, with natural language format and no CoN for reading.

For this paper, we make the conscious choice to not call these baselines by their names because their out-of-the-box performance is poor. For instance, MemoryBank has nearly zero accuracy on LongMemEval due to its tendency to memorize topics and high-level information instead of user details. In our preliminary experiments, we have to make substantial changes to improve its performance, such as using round as the values, using in-context learning for fact extraction, and reading-stage optimizations. At that point, it would be unfair and confusing to call it by the original name.

Due to these reasons, we opt for independent formulation and implementations to enable controlled comparisons and optimizations of different design factors. We argue that this presentation provides the readers with clearer understanding of the challenges of LongMemEval on different memory components.

S2.2 and S4 - Dataset size and question composition

We emphasize that our 500 human-annotated questions are high quality seed data. Each question itself could be embedded in an infinite number of chat histories to test the memory ability at arbitrary length. In terms of the size of the expert-curated data, we notice that it is on par with a number of (influential works such as HumanEval ([1], 164 questions), IFEval ([2], 500 prompts), and GPQA ([3], 448 questions). In fact, if one considers the original “needle-in-a-haystack” test, it essentially only contains a few questions, paired with greatly varying context setups. Furthermore, the current dataset size allows us to derive statistically significant results, which is further discussed in the general response.

For the question composition, we argue that comprehensiveness is one of the fundamental goals of LongMemEval. Driven by this goal, we spend a great amount of human effort to propose challenging and diverse questions, as well as balancing different question types. In fact, in the early stage of benchmark construction, we manually removed hundreds of low-quality questions. Each question in LongMemEval could be considered as a full-fledged test that represents a unique scenario with flexibly configurable history.

[1] Evaluating Large Language Models Trained on Code. Chen et al., 2021.

[2] Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models. Li et al., 2024.

[3] GPQA: A Graduate-Level Google-Proof Q&A Benchmark. Rein et al., 2023.

Dear Reviewer,

Thank you again for your efforts in reviewing this paper. We would also like to respond to the general weaknesses you raised to ensure that we are aligned with the big picture, although we have responded to the detailed points in the previous response.

Paper Presentation and Proofreading

We have thoroughly revised the paper and fixed the presentation issues such as table/figure arrangement, emphasis or compound word overuse, and rounding issues. We have also improved the content arrangement to enhance clarity, such as arranging the section contents in 3 and bringing in the original Table 7 (now Table 2) to the main text. Finally, we have added an ethics statement (after conclusion), a limitations section (section F), and statistical testing results (in the general response).

Balancing the sections was indeed a hard trade-off. We argue that as LongMemEval is a new challenge, it is important to emphasize the logic behind the proposed testing and justify its difficulty as well as quality. We therefore allocate five pages for this part. Meanwhile, instead of two pages, our empirical study actually spans page 6-10 (starting from section 3.4). We have presented comprehensive results in each subsection, along with highlighted discussions of the most important takeaways. While the paper’s main text is self-contained, we also present further results and analyses in Appendix B, D, and E for interested readers.

Baseline Coverage

As this paper is a resource and evaluations paper, there is no explicit “baseline” to compare with. As explained in the response to S2.1, the memory system studied in this paper subsumes previous methods such as RAG, CoN, and MemoryBank (further explained in Table 2 in the revised PDF). We intentionally avoided explicitly naming these methods because directly applying them results in nonsensical performance. They have to be modified substantially to achieve non-trivial performance on LongMemEval.

Framework Uniformity

We have provided a detailed formulation in our responses to S1, and we have added Table 2 to clarify the discussions in Figure 5. We note that uniformity means that various memory systems could be viewed as specific instantiations of the framework, as shown in the Table 2. We have also added an example in the general response to further illustrate our framework.

Contribution

We would like to highlight that LongMemEval is a novel and challenging benchmark for long-term memory of chat assistants. LongMemEval’s novelty lies in a combination of (1) diverse questions that comprehensively cover chat assistant memory abilities such as knowledge update, abstention, and temporal reasoning and (2) long and freely extensible chat history without invalidating the questions. These two features distinguish LongMemEval from other benchmarks such as MultiWOZ, AirDialogue, CoQA, and SituatedQA.

We provided a more detailed comparison in our previous response and further highlighted the comparison with the paper you suggested in our revised PDF to ensure clarity. We believe LongMemEval is distinct from those works and has a unique contribution to the field.

If you have additional questions that we can further clarify, please kindly let us know. Thank you!

Dear Reviewer,

We are glad that our previous response seemed to have resolved most of your concerns. We address your follow-up questions below.

Number of questions vs. data points

In LongMemEval, #Q is smaller than #data points (contrary to your intuition), because each question has its own unique set of evidence sessions and LongMemEval supports constructing a large number of test instances (i.e., “data points”) by combining the evidence sessions with other history sessions. In our evaluation, LongMemEval_S and LongMemEval_M each contain 500 data points. Reporting the number of data points could actually risk inflating the dataset size, and #Q is a more modest way to present the dataset sizes for LongMemEval.

“50k” sessions in Table 1

In Table 1, “50k sessions” refers to the pool of unique evidence sessions and other generated history sessions. The history sessions used in LongMemEval S and M are sampled from this pool. Since we release the entire LongMemEval pipeline as a resource, we use 50k as the total number of sessions. In practice, LongMemEval_M samples roughly 500*500 = 250k times from the pool and covers roughly 95% of these sessions.

Unified memory framework formulation

Thank you for recognizing the validity of the definition and that moving Table 7 to the main text clarifies the uniformity concerns. We will combine the definition with Table 2 in the main text in our final revision.

Dataset size and composition

LongMemEval prioritizes the quality of the questions instead of the size. Compared to previous work, LongMemEval has a uniquely comprehensive memory ability coverage, and extensive efforts are spent to ensure (1) information-dense user messages and (2) the integrity of the question even with many history sessions. We handle these challenges through the novel attribute-controlled data creation pipeline that generates attribute-specific user backgrounds. Extra human efforts are also required to ensure the integrity of time mentioned in all evidence sessions. These features are unparalleled by existing QA/IE literature. Partially reusing previous datasets is also prone to question inconsistency, potential test data leakage, inheriting the bias from previous works as well.

As LongMemEval is already comprehensive, difficult, and enabling statistically significant comparisons, we argue that further including questions of potentially lower quality harms the democratization of evaluation. In fact, obtaining a single test result on LongMemEval_M already requires 1.5M * 500 = 750M input tokens, which costs 1250 USD for a memory system purely based on GPT-4o. The cost is ~150 USD for LongMemEval_S. We argue that the current size strikes a good balance between evaluation accuracy and the viability for everyone in the community to use.

Knowledge update vs. multi-session questions

The major reason for keeping the two question types distinct is that they are created from distinct metadata schemas, which lead to distinct question distributions. Specifically, the metadata for knowledge update is a pair of facts (S, R, O) -> (S, R’, O’) where R’ or O’ or both is modified to invalidate the original R or O. Here (S, R, O) represents (subject, object, relation). In contrast, the metadata for multi-session questions is a list of facts (S, R, O_1), (S, R, O_2), …, (S, R, O_n).

We admit that the two examples presented in Figure 1 accidentally represent the overlap between the two question type distributions, which is infrequent in the full LongMemEval dataset. We will replace the example in Figure 1 to avoid confusions. For your reference, we provide two knowledge update questions that do not fall into this overlap:

Example 1 (R changes while O remains the same)

• Previous fact: my mom finds my grocery list method too difficult to use.
• New fact: my mom is using the same grocery list method as me.

Example 2 (both R and O change)

• Previous fact: I see my therapist Dr. Smith every two weeks.
• New fact: I see my new therapist Dr. Brown every week.

User-AI chat literature

We are aware of several works on using user-assistant chat data for training better chat assistants [1-3]. These works are marginally relevant to this paper’s theme of long-term memory, but illustrate the wide applicability of the user-AI chat format.

[1] OpenAssistant Conversations - Democratizing Large Language Model Alignment. Köpf et al., 2023.

[2] Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. Ding et al., 2023.

[3] LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset. Zheng et al., 2023.

Presentation issues

Thank you for recognizing that most of the presentation issues have been solved. We apologize for the typo in Table 3 and the duplicated bibtex entries. We will fix these minor issues in the final revision.

We hope our response further clarifies your questions, and please let us know if you have any other concerns.

Thank you,

All Authors

Thanks for the reply. I am still leaning positive towards this work. Good luck.

Thank you for acknowledging our effort to formulate and evaluate the long-term memory challenge. We address your insightful questions below.

Chat history quality

The goal of our history construction process is to simulate diverse yet consistent chats between a specific user and a chat assistant. To this end, we decompose a user’s persona into a number of attributes, and simulate task-oriented sessions along each attribute. During the history construction, beyond the evidence sessions, we draw the rest of sessions from (1) sessions with unique attributes and (2) public task-oriented chats that generally do not reflect user persona. We conducted a preliminary manual analysis of 50 generated histories and found that the pipeline can consistently produce a coherent history. More importantly, this pipeline minimizes the chance of any history session conflicting the information in the evidence sessions, which suffices for the integrity of the problem.

Timestamp quality

We uphold a high standard for timestamp consistency. Our annotation has a specific timestamp annotation step where we manually inspect all the questions. For all the questions that involve temporal reasoning or temporal mentions, we assign a fixed timestamp to each of the evidence sessions and the question. During history construction, the other sessions are randomly assigned with timestamps such that they interleave with the evidence sessions. This ensures the questions are always correctly answerable.

On the other hand, we ensure that the answers to other questions are timestamp-independent. Logically, they are answerable with arbitrary timestamp assignments. In practice, these questions in LongMemEval_S and LongMemEval_M have their chat history spanning within only three months to minimize potential confusions.

We will highlight these efforts in the final revision of the paper.

Uniformity of the proposed memory framework

Our framework is indeed not a monolithic architecture, but its uniformity stems from the ability to represent different types of memory-augmented chat systems. We outline the representation in Appendix Table 7. We have also included a formal definition in the response to reviewer LhuP to clarify the framework.

We argue that this framework allows us to have a clear view of the important design choices made by current literature and enables us to design controlled experiments on their effectiveness.

Long-term memory models

Thank you for this suggestion. As the major contributions of the paper are benchmarking and basic evaluations, we leave the investigation of advanced memory architectures to the future work. However, our memory evaluations are still general enough that informs a number of RAG-based memory models such as MemoryBank [1]. During the rebuttal period, we have also extended our evaluation to more LLMs. Please find the results in the general response.

[1] MemoryBank: Enhancing Large Language Models with Long-Term Memory. Zhong et al., 2023.

Thank you for acknowledging our work’s evaluation effort and raising insightful questions.

Quality control for human-AI talk

We argue that for LongMemEval to be effective, the core qualities to ensure are that (1) the questions, answers, and timestamps are logically correct and (2) the evidence statements are correctly reflected in the evidence sessions. We spend significant human efforts on each and every question to ensure these two properties. We have implemented a check to detect repetitions and stop the session simulation accordingly. However, we do not have other sophisticated quality controls, as it is beneficial to evaluate the chat assistants in real-world scenarios where these natural artifacts could occur.

Relationships between the considered aspects of memory ability

Thank you for raising this great question. Among the five abilities we evaluate, we find that information extraction (IE) is the fundamental ability. In fact, it covers all the questions in LongMemEval. Beyond IE, the other abilities have disjoint questions to represent them. We believe that these four abilities represent independent abilities that are all crucial to successful chat systems. In the table below, we analyze the performance by skills of GPT-4o and Llama-3.1 8B Instruct. LC = long-context. For the optimizations, we use top-10 round-level retrieval with fact expansion for key indexing, time-aware query expansion, as well as chain-of-note for memory reading.

As shown in the table, we find that while our optimizations significantly improve the performance, the abilities other than single-session IE have large rooms for improvement. For GPT-4o, knowledge update and abstention are the hardest abilities to improve. One potential reason is that it is tempting to generate a plausible answer for these questions based on partially retrieved knowledge. For the Llama-3.1 8B Instruction model, its multi-session reasoning ability limits the performance even with high quality memory retrieval. Finally, we also remark that under the LLM may be able to correctly abstain under poor memory retrieval, which complicates the evaluation and motivates us to propose evaluating the retrieval stage separately from generation.

Thank you for your great attention to the paper and insightful review.

Human interventions

We recognize that benchmark involves a lot of human intervention. However, we encourage the readers to view this as an advantage instead of a limitation, as this is the standard approach taken by numerous influential works (such as [1]). In constructing LongMemEval, we ensure that (1) all the questions are correctly answerable; (2) the evidence statements are clear and correct; and (3) the timestamps are logically correct for the related question. These crucial qualities are extremely difficult to guarantee without human interventions.

[1] Evaluating Large Language Models Trained on Code. Chen et al., 2021.

Multi-session specially considered in Figure 6

In Figure 6, we highlighted multi-session because the preference on the granularity is inconsistent with all the other tasks and the averaged performance. However, its preferences for the other design choices align well with most of the other tasks as well as the performance average. Therefore, we treat value granularity as an intriguing case and take special care to mention the results of multi-session.

Benchmarked models

We note that for all the QA experiments we always use three models as the reader model: GPT-4o, Llama-3.1 8B Instruct, and Llama-3.1 70B Instruct. These models represent the strongest models on the market of their sizes. For the retrieval-related experiments, we mainly used Llama-3.1 8B Instruct as we did not find significant differences between the 8B and 70B version and 8B is better efficiency-wise. We found GPT-4o performs much better than Llama-3.1 8B Instruct only for query expansion and thus discussed its results.

During the rebuttal period, we have also extended our evaluation to more LLMs as the reader models. Please find the results in the general response.

Writing issues

Thank you for identifying the issues and we have fixed them in the revised PDF, including Figure 2a, Figure 3, Table 2, and line 468.