Minerva: A Programmable Memory Test Benchmark for Language Models

Thank you for your constructive feedback and positive assessment of our paper. Below, we address your main concerns:

Use of in-context examples

We intentionally did not include in-context learning examples, as our goal is to evaluate models' inherent memory capabilities, rather than their ability to perform few-shot learning. We think that including in-context examples would conflate memory with adaptation to in-context cues, making it harder to isolate the specific skills we aim to measure.

Prompt variation and sensitivity

We recognize that LLMs can be sensitive to prompt formulation. However, our primary goal in this work is to establish a standardized benchmark for evaluating memory capabilities, rather than optimizing performance through prompt engineering.

To assess the impact of prompt variations, we conducted a preliminary experiment where we tested different wordings for the same atomic task. We found that, as long as no heavy prompt tuning is involved, the impact on performance remains minimal. For example, in the word presence task, we obtained identical results from multiple models for the prompt variant "Is xxx present in the context?" compared to "Given the context, determine if xxx is present in the context." This suggests that minor wording differences do not substantially alter model performance.

Given these findings, we believe that fixing the prompts to a single, simple version (as shown in Appendix A) ensures consistency and comparability across models. While prompt tuning could enhance performance for specific models, that is a separate research question beyond the scope of this work.

Number of trials

The results reported in the paper are based on a fixed snapshot of the benchmark, meaning we used our program to generate a predefined set of 1110 evaluation instances, which we assume corresponds to what the reviewer means by "one trial." However, we generated multiple instances for each task (see Appendix B), ensuring a more diverse representation. To maintain reproducibility and consistency, we fixed the experimental settings as described in the paper.

We also want to emphasize that our benchmark can be used to generate any number of test samples and is fully extendable to different context lengths. We will open-source our code and also provide the exact snapshot of our data used in this experiment to facilitate further experimentation with alternative configurations.

We sincerely appreciate the reviewer’s positive assessment of our work and thoughtful feedback.

One key takeaway from our experiments is that different models exhibit high variance across atomic memory tasks, reinforcing the need for a diverse evaluation suite. In addition, we found that a major failure pattern is that models perform well on direct retrieval (e.g., finding an exact string in a long context) but degrade significantly when required to store, update, and apply memory dynamically. We believe this might be due to the mainstream attention-based architecture, which makes point-wise access (e.g., retrieving a specific token) relatively easy but lacks explicit memory mechanisms for storing the information and updates. Future models may benefit from architectures that explicitly model memory, such as those explored in the recent Titan paper.

Beyond memory retention, models also struggle with comparing, integrating, or transforming stored information, particularly when relevant facts are dispersed across different parts of the context rather than appearing in a single block, or when the task requires keeping track of updates to an entity’s attributes over time. These results suggest that attention alone may not be enough for effective long-context understanding. Future improvements might require structured memory systems, such as hierarchical memory representations to help models better handle complex memory tasks.

We will expand this discussion in the paper to further highlight how our evaluations can inform future model development.

We appreciate the reviewer’s comments and the opportunity to clarify our contributions.

Position of the paper and fitness with ICML

The reviewer raises the concern that our paper does not introduce new theoretical concepts. The paper does not aim to be theoretical. The nature of memory analysis in LLMs is empirical. However, we believe that a rigorous and comprehensive benchmarking framework is a significant contribution. Using static data benchmarks to test needle-in-the-haystack type of analysis is not sufficient especially in a rapidly evolving field like LLMs. Our study provides:

1. A well-defined taxonomy of memory-related tasks that categorizes different types of memory demands in LLMs, providing a structured way to assess and analyze LLM memory capabilities, as described in Section 2.

2. New insights into LLM memory behavior across various tasks, revealing critical limitations that have not been studied before. As discussed in Section 4 line 434 - 436 (left) and 385-394 (right), previous benchmarks and tests focus mainly on information retrieval, and lack a comprehensive evaluation on memory. Many other benchmarks are also static dataset benchmarks.

3. A programmable, extensible benchmark that enables future research on memory efficiency, potentially informing both theoretical studies and model development. This mitigates the risk of overfitting on benchmarks, and biasing the empirical evaluations.

Historically, benchmarks have played a pivotal role in advancing ML research. For example, ImageNet reshaped computer vision, and MMLU influenced LLM evaluation. Similarly, our benchmark exposes key limitations in memory-intensive tasks, which could inspire future advancements. Moreover, evaluation is explicitly within the scope of ICML’s call for papers.

Clarifying the distinction between simple search and other tasks

Thanks for the suggestion and we will add additional explanations and examples to better illustrate this distinction in the paper.

By "simple search," we refer to tasks that require only the ability to locate and extract relevant information from memory without additional processing (see line 105, left). In our benchmark, this includes tasks like string search, key-value search, and batch search (see Table 1 and Appendix A). For example, given a keyword x, the model retrieves the associated value y. This "locate-and-extract" ability resembles traditional search tasks, and thus we refer to them as simple search. This type of task has dominated prior work on LLM memory utilization, such as the Needle-in-a-Haystack task (see Section 4, line 419, left).

However, real-world applications of memory demand more than just retrieval. Our benchmark explicitly evaluates tasks that require models to recall, edit, match, compare, and track state, which extend beyond this simple search (see Table 1 and line 55-67, right). For instance: A writing assistant that rephrases or edits a paragraph operates differently from a system that merely retrieves a fact embedded in a document. A financial analysis tool that verifies consistency across multiple reports needs to compare and reconcile stored information rather than just extract it. A personalized assistant managing ongoing tasks must track state changes over multiple interactions instead of retrieving isolated details. In practice, memory-intensive tasks often require a composition of these abilities. Our benchmark not only evaluates these capabilities but also provides a structured taxonomy to identify model strengths and weaknesses systematically.

We appreciate your thoughtful feedback and the chance to clarify our contributions.

Insights from the benchmark

Our main goal is not to rank known models, but rather to introduce tests to evaluate different functionalities needed for LLM (agents). Any model can be (1) tested using our benchmark, (2) updated/retrained/fine-tuned to improve its functionalities WITHOUT the problem of being overfitted on the benchmark, and (3) have older and newer versions of the model compared fairly by rerunning the benchmark with fresh randomness.

While some of our results align with prior benchmarks, we disagree that our work merely reinforces prior knowledge. It is true that GPT-4o performed the best in our tests, as expected, but our findings provide meaningful insights beyond the simplification of “GPT-4o is better than GPT-4o-mini”. Specifically: (1) Models of similar sizes (particularly open-source ones) show notable performance variations across different types of memory tasks (See Figure 3-5) (2) Performance gaps vary by task, e.g. while models perform similarly in information fetching tasks, more challenging tasks like "spot the difference" reveal larger gaps. (3) Smaller models can outperform larger ones on certain tasks. Notably, LLaMA-3.1-8B and phi-3-small outperformed GPT-4-turbo in “patch the difference”, despite being smaller. This suggests that factors beyond sheer model size, such as architectural choices and training data, play a significant role in memory-related capabilities. Generating insights doesn’t require contradicting prior benchmarks. Instead, we aim to uncover nuanced model behavior patterns that inform development and evaluation strategies. This goes beyond using benchmarks merely for ranking models.

We also would like to clarify on the following regarding our experiment setups:

1. Variation in context length

We mainly fixed the context length to 4K tokens to highlight that models already struggle in many memory tests at this length. We have also shown in one of our ablation studies (Figure 6) how the number of operation steps/context length affects performance. We also have other ablation results that show a similar trend: performance was near perfect at shorter lengths, but significantly lower well before reaching what is typically considered "long context". We can add more results in the paper to clarify these findings. Additionally, since our benchmark is fully programmable and transparent, it can be extended to evaluate models at different context lengths.

2. Evaluation criteria

For the substring presence test, exact match means that if the model’s response (yes/no) matches with presence/absence of the substring, it receives a score of 1, otherwise 0. We believe this is a fitting metric for a binary classification task.

3. Variations in prompt wording

As discussed in our response to Reviewer RVnF, we conducted a preliminary experiment to assess the impact of wording variations and found that performance differences are minimal for instruction rephrasing. This supports our decision to fix prompts to a single simple version (in Appendix A) for consistency and comparability across models. So, based on this result, we observe that the main challenge is not in “understanding” the instruction, but rather in “performing the task”. All that said, note that (1) our benchmark is not a static dataset and a simple script can be added to our benchmark to generate many variations; (2) we will add confidence intervals around the results to show also robustness to instruction variations.

4. Deterministic setting

We deliberately used a deterministic setting for consistency and controlled comparisons. While we understand the concern that this might disadvantage certain models, we argue that: (1) A deterministic setting provides more stable and reproducible evaluation. (2) This choice is an evaluation setting, not a limitation of the benchmark itself. While we used a deterministic setup in this study, our benchmark is fully configurable. Users can modify decoding parameters to introduce stochasticity and explore its effects on different memory tasks.

5. Separation between proprietary and open-source models

We separate proprietary and open-source models because these groups tend to be more comparable within themselves. For example, the open-source models we evaluated are of similar sizes, making side-by-side comparisons more meaningful. This grouping allows for clearer analysis while still enabling cross-category comparisons where relevant.

Final remarks on experiment setups

Our benchmark is designed to be fully programmable, extendable to different context lengths, evaluation criteria, prompting strategies, or other configurations. We will open-source our code and provide a snapshot of our data to encourage further research. However, for the sake of reproducibility, we fixed the experimental settings as described in the paper to ensure consistency in our results.