NoLiMa: Long-Context Evaluation Beyond Literal Matching

First of all, we want to thank the reviewer for their thorough and detailed review and constructive feedback.

[Limited needle set]: While we agree the set is limited, many comparable works—such as RULER or vanilla NIAH—use even fewer or similarly limited questions.

[The Eiffel tower painting example]: As noted in 148–150 (2nd Col) and Appendix A, the keywords W_q and W_n were carefully chosen to ensure unique and true associations, which is also reflected in the near-perfect base scores of top models.

[CoT Effect]: Our point is not to dismiss CoT, but to highlight that it’s not a fully robust solution in this setting. While it does improve performance, the gains diminish as context length increases, which remains a core challenge.

[Other models]: Gemini 2.0 and Deepseek R1 were released after or near the submission deadline. Gemini 2.0 released on February 5th (6 days after the ddl.) and Deepseek R1 on January 20th (10 days before the ddl.). Nevertheless, we will include additional results on Deepseek R1 (distill) and GPT o1 and o3-mini models to the CoT section of the camera-ready version. On a challenging NoLiMa subset, o1, o3-mini, and R1-distilled-70B all drop below 50% of the base score despite scoring near 100% on the base task, highlighting the difficulty even for reasoning-models. Also, for the main evaluation in Table 3 we will add results on Gemini 2.0 Flash. Moreover, the evaluation code and dataset will be publicly released to enable testing on future models.

[In Figure 2 is 0% depth...]: In Fig. 2(a) and (b), 0% depth marks the start of the sequence. In the last 2K (tokens depth), 0 means the end (Fig. 4) -- We'll clarify this in the camera-ready version.

[What does "minimize sensitivity to tokenization problems" mean]: Since the answers to the dataset questions are character names, relying on a fixed name could introduce bias in model performance. Some models may split names into more sub-tokens than others (e.g., "Ve ro nica" vs. "Vero nica"). Rotating character names helps reduce potential biases caused by this tokenization variance.

[Distractor filtering example and the removal process in "Conflicting Information Filtering"]: We can include some examples in the appendix. Here's one from the stories before filtering:

“…Yale, but then he (Steve) rebelled, cashed in his med-school scholarship, and went to Paris to study photography…”

For the question “Which character has been to France?, this passage introduces a conflict with the needle. Steve is a valid answer based on this span, but we want the model under evaluation to select only the needle as the relevant fact. To prevent this, the filtering model flags such conflicts—where additional spans introduce alternative correct answers—and we remove the entire conflicting span, expanding to the nearest sentence boundaries to preserve fluency.

[Is Figure 2 averaged over all models]: No, Figure 2 is only for LLaMA 3.3 70B. A high-resolution sweep, like Fig 2—since it involves twice the placements—would be too costly on closed models.

[Potential Inclusions]: Thank you for the great feedback! We can include the one-hop vs. two-hop breakdown for Table 3 and the unsmoothed Figure 2 in the appendix. Regarding question difficulty, it tends to vary with keywords and model behavior. As for observable patterns, as discussed in the paper, two-hop and inverted questions are generally more challenging.

First of all, we want to thank the reviewer for their thoughtful review and constructive feedback.

[Our works contribution compared to RULER]:

• Our related work section highlights how extensively literal matching is involved across various long-context benchmarks—including RULER.
• We show that literal matches play an impactful role in affecting the results (in Section 4.4.4) an analysis that cannot be done with RULER due to their existing literal matches.
• Since RULER is heavily based on lexical matching, it does not support an analysis of latent hops (4.4.1) and needle placement with reasoning variability (4.4.2). We would appreciate it if the reviewer could describe how such an analysis could be done on RULER.

First of all, we want to thank the reviewer for their thoughtful review and constructive feedback.

[32K limit]: NoLiMa has no inherent context length limit. As stated (lines 200–206, 2nd col.), haystacks are built from random story snippets (using the filtered stories) and can be extended to any length. Since all models drop below the 0.85×base score at 16K and 32K (10 out of 12 drop below 0.5×base score), extending further offers little to no extra insight.

Nevertheless, to demonstrate this, we will include results at 64K and 128K for two models: GPT-4o, the top-performing model in Table 3, and Gemini 2.0 Flash, a new model we plan to add.

[Differences with [1]]: While [1] requires global reasoning over entire novels, NoLiMa focuses on the retrievability of single facts. Notably, questions in [1] often include key cues (e.g., character names) that help narrow down relevant spans—similar to our multi-choice setup in Section 4.4.4. Even without strong literal overlap, the presence of answer choices allows the model to locate relevant regions and reason accordingly.

First of all, we want to thank the reviewer for their thoughtful review and constructive feedback.

[Multi-hop retrieval]: Multi-hop retrieval, which involves fact chaining, is discussed in our "Related Work" and "Introduction" (Hsieh et al., 2024; Levy et al., 2024). NoLiMa focuses on a prior step—locating relevant fact(s) from long contexts, which is a prerequisite for effective multi-hop reasoning.

Note: The U-NIAH paper was released after the submission deadline.

[Insights on attention weights]: Analyzing attention weights at long context lengths is extremely memory-intensive. For example, with 8K tokens, a LLaMA 3.1 8B model yields 32(layers) × 32(heads) × 8000**2 = 65 billion weights (~260 GB) for a single input. That said, we did limited analysis comparing inverted vs. default cases on some examples. We found that in inverted settings, W_n gets relatively more attention than the character name when W_n appears earlier—loosely supporting the theory in lines 300–306 (col 2). But due to high computational cost and the analysis being somewhat outside our main scope, broader experiments (e.g., on LLaMA 3.3 70B or full dataset or longer lengths) weren't feasible.

[Non-lexical synonyms]: While using non-lexically matched synonyms helps avoid literal matching, their embeddings are often similar enough to still act as a match. Moreover, not all our examples rely solely on hyponym-hypernym reasoning—some, like the dietary cases (e.g., lactose intolerance <==> milk), require commonsense reasoning.