Positional Biases Shift as Inputs Approach Context Window Limits

Thank you for your thoughtful comments.

Generalizability to real-world tasks and new results. The use of synthetic datasets is a precedent that was set by previous work Levy et al. 2024; we follow this precedent in our paper. The advantage of synthetic data is that (a) we can precisely control the position of the relevant information and (b) the model cannot rely on knowledge memorized in pretraining. New results: Nonetheless, to investigate the questioned generalizability, we constructed a dataset by concatenating Wikipedia pages, such that only one of the pages could answer each retrieval question. Addressing point b) above, we precede this with a test if the model is able to answer the question without the wikipedia page via its in-weight knowledge and discard such cases. This yielded a test set of 1500 samples. The findings are consistent with those on the synthetic datasets in the paper (see new material Section 3), suggesting that the same mechanisms are at play in more real-world settings.

Causality of retrieval and reasoning errors. Our main goal was to uncover how relative input length drives positional biases. During that investigation we noticed a close relation of retrieval- and reasoning positional biases (lines 302-306 in the manuscript). This empirical overlap suggested that retrieval failures might propagate directly to reasoning mistakes (e.g. Karpinska et al., 2024; Levy et al., 2024), motivating an additional strictly supplementary causality check for context.

Following the reviewer’s suggestion, we measured P(Reasoning=correct | Retrieval=incorrect), i.e., the probability of a correct reasoning answer when the corresponding retrieval answer is wrong. If retrieval truly gates reasoning, this conditional accuracy should sit near the 0.5 chance level of our balanced binary tasks. The conditional probabilities for Llama 3 70B, Gemma 2 27B, Mistrall Small 3 24B and Llama 3.1 8B are 0.54, 0.48, 0.44 and 0.38, respectively. While Llama 3.1 8B shows a lower conditional accuracy, the values lie around the 0.5 baseline, supporting the hypothesis that retrieval failures largely determine reasoning errors. While this causality check is secondary to our main focus on input-length effects, it reinforces the interpretation that the positional biases observed in reasoning are downstream of those in retrieval.

Deeper analysis of the cause of the phenomenon and its implication. Our primary goal was to explain discrepancies in prior work on positional biases effect through systematic empirical analysis. Understanding why such patterns occur first requires robust testing - but evaluating models at full context length is resource-intensive (e.g., ~$10,000 for GPT-4 Turbo; 80 GPU-hours on 4xH100 per dataset for Llama 3.1–70B). These resource and time demands naturally limit the scope of any single study. Our findings have concrete implications for model evaluation and deployment: by introducing relative input length, we clarify when positional biases emerge, enabling more reliable benchmarking and better model choices for long-context tasks like retrieval-augmented QA.

That said, we agree with the reviewer that understanding how relative input length affects LiM is a valuable next step and will add a discussion in the paper. A plausible explanation, supported by Wu et al. (2025), is that the LiM effect specifically arises from training data where information often appears at the beginning or end. Since models like Llama 3 (see Llama Team), are first trained on shorter sequences and only gradually adapted to longer ones, the LiM effect in the early context window may be a training artifact. The early training emphasis on shorter inputs could lead to stronger position sensitivity in the first half of the context window - possibly explaining why LiM weakens as input length increases.

Why not use evenly spaced relative input lengths? We chose relative input lengths (0.06, 0.12, 0.25, 0.38,...) to align with Levy et al. (2024), where absolute input lengths (500–3000 tokens) map to our relative spacings (0.06–0.38) for an 8K context window, enabling direct comparison. We also tested evenly spaced values (e.g., 0.1, 0.2, 0.4, 0.6, 0.8, 1) and found the qualitative patterns unchanged (see new material Section 3.1); we will include these results in the paper.

Comparison between model families and sizes. Please refer to the general response.

LiM as a relative concept. We follow the original definition of LiM from Liu et al. 2023 by describing it as the difference between accuracy in the middle and at the beginning/end. LiM does not disappear because the middle gets better, but rather disappears even though middle accuracy also declines - driven by a collapsing primacy bias. See the corresponding remark in lines 269 ff of the main paper.

Elaborate on model and dataset differences. Thank you for the suggestion - we will, of course, include all detailed (non-averaged) results (see new material Section 2, p. 2-4) in the appendix and add a discussion of significant differences in the manuscript. Overall, our findings on positional bias are consistent across both models and datasets, even when examining individual conditions. Retrieval tasks show highly stable patterns—i.e., a Lost in the Middle (LiM) effect for relative input lengths up to 0.5 and steep primacy drops beyond that point. This likely reflects the relatively uniform complexity of retrieval tasks, which does not vary substantially across datasets. Reasoning tasks also display consistent positional trends, but detecting these effects becomes more difficult when model performance approaches chance level, as random guessing can hide sensitivity to input position. This issue is most evident in more complex datasets like Simplified RuleTaker and Box. Nevertheless, we still observe strong recency biases and sharp primacy drops with increasing input length, confirming the robustness of our findings. We will add this discussion to our manuscript.

More systematic selection of models. We appreciate your feedback. Please see evaluation on more models in the general response and for visualizations of the requested experiments see new material here.