RAGGED: Towards Informed Design of Scalable and Stable RAG Systems

We thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of our empirical contributions and insights into retrieval dynamics, particularly the introduction of RSS and RSC as tools for understanding reader robustness and retrieval scalability.

1. Symmetry of RSS

Thank you for the insightful observation. While it is true that adding lower-ranked documents (right side) and removing top-ranked ones (left side) may have different theoretical impacts, our empirical results show that this asymmetry is not consistent in practice. For example, LLAMA2-7B degrades more on the left (removing documents), whereas LLAMA3-70B degrades more on the right (adding documents). This variation supports our use of a symmetric window, which offers a model-agnostic and balanced way to measure stability. We will clarify this in the revision.behaviors. We will clarify this rationale in the revised manuscript.

2. Clarifying Design of RSC

RSC is defined as the product of two complementary components: 1) the last retrieval depth before plateau/decline, which captures how far the model scales before benefits taper off, 2) the cumulative performance gain up to that point, which quantifies the total utility derived from increasing retrieval.

By multiplying these factors, RSC distinguishes clearly between models that achieve quick but limited improvements (early plateau) and those that sustain meaningful improvements over deeper retrieval depths.

The hyperparameter ε, it is intentionally user-defined, similar to early stopping in training. It allows users to specify what constitutes a meaningful gain, based on their deployment trade-offs. We chose ε = 0.5 for RSC based on empirical analysis: the standard deviation of F1 across all models and k values is consistently below 0.5 (mean ≈ 0.38, max = 0.46). This makes 0.5 a conservative but meaningful threshold that exceeds metric noise. Moreover, increasing ε to 0.6 or 0.7 preserves model ranking, indicating robustness to the threshold choice.

3. F1 vs. Semantic Metrics

We recognize that unigram-based F1 may underrepresent semantic correctness. To address this, we validated our findings using an LLM-based semantic correctness metric on a subset of responses (Appendix H), and found strong alignment with the trends captured by F1. We will make this point clearer in the revised version.

4. Memorization of Dataset by LLMs.

Thank you for the comment. We agree that newer datasets reduce the risk of memorization. Our original evaluation includes NQ (2019), HotpotQA (2018), and BioASQ (2023) to cover open-domain, multi-hop, and biomedical QA settings. To extend this, we incorporate a 2024 dataset, CRAG, as suggested by reviewer jqB4. We ran preliminary experiments on CRAG using two representative readers: FLAN-T5 (of improve-then-plateau behavior) and LLAMA-2 (of showing peak-then-decline). These behavioral trends remain consistent on CRAG, suggesting that the retrieval-depth dynamics generalize well to newer datasets. We will include full CRAG results in the camera-ready version.

4. Applicability to Newer LLMs and Time-Sensitive Queries

We conducted additional experiments on dynamic questions in CRAG using both FLAN-T5 and LLAMA, and observed the following:

• FLAN-T5 maintains its improve-then-plateau retrieval-depth behavior on dynamic questions, consistent with its pattern on static ones. This suggests strong generalization and robustness to retrieval noise across domains.

• LLAMA, in contrast, shifts from a peak-then-decline trend on static questions to an improve-then-plateau trend on dynamic questions. We attribute this to differences in internal knowledge: on static questions, LLAMA already knows the answer and retrieval introduces redundancy or contradictions, degrading performance at higher depths. On dynamic questions—where LLAMA lacks internal knowledge—it relies more heavily on retrieved context, and even noisy documents provide helpful signal, leading to continued improvement.

This contrast reinforces the value of our reusable framework: even when retrieval dynamics vary across models or tasks, RSS and RSC offer a principled way to detect, quantify, and interpret these differences. Rather than assuming fixed behavior, our reusable framework and metrics help uncover how model robustness and scalability shift across domains.

Wwe expect newer LLMs—particularly those fine-tuned for retrieval-augmented tasks—to exhibit higher RSS and RSC due to improved handling of noisy or diverse context. Regardless of the model architecture, however, we believe our metrics remain essential tools for diagnosing retrieval sensitivity and informing robust, real-world RAG deployment.

We sincerely thank the reviewer for the constructive and thoughtful feedback. We appreciate that the reviewer recognizes that the paper resolves open questions in existing works and finds all the claims to be well supported.

1. Expanding to Newer Embedding Models

We appreciate the suggestion to evaluate newer embedding models. We are in the process of integrating models from the MTEB leaderboard, including leading open-source options like Linq-Embed-Mistral. However, due to the scale of the corpus (~111M passages) and the limited resources and time, we were unable to complete these runs within the rebuttal period. We are actively running the experiments and will include updated results in the revision. We agree that this extension will further strengthen our analysis.

2. Evaluating with CRAG

Thank you for highlighting the importance of more modern evaluation datasets. We agree that CRAG offers a more recent and challenging benchmark for RAG evaluation.

To assess generalization, we ran preliminary experiments on CRAG during the rebuttal window using two representative readers: FLAN-T5 (of improve-then-plateau behavior) and LLAMA-2 (of showing peak-then-decline). These behavioral trends remain consistent on CRAG, suggesting that the retrieval-depth dynamics generalize well to newer datasets. One difference is that the overall performance is worse (shifted down) compared to the performance we observed in our paper, which is as expected since this is a newer dataset. We will include full CRAG results in the camera-ready version.

3. In Figure 5, FLAN-T5 RSS is at 0.99. Do you suspect this is a function of the eval setting (context length cap), or do the authors happen to have any additional insights?

Thank you for this insightful question. We investigated whether FLAN-T5’s high RSS score might be influenced by context length limitations, particularly around the optimal retrieval depth of k = 25.

To assess this, we compared token lengths after truncation between k = 20 and k = 25. We found that in at least 32% of examples, the input at k = 25 includes more tokens post-truncation than at k = 20, confirming that additional retrieved content is being processed. Despite this added context, the performance difference between k = 20 and k = 25 is < 0.5 points, suggesting the model remains robust even as more content is introduced.

This supports our interpretation that FLAN-T5’s high RSS is not an artifact of truncation shielding the model from additional noise, but rather reflects genuine retrieval-depth stability. We will include this clarification and supporting analysis in the revised manuscript.

We thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer recognizes this paper provides a significant contribution by introducing structured metrics and is backed by extensive empirical analysis across multiple datasets and models.

1. RSS and its Relationship to Task-Specific Performance

Indeed, RSS is not intended to capture absolute task performance, and we were careful not to make such a claim. Instead, it measures the stability of performance near the optimal retrieval depth—capturing sensitivity to retrieval parameter changes, not performance magnitude. As the reviewer notes, these are distinct dimensions of system behavior.

2. Metric Comprehensiveness and Practical Relevance

While absolute performance is essential, stability and scalability are equally critical in real-world deployment:

• Stability (RSS) robustness to retrieval depth variation, which is particularly important in real-world scenarios where precise tuning is costly.

• Scalability (RSC) reflects how well a model benefits from deeper retrieval, which is especially important when relevant information is sparse or buried.

Thus, while task-specific performance is foundational, it does not provide a complete picture of real-world robustness or efficiency. We believe that stability, scalability, and absolute performance together form a more holistic and practical evaluation of RAG systems. We also discuss the importance of absolute task performance in sections 4, 6, 7, paying attention to compare with-context and no-context performance to see when RAG actually helps.

3. Can models with lower RSS but higher task-specific performance still be considered stable or robust?

Yes, a model can have high task-specific performance but low RSS. This would however mean it is brittle and harder to deploy reliably. This underscores the value of explicitly measuring stability and scalability alongside performance.

4. Theoretical analysis for generalizing findings beyond the empirical settings.

We appreciate the reviewer’s interest in generalizing the findings. While our focus is empirical, we hypothesize that a key factor driving behavior is the internal-external knowledge tradeoff: models with strong internal priors but weak integration of external input may degrade with increased retrieval.

Our empirical results support this. For example, LLAMA begins with high no-context performance, and, at low k, its with-context answers closely mirror its no-context outputs—indicating strong reliance on internal knowledge. However, as more documents are retrieved, LLAMA’s answers diverge from its initial predictions (as shown in the attached figure), yet performance worsens. This suggests that LLAMA is incorporating context—but in a way that disrupts rather than improves its predictions. In contrast, FLAN shows lower no-context overlap from the start, adapts more readily to retrieved information, and maintains more stable performance as k increases. We will incorporate this framing into the revised draft to better contextualize model behavior across tasks and domains.

5. A model like FLAN may benefit from retrieval more consistently than GPT-3.5, but are these differences purely due to the models’ architectures, or are there other factors like training data or task-specific tuning influencing the outcomes?

Regarding FLAN-T5 being more scalable than GPT-3.5, our observations suggest that differences could arise from both architecture and training objectives:

• FLANT5 explicitly has denoising training, where it learns to reconstruct original text from corrupted inputs, potentially helping it handle noisy or partially irrelevant retrieval contexts better. GPT-3.5, in contrast, does not explicitly train with a denoising objective, possibly explaining its comparatively limited scalability.
• FLANT5 (encoder-decoder) explicitly encodes the retrieved context separately from decoding (generation), potentially enabling it to manage larger context sets more effectively. GPT-3.5, being decoder-only, processes context in a strictly autoregressive manner, which may cause diminishing returns as the amount of context increases due to difficulty in attending equally well to all retrieved information.

6. Related Work: REALM and FiD

Thank you for raising this. We agree that REALM [Guu et al., 2020] and FiD [Izacard & Grave, 2021] are foundational RAG systems: REALM introduced end-to-end retriever-reader pretraining and FiD showed the benefits of fusing multiple retrieved passages via decoder-only models.

Our work is complementary: RAGGED offers tools to analyze reader-retriever behavior under varying retrieval depth and noise—dimensions not explicitly explored by these prior works. We will expand our Related Work section to reflect these connections.

7. Other Comments Or Suggestions Thank you for the suggestions about the figures and formatting. We will fix them in the revision.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We especially appreciate the recognition of how RAGGED addresses a gap in the literature by providing a unified framework for systematically evaluating scalability and stability and deriving actionable insights for real-world deployment.

1. Noise Assumptions in Robustness Evaluation

We appreciate the reviewer’s clarifying question about the robustness of our setup. To clarify: our “noise” is not artificially injected or random. Rather, it consists of non-gold passages retrieved by a real retriever. These are often topically relevant but incorrect, representing a realistic and common failure mode in deployed RAG systems. This setup reflects naturally occurring retrieval imperfections that practitioners routinely encounter.

We agree that adversarial or temporally inconsistent noise (e.g., contradictory or outdated passages) is an important direction for future robustness research. However, modeling those scenarios requires assumptions (e.g., degree of contradiction, factuality, intent) that are beyond the scope of this work. That said, RSS could readily be extended to those settings, and we hope future work on adversarial or knowledge-conflict RAG systems will incorporate it.

2. Applicability to Extreme-Scale Corpora

We agree that testing scalability on corpora with billions of documents would be highly valuable. However, such datasets with annotated gold passages are currently limited. RAGGED is designed to be scalable by construction and can be readily applied to larger corpora as they become available.

3. Empirical vs. Theoretical Foundations

We appreciate the reviewer’s point. Our focus is empirical by design, aiming to provide actionable metrics for analyzing RAG system behavior in real-world settings. This mirrors the trajectory of many standard metrics (e.g., BLEU, ROUGE, F1), which were adopted for their practical value before receiving formal analysis. That said, we agree that theoretical grounding—such as bounding the relationship between noise and RSS—would strengthen the framework, and we will note this as an important direction for future work.

4. How would RSC metrics change if evaluated with adaptive retrieval strategies (e.g., FLARE’s iterative retrieval based on uncertainty)? Could such strategies invalidate the trade-offs observed in your experiments?

This is an excellent and insightful point. Our current analysis focuses on the standard retrieve-then-generate paradigm with a fixed top-k retrieval depth, which remains a common baseline in RAG systems. However, the core idea behind RSC—measuring how performance scales with increased retrieval—can be naturally extended to adaptive retrieval strategies like FLARE. In fixed-depth retrieval, RSC reflects the trade-off between performance gains and increasing the retrieval cutoff. In adaptive systems like FLARE, a similar trade-off exists between performance and the number or frequency of retrieval calls as determined by an uncertainty threshold. One could analogously define an RSC-style metric that varies the retrieval triggering threshold and measures how performance scales with the total retrieval budget. This would preserve the core RSC insight: quantifying trade-offs between retrieval cost and performance gain. We will mention this extension as a valuable future direction.

5. Justification of Metric Thresholds

We chose ε = 0.5 for RSC based on empirical analysis: the standard deviation of F1 across all models and k values is consistently below 0.5 (mean ≈ 0.38, max = 0.46). This makes 0.5 a conservative but meaningful threshold that exceeds metric noise. Moreover, increasing ε to 0.6 or 0.7 preserves model ranking, indicating robustness to the threshold choice. For RSS, we use a window of δ = ±5, which aligns with practical tuning ranges and matches how performance curves behave in most models (either plateauing or gently peaking). We also tested δ = 10 and found rankings unchanged. Smaller windows (e.g., δ = 1) lead to RSS variances across models of < 0.002, making the metric insensitive and less useful.

6. Statistical Robustness and Variability

To address concerns about statistical rigor, we analyzed standard deviations across k values for each model and confirmed that they remain consistently low (all < 0.5). Additionally, we verified that model rankings remain stable under variations in ε and δ. These findings support the reliability and robustness of our conclusions, and we will include supporting variance analyses in the revised draft.