[W1, Q2] Retrieving and reranking a sizable set of documents, with potential query reformulation, may introduce non-trivial latency. Retrieving and reranking a sizable set of documents and performing query reformulation adds computational cost. Have you analyzed the detailed inference time overhead?

Recent LLM-based reranking methods [1,2,3] typically require feeding multiple documents into the LLM simultaneously to assess their relative importance. This involves generating token-level outputs over all N documents, making the comparisons relative and thus difficult to parallelize. Moreover, such approaches are sensitive to the context length limitations of LLMs. As a result, their time complexity is approximately O($N^2⋅L^2+NL^2$), where L is the average document length, and if we assume that the length of the generated token is also L.

In contrast, CalibRAG independently processes each query-document pair ($q$, $d_i$) to predict a confidence score. Here, the theoretical time complexity for the CalibRAG is O($N⋅L^2$), which is smaller than the recent LLM-based reranking methods. Furthermore, the confidence score is computed via a lightweight MLP without requiring autoregressive decoding, making it substantially faster and more efficient.

To support this, we provide a direct comparison of inference time between CalibRAG and LLM reranking using the same document set in Table R.1, with interpretation to be included in the final version. Meanwhile, query reformulation is an optional component. For the sake of fair and consistent comparisons, we did not apply it in most experiments. It is only used once in Figure 5-(c) to explicitly assess its effect. In practical deployment, users may choose to apply reformulation depending on their desired trade-off between performance and computational cost.

In summary, CalibRAG is designed for efficient document reranking and confidence estimation, and demonstrates clear advantages in terms of inference efficiency.

Table R.1 Inference time comparison between CalibRAG and LLM-based reranking. We used Llama-3.1-8B-Instruct and measured the average inference time over 20 randomly selected queries from the NQ dataset. CalibRAG evaluates each (query, document) pair independently and in parallel, while LLM-rerank jointly processes all documents via ~500-token generation. Measurements were taken on a single A6000 48GB GPU.

[1] Zhuang, Shengyao, et al. "A setwise approach for effective and highly efficient zero-shot ranking with large language models." Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024.

[2] Qin, Zhen, et al. "Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting." Findings of the Association for Computational Linguistics: NAACL 2024. 2024.

[3] Kim, Jaehyung, et al. "SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs." The Twelfth International Conference on Learning Representations.

[W2] The forecasting function is trained on synthetic data derived from a surrogate language model, leaving open questions about how well it mirrors real human decision behavior.

Thank you for the thoughtful comment. While it is true that the forecasting function is trained on synthetic data generated from a surrogate language model, we would like to highlight several key points that support the practicality and reliability of our approach.

First, we conducted a human evaluation to assess how well the model’s predictions align with actual human decisions. Specifically, we randomly sampled 2 questions per confidence bin across 5 bins from WebQA, NQ, and BioASQ (totaling 10 examples), and presented the question and retrieved context to 10 human annotators. They were asked whether they agreed with the user model’s judgment—that is, whether the retrieved context was helpful for answering the question. The results showed an agreement rate of 81.3%, indicating that the surrogate-based forecasts meaningfully reflect human judgment.

Second, conducting full-scale user studies where humans serve as decision-makers in downstream tasks is logistically challenging and costly. As an alternative, we simulate diverse user preferences by evaluating CalibRAG under multiple distinct surrogate user models (e.g., Phi-4, DeepSeek-Distill), which reflect different decision criteria and levels of background knowledge. As shown in Appendix D.4, CalibRAG consistently outperforms baselines across these varied simulated user types, demonstrating robustness to user variability.

Third, although our current setup relies on LLM-based simulations, it can be naturally extended to incorporate real user data. For example, future work could fine-tune the forecasting model using expert annotations or crowd-sourced feedback to enhance realism and generalization.

[Q1] Do you foresee applying CalibRAG to other generative tasks such as summarization or code generation?

Thank you for the question. We believe CalibRAG can be extended to other generative tasks such as summarization or code generation, as long as a meaningful retriever can be constructed. For summarization, this applies in settings where specific summary formats are desired and can be retrieved. For code generation, the applicability depends more on retrieving relevant algorithmic patterns, which is feasible but more domain-specific. Importantly, although our model was trained on general-domain data, it already shows strong generalization to out-of-domain tasks (e.g., medical QA), suggesting broader applicability of our framework.

Thanks for your feedback! The rebuttal has fully resolved my questions and I decide to keep my score positive.

[W1] Is the idea truly novel, given its simplicity, and is the dataset contribution substantial enough to strengthen the paper's overall impact?

Thank you for the kind feedback. We also found it surprising that prior RAG research has largely overlooked decision-aware confidence calibration, especially given the importance of reliability in downstream applications. We believe this is due to the lack of clear formulation and evaluation settings for decision-centric RAG scenarios, which our work aims to establish. We appreciate your recognition that our dataset, while a smaller contribution, serves as a concrete step toward enabling standardized evaluation in this space.

[W2] The people in the figure are pretty, but maybe keeping mostly their faces would make the differences between them more salient? In the contributions paragraph, references to where each part is written could be useful for people to jump to the right place (e.g., the dataset is not reflected in the section structure).

Thank you for the helpful suggestions. We will revise the figure to focus more clearly on the distinctive facial features, which better convey diversity among user types. Additionally, we will edit the contributions paragraph to include references to the relevant sections for each component, including the dataset, to improve navigability and clarity in the final version.

Great, note that while I gave suggestions, I will not be offended or anything if you choose some of them to be not helpful. (Of course I believe they are :-) )

My score partially reflected my low confidence and the (hard to calibrate) impact, so I am leaning towards acceptance still, as far as I am concerned.

[W1, Q1] Can CalibRAG demonstrate effectiveness on decision-making tasks beyond QA or in real-user scenarios?

We agree that QA tasks alone may not sufficiently capture the broader notion of decision-making, as direct answering can reduce the role of a user model. To address this, we designed our setup to simulate decision-making: instead of answering the original question $x$, CalibRAG generates an open-ended query $q$ to retrieve information relevant to user decisions (L195–L196). This aligns with recent work on LLM-based uncertainty calibration and decision support [1, 2].

We acknowledge that surrogate user models cannot fully replicate real human behavior. To mitigate this, we employed diverse LLMs including LLaMA 2 7B (Table 1-(b)), Phi 4, and DeepSeek Distill (Appendix D.4) to reflect varying user backgrounds and preferences.

Additionally, we conducted a human evaluation to assess how closely CalibRAG's judgments align with human decisions. Specifically, we sampled 2 questions per confidence bin across 5 bins from WebQA, NQ, and BioASQ (10 examples total), and asked 10 human raters whether they agreed with the user model’s judgment (i.e., whether the retrieved document was helpful). The result was an average agreement rate of 81.33%, suggesting strong alignment between surrogate model outputs and human intuition.

Together, these quantitative and qualitative results demonstrate that CalibRAG goes beyond standard QA and effectively supports realistic decision-making scenarios.

[1] Band, Neil et al. "Linguistic Calibration of Long-Form Generations." ICML, 2024.

[2] Kapoor, Sanyam et al. "Large Language Models Must Be Taught to Know What They Don’t Know." NeurIPS, 2024.

[W2, Q2] Recent RAG models can handle multiple documents at once. Why did you use only the top-1 document for generation? Wouldn’t using more documents reduce the need for reranking? Do you have any experiments evaluating this?

Thank you for the insightful question. In response to your suggestion, we conducted additional experiments to compare the performance of CalibRAG and the baseline (Number) in a multi-document RAG setting.

Specifically, we used the BM25 on the WebQA, with Llama-3.1-8B-Instruct as the user model. In this setup, we provided the LLM with the top-k documents selected either by the retriever (baseline) or by CalibRAG’s confidence-based reranking, and generated final answers based on the combined input. As shown in Table R.1, even when using k=3 documents, CalibRAG consistently outperforms the baseline in terms of ACC as well as calibration metrics.

Due to resource constraints, we limited our evaluation to k ≤ 3. Within this range, we observed that increasing k improves ACC for both methods. However, prior work [1] has shown that in multi-document RAG settings, the order of documents significantly affects performance, and [2] further reports that providing more than three documents may introduce contradictory information, which leads to increased noise and degraded performance. These findings suggest that simply adding more documents does not necessarily lead to better outcomes. Instead, effective document selection and reranking remain essential.

Table R.1 Comparison of performance in multi-document generation settings.

[1] Liu, Nelson F., et al. "Lost in the Middle: How Language Models Use Long Contexts." TACL., 2024.

[2] Kim, Jaehyung, et al. "SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs." ICLR., 2024.

[Q3] L263: For baselines, guidance, and confidence are generated based on the top-1 document retrieved. Is it fairer to generate the confidence scores for all documents and utilize the most confident one?

As noted in L299–L301, we would like to emphasize that we have already conducted this experiment. As described in Appendix D.7, we explored an extended baseline where decoder outputs $z$ are generated for all retrieved documents. A separate confidence model is then used to predict confidence scores for each document, and the one with the highest score is selected. While this approach is conceptually valid, it suffers from significant computational overhead due to the need to generate responses for all documents. Despite this additional cost, the calibration performance of this baseline remained inferior to that of CalibRAG.

Table 9. Evaluation metrics of Number + Rerank and CalibRAG on WebQA

For completeness, we summarize the results from Appendix D.7-Table 9 again here.

[Q4] Could you use another evaluator in the final judgment to show the robustness of CalibRAG if possible?

Thank you for the suggestion. Table R.2 reports results on the NQ using BM25, and the evaluator used for decision scoring is LLaMA-3.2-3B-Instruct. We acknowledge the value of evaluating robustness across different evaluators and plan to include results on additional tasks with alternative models in the final version. Nonetheless, the clear and consistent improvements observed in all metrics indicate that CalibRAG exhibits robust calibration performance under the current setting.

Table R.2 Evaluation on NQ dataset using BM25 and LLaMA-3.2-3B-Instruct as the evaluator.

[Q5] What is the reason for not taking guidance as one of the input params of the forecasting model?

As you noted, our goal is to predict how the generated guidance $z$ affects user decisions. While explicitly generating $z$ for every $(q, d)$ pair is computationally expensive, our model addresses this by using the internal representations of the LLM that generates $z$ as features (L160–161, L168), allowing it to capture $z$'s influence implicitly. This design preserves efficiency, and its effectiveness is validated by strong empirical performance.

[Q6] Why do you use Llama-2-7B in Table 1b, not aligned with the main setup?

We used LLaMA-2-7B in Table-1-(b) for two main reasons. First, the Self-RAG model is only publicly available with LLaMA-2-7B and 13B, so we used the same base model to ensure a fair and reproducible comparison. Second, we aimed to show that CalibRAG performs robustly across different backbone architectures. The results in Table-1-(b) confirm that CalibRAG remains effective even when used with a different LLM.

[Q7] L342: "gains diminish beyond 40 documents". However, it seems that using 40 documents yields larger gains compared to 20. Can you clarify this statement?

Thank you for the observation. We have conducted additional experiments on both WebQA and NQ using BM25 and Contriever as retrievers. Across the majority of these settings, we observed that performance gains tend to saturate beyond the top-20 retrieved documents. The statement in L342 was based on this general trend. However, Figure 5-(b) happens to show one case (WebQA with Contriever) where the performance improvement from 20 to 40 documents appears more pronounced, which may have caused the perceived inconsistency. We will revise the text and figure to clarify that gains generally diminish after 20 documents across most settings and support this with a more complete set of results in the final version.

Table R.3 Performance of CalibRAG across different numbers of retrieved documents.

[Q8] What model is used for the query reformulation? What are the maximum reformulation times when the threshold cannot be satisfied?

Thank you for the question. The model used for query reformulation is the same LLM used to generate the guidance $z$. We perform query reformulation only once in the experiment shown in Table 2-(c), where we explicitly evaluate its effect. In principle, query reformulation can be applied repeatedly based on user configuration or an adaptive stopping criterion. However, in our main experiments, we did not apply reformulation at all in order to ensure fair and consistent comparison across all methods. We will clarify this implementation detail in the final version.

Additionally, thank you for the helpful suggestions. We will incorporate Zhao et al. (2024) into the related work and correct all noted typos (L224, L250, Figure 5c) in the final version.

Dear Reviewer gigS,

Thank you for your thoughtful feedback and for raising your score.

We appreciate your suggestions and will reflect them in the revision.

Sincerely,

The Authors

[W1, W2] Can the paper provide a clearer positioning within existing RAG literature and discuss limitations and future directions more explicitly?

Thank you for the thoughtful suggestions. While we have included related RAG works in Appendix A.2, we recognize that their placement may reduce accessibility. We will revise the introduction to more clearly discuss RAG-related literature and better contextualize our contribution. Additionally, we will update the conclusion to include a concise discussion of limitations and potential future directions to support ongoing research in this area.

[Q1] I wonder what the stop criterion is for the inference iterations? Or if step 3 will only be executed once, then what if there is still no quality retrieved document?

Thank you for the insightful question. To ensure a fair and consistent comparison across all methods, we did not perform Step 3 in any setting except for Figure 5-(c). In that experiment, we applied query reformulation only once, explicitly to evaluate its potential impact. We also note that query reformulation is an optional module in our pipeline. In real-world deployments, users can decide whether to enable it based on the trade-off between computational cost and performance gain. That said, we agree that if the reformulated query still results in low-quality retrieved documents, a single iteration may be insufficient. This limitation stems from both the inherent recall limitations of the retriever and the limited representational coverage of the document pool. To address this, incorporating adaptive multi-step retrieval mechanisms, possibly coupled with uncertainty-aware stopping criteria, is a promising direction for future work. We will clarify this detail in the final version of the paper.

[Q2] Any intuitions on using Fourier positional encoding for the temperature parameter? Are there other potential choices?

Thank you for the question. Using a scalar $t$ directly often limits representational power and gradient flow, making it difficult for the model to condition on user behavior effectively. To address this, we adopt Fourier positional encoding, which maps $t$ into a high-dimensional space using sinusoidal functions. This approach is widely used in neural fields (e.g., NeRF [1]) to inject continuous inputs into deep models and enables smoother generalization across varying $t$. We found it effective and lightweight, though other encodings (e.g., learned embeddings) are also possible.

[1] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis., Ben Mildenhall, et al., ECCV 2020.