Single-Pass Document Scanning for Question Answering

We appreciate your positive assessment of our work, particularly your recognition that it addresses a critical challenge in NLP and your acknowledgment of the clarity in our methodology and experimental details.

Marginal Improvement: The performance gains over vector-based retrievers, while consistent, are relatively modest. A stronger case for the practical impact of these improvements would strengthen the paper.

SPScanner is designed for scenarios where a document is introduced for the first time and then queried, demonstrating significant practical advantages over vector-based retrievers in both computational efficiency and accuracy.

Our 130M model achieves higher overall accuracy than all tested embedding baselines while using only a fraction of the computational resources - 19.0 TFLOPs compared to 1287.5 TFLOPs for the most accurate baseline (NV-Embed-v2-7B). Furthermore, our 1.3B variant closes 50% of the performance gap between NV-Embed-v2-7B and GPT-4o Full Context while still using significantly fewer computational resources (197.9 TFLOPs).

These improvements are particularly valuable in resource-constrained environments and applications requiring rapid document processing, where the combination of higher accuracy and reduced computational overhead provides clear practical advantages over existing methods.

Missing Retrieval Evaluation: The paper lacks direct evaluation of retrieval performance against other retrievers, which would provide a more comprehensive comparison.

Since the dataset includes sentence-level relevance annotations, why not evaluate retrieval performance directly?

We do not have access to ground-truth annotations of relevant sentences in the 41 long document datasets we used. However, we can use a proxy measure: having GPT-4o annotate relevant sentences while knowing the actual answer to the question, and evaluating both our model and the baselines against this proxy using standard information retrieval metrics such as recall, precision and nDCG. We will report these retrieval evaluation results by the end of the response period.

The use of binary relevance labels may oversimplify sentence relevance. Could a graded or probabilistic labeling scheme improve performance?

We acknowledge that explicitly graded training labels could potentially capture additional nuances and represent an interesting direction for future work.

Why was top-k sentence selection chosen over top-p sampling? Was this decision empirically validated?

We chose top-k sentence selection because the approach aligns with established RAG setups (Xu et al., 2023; Li et al., 2024). We will perform an additional experiment using top-p sampling for comparison and provide the results before the end of the discussion period.

Why not compare with LLMs that support very long contexts (e.g., millions of tokens) for generating synthetic data, besides chunk and pair-based approaches?

While comparing with long-context LLMs would provide valuable insights, this approach would substantially increase per-document processing costs. Given our budget constraints, we chose the current methodology, which enabled us to generate all synthetic data for $1,314 - a feasible cost for us.

Is the search for connections between chunks performed recursively, or is it limited to nearby context?

The link search is single-pass, not recursive. For each randomly selected 20-sentence chunk, we provide the LLM with a 200-sentence context window (several thousand tokens), in which the initial chunk is located either at the very beginning or the very end of this window. Within that context span, the model identifies meaningful dependencies., and we select the most distant one to formulate a question and label the relevant sentences. Importantly, no newly identified sentences are fed back into the search process.

We apologize for the inaccuracy in Line 134 of the manuscript. It will be corrected with the above clarification.

Thank you for your detailed review and for your assessment that link-based synthetic data generation is interesting, that SPScanner demonstrates performance advantages over more expensive baselines, and that our analysis and ablations are thorough.

Lacks discussion on why NV-embed outperforms this method on conversational datasets. Is the data generation strategy followed here not suitable to conversational tasks?

We thank the reviewer for this important observation. NV-embed’s superior performance on conversational datasets can be attributed to the specific nature of these tasks, which often do not require capturing long-range dependencies from distant parts of the document. Since embedding models can retrieve local context but falter at long-range dependencies, these conversational datasets avoid the weakness of embedding models.

Of the 707 data points in the conversational datasets, 400 focus specifically on speaker identification tasks, divided into two categories: 200 questions about masked name entities and 200 requiring identification of speakers for given sentences. Both tasks only require knowing the immediate context to answer the question. We provide one example for each task below.

Masked Name Entities Task:

Question: “Which character is \$\\$MASK\$\\$?”

Context: PATRICK: I do not care for your language. LEONARD: I'm only trying to help you guys. Leonard stops exercising. \$\\$MASK\$\\$: (CONT'D) You're fat dad. Mom is fat. Us kids are fat. (CONTINUED). CONTINUED: PATRICK: What's your point?

Answer: LEONARD

Speaker Identification Task:

Question: “‘Who would like to come in on that? Go on, Catherine.’ said by ___”

Context: "And Catherine, I just wonder, extending Alun's point, ..." said by Huw Irranca-Davies AM "On the point about being boring, generally boring is good as far as the EU is concerned ..." said by Alun Davies AM "Who would like to come in on that? Go on, Catherine." said by ___

Answer: Huw Irranca-Davies AM

We will include this analysis in the revised manuscript.

We appreciate your comprehensive evaluation of our work and your assessment that it has clear motivation, technical contribution, and novelty of link-based synthetic data.

Limited architectural novelty – core model is a straightforward repurposing of Mamba‑2 with a binary head; advances over prior SSM retrievers (Hydra, Monarch Mixer) are incremental.

Our primary novelty lies in determining how to effectively train an SSM model as a single-pass, discriminative scanner for identifying relevant information in long documents.

Straightforward adaptations of existing approaches fail to achieve similar performance. Training Mamba for generative retrieval achieves only 33.5% accuracy (Appendix E.2), while direct question answering achieves only 27.6% (Appendix E.3). Additionally, straightforward synthetic data approaches like chunk-based (using individual document segments) and pair-based (matching similar chunks) produce questions that don't require understanding document-wide connections and lead to worse performance (Table 4).

The effectiveness of our approach is demonstrated by: (1) the model outperforms embedding models 8-60x larger while using fewer computational resources (Table 1 and 2), (2) fine-tuning these embedding models on our synthetic data shows no improvement (Line 270-273), and (3) we observe strong performance on documents up to length 256k, which is more than 20x longer than any documents observed during training (Line 264-269).

What specific characteristics of the link‑based questions (e.g., span distance, discursive cues, entity co‑reference) do you believe most strongly encourage the model to learn long‑range dependencies?

We believe that the semantic relationship between linked chunks is a primary driver for learning long-range dependencies. If the span distance between linked chunks is long (e.g. 200 sentences apart rather than 20 sentences apart), it is possible such a natural long-range relationship (e.g., recurring mention of characters in a novel) would better encourage the model to learn long-range dependencies.

Thank you for your thorough review and for recognizing both the performance advantages of SPScanner over RAG methods and the quality of our link-based synthetic data generation approach.

The novelty of this paper is limited, since such sentence extraction method has been widely utilized in open domain question answering[1][2]. Besides there is no discussion about these methods in related work...

We appreciate the pointer to RECOMP and FastFid and will add a paragraph in the Related Work section explicitly comparing our approach to these sentence-extraction methods. We would like to evaluate these models on our tasks. However, these methods rely on in-domain fine-tuning for each specific dataset, whereas SPScanner uses the same checkpoint for all 41 datasets. We also do not use any of the train sets of these 41 datasets for training SPScanner. Since only RECOMP provided a checkpoint (which has been finetuned for HotpotQA using its train set), this checkpoint cannot be fairly evaluated on these 41 datasets.

Our contributions are distinct in two key ways. First, RECOMP and FastFiD employ transformer-based approaches that scale quadratically with input sequence length. In contrast, our SPScanner leverages a linear-time state space model, enabling efficient processing of extremely long documents that would be much more expensive for these existing methods.

Secondly, we introduce a novel link-based synthetic supervision signal that trains the model to identify and utilize sparse, interconnected information across the given long document. The idea to construct synthetic data in this way is fundamentally different from using the in-domain train sets to fine-tune models for specific test sets.

In section 5.5, you finetune the embedding models with 1 million link-based synthetic data. But during inference, the documents are processed in chunks of fixed-length 300 words, which may cause unfinished sentences in the chunk. Will this gap hinders the performace of finetuned embedding models? (The FT models only show little improvement in Table 1)

In Appendix A.4, we compare different setups (e.g. retrieve 50 sentences or retrieve 5 chunks of 300 words) for embedding models. Embedding models’ setup to process documents in chunks of 300 words is chosen because this setup often elicits the best performance from embedding models. For the fine-tuned embedding model, our experiment shows the 5-chunk setup is comparable to the 50 sentences setup:

Before the end of the discussion period, we will perform this additional experiment: We will transform the linked-based data into 300-word chunks and fine-tune the embedding model on the transformed training data. We will then evaluate the fine-tuned model with the 5 300-word chunks setup.

Is there any ablation study about the number of selected sentences? Since there is still some gap between SPScaner and GPT-4o Full Context, will more selected sentences reduce this gap?

The 10-sentence retrieval results are provided in Appendix A.4 and A.5. Results show that retrieving 50 sentences consistently outperforms retrieving 10 sentences:

Dear Reviewer Dc8t,

Thank you again for your thorough review of our paper. Since the discussion period is coming to an end soon, we would greatly appreciate it if you could indicate whether our rebuttal addresses the questions raised in your initial review.

Thank you for your time,

SPScanner Authors