LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA

About the chunk size

First, the rigid chunk-divide strategy is directly adopted from previous works such as [1][2]. We just point out their drawbacks, including the coarse granularity and the possibility of producing incomplete evidence sentences.

In addition, we also try strategies that split the text based on sentence terminators as you mentioned, which are just LAC-S, RAC-S, post-LC-S, and post-RC-S introduced in Section 3.2. Their performance is shown in Table 4. The reasons why we still include chunk-level citations in the CoF pipeline are as follows:

1. From Table 4 we can see that both CoF and post-RC-C achieve better citation F1 than post-RC-S, indicating that using a course-to-fine strategy, i.e, first finding the relevant chunks and then pinpointing evidence sentences from these chunks, is beneficial for improving the performance. Table 9 also shows that SFT with CoF data significantly outperforms that with post-RC-S data.

2. Another advantage of our course-to-fine strategy is that it can extract continuous evidence sentences such as "[5-7]" from the relevant chunks, resulting in better semantic integrity and coherence, while sentence-level strategies such as post-RC-S often produce too many discontinuous citation numbers (such as "[3][7][15]..."), which makes subsequent learning difficult and is also less user-friendly.

3. As mentioned in line 323, before extracting the evidence sentences, we expand each relevant chunk by concatenating it with its preceding and succeeding chunks and only retain the complete sentences. This also guarantees that we can extract complete evidence from the relevant chunks.

[1] Enabling large language models to generate text with citations. Gao et al. EMNLP 2023.

[2] Attribute or Abstain: Large Language Models as Long Document Assistants. Buchmann et al. EMNLP 2024

About the novelty

In fact, the core parts of our evaluation method (i.e., evaluation for citation quality) and dataset construction (i.e., the citation generation steps in the CoF framework) are totally different from LongAlign, since LongAlign is for long-context alignment and we focus on long-context citation generation instead. We only adopt their evaluation method for correctness and their QA generation method (i.e., the first step in CoF).

We are the first work to study how to construct high-quality SFT data for LQAC task, and our data construction method is novel and carefully designed, significantly outperforming other regular citation generation pipelines. Our improved evaluation method for citation quality also aligns much better than previous methods (see human evaluation in Table 7) in long-context QA scenario. Besides, we propose a new metric correctness ratio, which helps us to find that adding citations via simple in-context learning can harm response correctness while SFT with citation information can improve correctness, which are not mentioned in previous work.

About the typo

Thanks for pointing out the typo in line 82. We will correct it in revision.

Thank you for your valuable reviews. Since you may miss some detailed information in our work, we would really appreciate it if you would like to reconsider the score of our paper. Please feel invited to leave more comments in case you have more detailed suggestions and questions.

Regarding the unique challenges we addressed

Thanks for your valuable review. In fact, our work addresses the following unique and important challenges in the long-context scenario:

1. Prolonged user waiting time. Most previous works on fine-grained citation generation rely on relatively complex pipelines that will significantly increase the inference time. For example, [1] breaks down the generation process into three steps: content selection, sentence planning, and sentence generation, and fine-tunes a model for each step. [2] requires the model to first generate all evidence snippets before producing the response. Compared with vanilla QA, these methods more than double the user waiting time, which is unacceptable in long-context scenario. In contrast, our work stays true to first principles and is the first to explore directly employing the long-context LLM to generate in-line citations on the fly (though straightforward, no one ever succeeded before), which basically does not increase the inference time and makes the whole product practicable in real-world usage.

2. Compromised response quality. Different from short-context open-domain QA, we first notice that in long-context scenario, one-pass methods (including retrieved-based and long-context-based methods) via in-context learning always hurt the response quality (Table 4). Therefore, we decide to adopt post-hoc strategy in the CoF pipeline, i.e., first obtaining the answer and then adding citations into the answer in subsequent steps, to fully utilize model's long-context capacities. The post-hoc characteristic also allows our pipeline to augment any existing long-context QA datasets with citations. Besides, we also find that SFT with citation information further improves the long-context-based response quality (Table 3), which is also not mentioned in previous works.

3. Increased learning difficulty. Another challenge of LQAC is the increased learning difficulty caused by the lengthy context and up to thousands of citation indices compared to the short-context scenario. The conventional RAG-based methods (e.g., post-RC-S) for data construction often recall nonadjacent sentences and thus produce too many discontinuous citation numbers (such as "[3][7][15]..."), which would confuse the LLM and make the training difficult. While the course-to-fine strategy in CoF allows extracting adjacent evidence sentences such as "[5-7]" from the relevant chunks, resulting in better semantic integrity and coherence of the cited information. Table 9 shows that using data constructed by our CoF pipeline can achieve significantly better citation quality than conventional RAG-based methods in long-context scenario.

4. Inappropriate evaluation method. As shown in Table 6 and 7, we find that previous automatic evaluation method (i.e., ALCE) for short-context citation generation has low alignment with human in long-context scenario because it does not exclude "functional sentences" in the model responses and overlooks partially supporting cases in the calculation of citation precision. In this work, we improve the evaluation method accordingly (Section 2.3.2) and achieve better agreement with human.

Thanks for your valuable feedback again. We would appreciate it if you could recognize our contributions and would like to increase the score of our paper.

[1] Attribute First, then Generate: Locally-attributable Grounded Text Generation. Slobodkin et al. ACL 2024.

[2] Learning Fine-Grained Grounded Citations for Attributed Large Language Models. Huang et al. Findings of ACL 2024.

W3: Insufficient benchmark

R: As we have mentioned in Section 2.4, we select LAC-S strategy as the default setting due to its efficiency, losslessness of context information, and no reliance on additional retrieval systems. It can purely examine the models' LQAC capacities and is also the most ideal strategy in practical usage. In addition, we also list the performance of different strategies in Table 4. By checking the generated responses, we can actually clearly find that: smaller models such Llama-3.1-8B-Instruct and GLM-4-9B-chat suffer more from poor instruction-following ability because they often generate responses that do not conform to the prescribed format, while larger models such as Llama-3.1-70B-Instruct and Mistral-Large-Instruct are influenced more by weak evidence-searching ability.

W4: Lack of baseline comparison

R: We evaluate the best few-shot method in [3], i.e., Attr. First CoT, on LongBench-Chat, using their open-sourced code and taking GLM-4 as the backbone LLM.

We find that when the context becomes longer, the content selection step of [3] often fails to locate the most relevant information, especially for some complex queries. This leads to bad response quality as well as low citation quality.

W5: Evalution of correctness

R: Thanks for your advice. Since our evaluation of correctness follows the tradition of long-context benchmarks such as [5][6][7], which typically score the model response with the golden answer, we will change the definition of correct into "Whether the response is accurate and consistent with the groundtruth".

W6: High cost of constructing SFT data

R: Considering that our goal is to achieve better performance than the most advanced proprietary model, the cost of data construction is acceptable, especially compared with human annotation. In addition, we have open-sourced our code, models, and LongCite-45k dataset for reproducing and further research.

W7: Quality of the question and answer generated by self-instruct

R: We directly employ the QA generation method proposed by [5], which has been proven high-quality in their paper. Our experiments also show the SFT models using the self-instructed data achieve comparable or even better long-context QA performance (i.e., $\text{C}_\text{LQA}$) than their officially post-trained counterparts. The good performance on multi-doc QA also indicates that the generated questions do not focus too much on local information.

W8: Control over granularity

R: In real-world usage, when the user clicks on the citation number, the corresponding text will be highlighted in the original context, without the need to find the evidence by the user himself/herself. The human evaluation in Table 6 and our online test have shown that our system has good practicality.

Q1 & Q2: Typos

R: Thanks for pointing out the typos. We will correct them in the revision.

Q3 & Q4: Evalution for Correctness

R: We just directly adopt the official evaluation methods and prompts for correctness from [5]. Since LongBench-Chat is a harder and smaller benchmark with long-form groundtruth, the authors of [5] set a score range of 1-10 for better discriminability and annotated 3 scoring examples for each test instance as the pivots for better alignment with humans. While the tasks in LongBench are relatively easier(the context is shorter and the groundtruths of QA tasks are short phrases), so zero-shot approach and smaller score ranges are enough. In Table 3, the correctness scores are unified into a percentage scale, and the average score is the average of the five datasets.

Q5: Performance of GLM-4-9B on GovReport

R: Thanks for pointing out the exception. By checking the generated response, we find that for 31 out of 200 test instances of GovReport, LongCite-9B directly outputs the summary without generating citations. A feasible fix method is to force the model to first generate the special token "<statement>" so that it will generate citations for each instance. This leads to 80.7 citation F1 on GovReport.

[3] Attribute First, then Generate: Locally-attributable Grounded Text Generation. Slobodkin et al. ACL 2024.

[5] Longalign: A recipe for long context alignment of large language models. Bai et al. Findings of EMNLP 2024.

[6] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. Bai et al. ACL 2024.

[7] ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens. Zhang et al. ACL 2024.

W1: Lack of significant novelty & missing related works

R: Thanks for your valuable feedback. We will add these related works in the revision. Nevertheless, we clarify the novelty and difference of our work as follows:

1. Our work is the first to present a comprehensive body for long-context QA with fine-grained citation, including a new evaluation benchmark, a novel data construction pipeline, and thorough model training experiments to prove the effectiveness. In contrast, [1] is a benchmark rather than a methodology for LQAC. Besides, it focuses on coarse-grained citation and only considers citation recall for free-form QA, overlooking the citation precision. In addition, its measurement method for citation recall, which is adopted from [2], has been shown to have relatively low agreement with human because it overlooks the "functional sentences" in model responses. On the other hand, we improve the automatic evaluation method for citation quality and achieve significantly better alignment with human than [2] in long-context scenario.

2. Moreover, we address several unique and important challenges of LQAC, making the whole product practicable in real-world usage for the first time:

   (1) Prolonged user waiting time. Though [3] and [4] discuss fine-grained citation, their approaches more than double the inference time compared to vanilla QA without citation, which is unacceptable in long-context scenario. [3] breaks down the citation generation into 3 sub-tasks, and [4] requires the model to first output the evidence before generating the response. In contrast, we are the first to explore directly employing long-context LLMs to generate in-line citations on the fly by carefully designing the CoF pipeline for data construction, making the inference time basically the same with vanilla long-context QA.

   (2) Compromised response quality. We are the first to notice that one-pass citation generation via in-context learning always hurts the response quality in long-context scenario. Moreover, the content-selection of [3] and the pre-retrieval of [4] will discard most parts of the context, which would further affect the correctness of responses. Therefore, we decide to adopt a post-hoc strategy in the CoF pipeline to fully utilize the model's long-context capacities and guarantee the response quality. The post-hoc characteristic also allows our pipeline to augment any existing long-context QA datasets with citations.

   (3) Increased learning difficulty. Another challenge of LQAC is the increased learning difficulty caused by the lengthy context and up to thousands of citation indices compared to the short-context scenario. The conventional RAG-based methods (e.g., post-RC-S and the method of [4]) for data construction often recall nonadjacent sentences and thus produce too many discontinuous citation numbers (such as "[3][7][15]..."), which would confuse the LLM and make the training difficult. While our proposed course-to-fine strategy in CoF allows for the extraction of adjacent evidence sentences such as "[5-7]" from the relevant chunks, resulting in better semantic integrity and coherence of the cited information. Table 9 shows that using data constructed by our CoF pipeline can achieve significantly better citation quality than conventional RAG-based methods in long-context scenario.

W2: Unoriginal conclusions

R: In fact, we present the following new conclusions that are not mentioned in previous works:

1. Through LongBench-Cite, we reveal current LLMs’ limited performance on LQAC.

2. We are the first to notice that one-pass citation generation via in-context learning will hurt the response quality, while SFT with high-quality citation information can further improve the response quality.

3. We notice that conventional RAG-based methods (e.g., post-RC-S) for data construction lead to too many nonadjacent citation numbers, which will increase the training difficulty in long-context scenerio, and we adopt a course-to-fine strategy in CoF pipeline to address this challenge.

4. The similar citation length between the trained models and constructed data demonstrates that not only the evidence-locating skill but also the citation granularity can be well learned through SFT.

5. LongCite-8B and LongCite-9B even attain higher citation F1 than the data construction pipeline CoF (72.0 and 69.2 v.s. 65.8), implying a potential for iterative self-improvement.

[1] Attribute or Abstain: Large Language Models as Long Document Assistants. Buchmann et al. EMNLP 2024.

[2] Enabling large language models to generate text with citations. Gao et al. EMNLP 2023.

[3] Attribute First, then Generate: Locally-attributable Grounded Text Generation. Slobodkin et al. ACL 2024.

[4] Learning Fine-Grained Grounded Citations for Attributed Large Language Models. Huang et al. Findings of ACL 2024.

Thanks for your valuable feedback.

W1: missing details

The retrieval method is simple dense retrieval, using Zhipu embedding-2 as the embedding model. Specifically, for each sentence in the model response, we retrieve the top-$l_\text{max}$ chunks whose embeddings are most similar to the sentence's embedding. The hyperparameter $l_\text{max}$ and $k$ ($n_\text{sent}$ is the number of sentences in the answer rather than a hyperparameter) are chosen based on several samples from the training set so that the retrieval recall of evidence chunks surpasses 90%.

In pipeline validation, the evaluations are carried out automatically, using the evaluation method described in Section 2.3.

W2: Over-reliance on LLM evaluation

Thanks for your advice. As mentioned in the paper, we rely on LLM evaluation because the conventional evaluation methods are no longer suitable for the LQAC scenario and have low agreement with human. In addition, a possible method to defend the hack you mentioned is to ignore these "functional sentences" when calculating the citation recall.

W3 & Q7: Regarding the prompt format

In fact, we have tuned the prompt so that the models can achieve better performance. There are three key points for the prompt tuning:

1. As shown in Figure 10, we repeat the instruction at the end of the prompt, after the context and question presented. This reminder of instruction is helpful for the models to follow the instruction better, especially when the context is too long.

2. We require the models to generate citations in the format of spans such as "[3-5][8-9]" instead of conventional separated indexes such as [3][4], since we found that the latter often leads to non-stopping enumeration (e.g., [3][4][5][6][7]...).

3. The demonstration contains citations from each part of the context so that the models would not focus too much on local information.

Q1: Construction of LongBench-Cite

Yes, LongBench-Cite is a concatenation of existing datasets from LongBench-Chat and LongBench. We combine them for unified and convenient evaluation and benchmarking. A difference is that we require the models to generate long-form responses for all instances, while the original LongBench asks the models to only output the answer phrase for the QA tasks, which is not suitable for citation generation.

Q2: Regarding the Cohen's Kappa

As shown in Table 7, the citation recall/precision at the sentence/citation level measured by GPT-4o and human are compared in the Cohen's Kappa calculation.

Q3: Using CoF to do the LQAC task

Yes, CoF can be directly used for the LQAC task. However, the main disadvantage is that the complex pipeline will significantly prolong user waiting time, while the fine-tuned models can generate the in-line citations on the fly, which basically does not increase the inference time compared to vanilla long-context QA.

Q4: Regarding the GPT-4o evaluation for itself

In the evaluation for correctness, there is indeed a risk that GPT-4o will assign higher scores for itself, as found by previous works. However, since we focus more on the correct ratio, which is a relative ratio, the influence will be alleviated. In addition, because the evaluation of citation recall and precision is similar to sentence-level NLI tasks, GPT-4o would not present obvious bias.

Q6: Regarding the "lost-in-the-middle" phenomenon

Yes, we notice some "lost-in-the-middle" phenomenon for vanilla LongSFT models. Though [1] has shown that using self-instruct QA data can achieve all-green performance on the simple "needle-in-a-haytask" test, we found that when faced with a query that requires a global view, LongSFT models tend to use more information from the head part of the context and only roughly utilize or even ignore the rest of the context (Section 4.2.1). In contrast, the generated citation numbers allow LongCite models to be aware that current response content has covered which parts of the context, so that they can utilize different parts of context more uniformly, resulting in a more comprehensive response.

Q8: Citing non-contiguous sentences

Yes, our models support such citations such as [1-2][5-5], as shown in Table 10.

[1] Longalign: A recipe for long context alignment of large language models. Bai et al. Findings of EMNLP 2024.