Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models

Firstly, thank you for your comments. We hope to clear up any misunderstandings so the contribution of our paper is more clear.

Now we would like to answer your questions.

In Table 2, it seems that late chunking aims to better segment chunks, yet the use of sentence boundaries and fixed-size boundaries indicates that both late chunking and naive chunking methods are dividing chunks in the same way. Then why does late chunking still can generate higher-quality embeddings and achieve better performance?

There might be a substantial misunderstanding of the late chunking idea we proposed in the paper. Late chunking itself does not modify the way texts are segmented. The modification that we propose is to apply the chunking after the whole text is encoded with the language model, i.e., we first obtain a sequence of token embeddings and then apply the chunking after that just during the mean pooling step that combines the token embeddings into a text embedding (for each chunk). So the use of late chunking is independent from the technique that is used to determine the segments and we primarily evaluate late chunking against chunking without our late chunking when using the same boundaries for the segments (with call the normal embedding method "naive chunking"). This process is described extensively throughout the paper, and a high level overview can be found in the introduction at the bottom of page 1. Our hypothesis for the improved performance of late chunking is that it can retain contextual interactions between chunks within the embeddings themselves, due to the embedding process happening first, on the unchunked text.

Do the authors believe that late chunking could yield better results with LLM retrievers employing causal attention mechanisms?

Most embedding models rely on bi-directional attention. When training LLMs for a text embedding task, the attention mechanism is often changed from a causal to a bidirectional one. For example the two best LLM-based embedding models on the MTEB (Massive Text Embedding Benchmark): NV-Embed-v2 [1] and gte-Qwen2-7B-instruct (https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct) make such a modification. Accordingly, we don't think that late chunking is particularly useful for causual attention. We don't have an educated guess whether late chunking would perform better with causual attention. However, this is beyond the scope of this paper, which is primarily focused on improving the representations of chunked text primarily for existing embedding models. This is an interesting direction for future research though, so thank you for raising this point.

[1] Lee, Chankyu, et al. "NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models." arXiv preprint arXiv:2405.17428 (2024).

Thank you for your review. We would like to address the two points you mentioned in the "Weaknesses" section of your review:

For naive chunking, the standard practice is to have some overlapping strides between chunks, and to include meta information such as document title in every chunks when available. It is unclear whether the author of this paper follows this practice in implementing the baselines.

• We included the title (when available, as in NFCorpus and TRECCOVID) in the text, as this is the standard practice for evaluating these tasks. No additional meta-information is provided for the evaluation sets. Indeed, we did not evaluate chunking strategies that use overlapping chunks. Since there are numerous methods practitioners use for chunking, we could not cover all of them in our experiments. However, thank you for your suggestion, we have added an evaluation using overlapping chunks in Appendix A.2 of our revised submission. Since overlapping chunks is related to improving contextual dependencies between chunks, it is important to compare against. Overall, the results demonstrate no significant advantage of overlapping chunks for the BeIR benchmark tasks we evaluated. However, overlapping chunks did not reduce retrieval performance for naive and late chunking either.

The paper uses a relative small chunk size (up-to 512) in the experiments when the embeddings studied support 8k context length. As shown in the ablation, the gains from late chunking diminish when the chunk size goes from 16 up to 512. It is unclear whether it is still effective when the chunk size approaches the embedding length limit of 8k, where the benefit of chunking is most useful.

• We completely disagree with the claim that chunking is mainly useful to handle cases where the texts reaches the maximum token length of the model, however, we acknowledge that we did not make this clear enough in our original submission. Therefore, we added some citations to previous works that have demonstrated that languange models in general [1] as well as embedding models in particular [2] cannot handle long text as well as short texts and using small chunk sizes is therefore more useful. In addition, we conducted another experiment (which is added to the updated submission in Appendix A.1) to demonstrate the limitations of long text embedding models and the advantage of (naive) chunking for retrieval applications. In particular, we show that even when only processing text which truncated (cut off) at the token limit of the model chunking performs on average ~24% better across all non-synthetic retrieval tasks in the LongEmbed benchmark [3].

Furthermore, chunking has application outside of retrieval, such as in text classification for sentiment analysis [4], where chunking is necessary, independent to the length of the input document. This reference as well as a brief description has also been added to the revised paper.

Given your low score of the paper, is there any other feedback you have aside from the two previously mentioned weaknesses that we could use to strengthen our work? We believe to have addressed your criticisms, and are therefore looking for other ways in which our research is deserving of the low score.

[1] "LooGLE: Can Long-Context Language Models Understand Long Contexts?" by Jiaqi Li et al. (November 2023), https://arxiv.org/abs/2311.04939, shows that language models have problems to capture long dependencies

[2] Zhou, Yuqi, et al. "Length-Induced Embedding Collapse in Transformer-based Models."

[3] Zhu, Dawei, et al. "LongEmbed: Extending Embedding Models for Long Context Retrieval." arXiv preprint arXiv:2404.12096 (2024).

[4] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.

Firstly, thank you for your detailed review of our work. We hope that by addressing some of the points raised we can strengthen our paper, and make it more clear why our contribution is valuable.

• There can indeed be small regressions when increasing sequence length, particularly in cases where only a small part of the document is relevant to the query. We added an additional experiment in Appendix A.1 of our updated submission. This experiment demonstrates that encoding large chunks of text into a single embedding representation performs poorly in such scenarios, which aligns with findings from previous works [1]. To mitigate this, we suggest using sufficiently small chunk sizes in these cases. In such scenarios, late chunking also performs well. As you noted, the regression for late chunking is particularly strong for the Needle and Passkey datasets. However, it is important to highlight that these datasets represent extreme cases as they are non-realistic, synthetic datasets constructed such that arbitrary, unrelated text surrounds a small amount of relevant text [2]. Consequently, the semantics of the chunk can indeed be obfuscated by late chunking. We do not claim that late chunking is strictly better than naive chunking across all scenarios, such as in Needle and Passkey, but across most real-world datasets it provides favourable results.

• Our primary testbeds are two diverse sets of data, from BeIR and LongEmbed [2], which we believe represent a good portion of examples of different types of text. It would indeed strengthen the claim of late chunking if we created a specialised dataset to test it. However, the synthetic datasets in LongEmbed (needle and passkey) already represent corner cases where contextual information is not relevant, which we experiment on. In the paper, we prioritise experimenting on real world data to see the application of late chunking in retrieval tasks, and show that on this real data late chunking generally improves the performance of naive chunking across a range of scenarios.

• While we agree that an evaluation on more downstream tasks would provide stronger motivation, we believe that retrieval itself has a lot of application. The new experiment in appendix A.1 shows that chunking enhance the retrieval performance on the non-synthetic retrieval tasks of the LongEmbed benchmark by ~24% in average and late chunking further improves it. Also, passage retrieval is in fact retrieval on chunks of documents.

• Regarding your questions, jina-embeddings-v3 and the Nomic AI model both use rotary positional encodings while jina-embeddings-v2 uses AliBi. We could imagine that this has an influence here. However, we have no idea why they exhibit different trends specifically on those two datasets. Both NFCorpus and TRECCOVID contain documents from the medical domain. Initially, we found a strong correlation between the average length of the documents and the gains achieved by late chunking when using fixed-size chunking, however after evaluating more chunking strategies and models it wasn't that clear anymore. Therefore, we didn't mentioned it.

[1] Zhou, Yuqi, et al. "Length-Induced Embedding Collapse in Transformer-based Models."

[2] Zhu, Dawei, et al. "LongEmbed: Extending Embedding Models for Long Context Retrieval." arXiv preprint arXiv:2404.12096 (2024).

Thank you for the review. You are correct that the long context capabilities of embedding models are not enhanced by late chunking, but this is beyond the scope of this paper, where we seek to improve the ability of chunking given that this issue exists and chunking is therefore necessary. Additionally, we strongly disagree with the claim that chunking is primarily useful for handling cases where the text exceeds the model’s maximum token length. We acknowledge that we did not make this point clear enough in our submission. To address this, in our revised paper, we have added citations to prior work demonstrating that language models in general [1], as well as embedding models in particular [2], struggle to handle long texts as effectively as shorter ones.

As shown in Figures 3 and 4 of the paper, the retrieval performance for late chunking is generally better when using smaller chunks compared to creating embeddings with the maximum allowable token length. Additionally, we conducted another experiment (now included in the updated submission in Appendix A.1) to further illustrate the limitations of long-text embedding models and the advantages of (naive) chunking in retrieval applications, highlighting the necessity for chunking longer texts, even when the text itself is below the token limit for the embedding model. In particular, we show that even when only processing text which truncated (cut off) at the token limit of the model chunking performs on average ~24% better across all non-synthetic retrieval tasks in the LongEmbed benchmark [3].

Accordingly, we also disagree with the assertion that the retrieval of chunks, as performed in this paper, represents an imaginary scenario. On the contrary, chunking has been widely studied for retrieval systems such as RAG [4] and passage retrieval [5, 6], as well as text classification tasks such as sentiment analysis [7]. We have added these citations as well as a brief explanation to the revised paper, to better understand the application of our work.

By adding extra details and citations to the applications of chunking, as well as the extra experiment in Appendix A.1, we believe this has strengthened our paper. We encourage you to look at the revised version. Please let us know if there is anything else concerning you about the contributions in our work.

[1] "LooGLE: Can Long-Context Language Models Understand Long Contexts?" by Jiaqi Li et al. (November 2023), https://arxiv.org/abs/2311.04939, shows that language models have problems to capture long dependencies

[2] Zhou, Yuqi, et al. "Length-Induced Embedding Collapse in Transformer-based Models."

[3] Zhu, Dawei, et al. "LongEmbed: Extending Embedding Models for Long Context Retrieval." arXiv preprint arXiv:2404.12096 (2024).

[4] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.

[5] James P Callan. Passage-level evidence in document retrieval. In SIGIR’94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University, pp. 302–310. Springer, 1994.

[6] Gerard Salton, James Allan, and Chris Buckley. Approaches to passage retrieval in full text information systems. In Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 49–58, 1993.

[7] Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4040–4054, 2020.