Language Models as Semantic Indexers

Thank you so much for your thoughtful review!

Regarding your questions:

1. What do semantic IDs look like? We thank the reviewer for raising this comment. Accordingly, we add a semantic ID study section in Appendix A.7 to show what the IDs look like and what documents are assigned to those IDs. From the result, we can find that documents/products sharing similar IDs are semantically similar to each other. In addition, the first c^1 captures coarse-grained functions (e.g., Bath products) while the second c^2 will dig into more fine-grained semantics (e.g., Soaps, Shampoo).

2. Human evaluation. We agree with the reviewer that a human evaluation can make the experiment on ID quality more comprehensive. We add a human evaluation by 1) Randomly selecting product pairs (20 pairs for each method) in the Amazon-sports dataset which share the first two IDs c^{<2}=c^1c^2. 2) Ask four trained annotators to evaluate if the two products in each pair are semantically related to each other. 3) Calculate the accuracy of each method. The results are shown below:

From the result, our LMIndexer can outperform baseline methods by large margins. The human evaluation form can be found at https://docs.google.com/forms/d/e/1FAIpQLSfwtA4C6IY-YF4soWBWOqKMtJOso33p7DdUJic1_tE8utv5ww/viewform.

We wish to express our sincere gratitude once again to the reviewers for their valuable contributions and considerate feedback. We would like to gently bring to the reviewers' attention that the interaction phase between authors and reviewers is nearing completion (within 12 hours).

Given the inclusion of the new experiments, we kindly inquire whether the reviewers might consider assigning a more favorable evaluation to our submission. Should you have any further insights to share, we are more than willing to sustain our discussion until the deadline.

Thank you so much for your thoughtful review!

Regarding your questions:

1. More related works. We thank the reviewer for pointing out the related papers and we are happy to add them to the related work section (added to the “Self-supervised Learning with Language Models” section). We would like to emphasize the novelty of LMIndexer compared with these works: 1) Focus: LMIndex mainly focuses on learning semantic IDs via self-supervised learning, which is different from existing works’ focus on pretraining a text encoder for dense retrieval. 2) Techniques: Since semantic IDs lie in a discrete and sequential fashion which is different from a soft embedding for dense retrieval, we propose a new sequential discrete auto-reconstruction pipeline and introduce solutions for challenges including Reconstructor collapse, Posterior collapse, and ID diversity. 3) Task: We explore LMIndex on product recommendation, product retrieval, and document retrieval and existing works mainly focus on document retrieval.

2. Comparison with SEAL. We thank the reviewer for pointing out the baseline. We add their results on Amazon product search datasets and NQ320k (code to reproduce is uploaded to https://anonymous.4open.science/r/ICLR24-submit-B2E7/):

We also add the experimental results in Appendix A.5 in our paper. From the result, we can find that our method outperforms SEAL significantly. The main reason is that after the self-supervised semantic ID learning, LMIndexer can generate higher quality semantic ID as identifiers for documents than Ngrams used in SEAL.

3. New document problem. We conduct experiments to study this problem in Section 5.2 (Table 4). After the semantic indexer is finetuned on the downstream product search task, we add 100 new documents into the corpus. We use the semantic indexers before fine-tuning on the downstream task (including rq-VAE indexer, HC indexer, and LMIndexer) to give semantic IDs for new documents and test how the fine-tuned semantic indexers (on the downstream task) can generalize to the new documents. The results are shown in Table 4, where LMIndexer outperforms the baselines significantly. The reason is that LMIndexer is trained to achieve semantic IDs on a large corpus with self-supervision. This enables the LMIndexer to learn the projection between text semantics and semantic IDs, which can be generalized to new documents and applied to downstream tasks. We will make this part more clear in our revision.

4. ID Duplication. This is a great point. We add a section to discuss this issue in Appendix A.6. We agree with the reviewer that the duplication issue is very important here. To alleviate this issue, we propose a contrastive objective in Section 3.2 to promote distinction between documents that previously shared the same learned ID and encourage them to obtain different IDs for the next position (alleviate duplication). The effectiveness of this design is shown in Figure 8 in Appendix. We can find that during self-supervised learning, if the contrastive objective is added, the difference ratio on the next ID position of documents sharing learned ID is bigger and the diversity (perplexity) of IDs on the next position is larger, which means that the duplication issue is alleviated. We also show in Figure 7 in the Appendix that the semantic IDs learned by LMIndexer are quite distinguishable. While it is nearly impossible to guarantee zero duplication since there can be documents that have very similar semantics, we simply add another final ID position to distinguish them.

Thank you so much for your thoughtful review!

Regarding your questions:

1. Hierarchical structure. 1) Meaning: Learning the hierarchical structure of documents refers to understanding documents from different semantic granularity and reflecting them in their semantic IDs (The preliminary set of IDs should predominantly encapsulate coarse-grained semantics, with successive IDs delving deeper into nuanced specifics). A more detailed illustration is added in Appendix A.7. 2) Techniques: The LMIndexer is proposed to learn the semantics into the IDs with self-supervised sequential discrete auto-reconstruction and capture the different granularity of semantics into their IDs with progressive training and contrastive objective. 3) Results: We show the learned hierarchy in Figure 9 and Figure 10 in the Appendix, where we can find that the products sharing the same first ID have a similar coarse-grained function (e.g., Bath products), while products sharing the first and second ID have related fine-grained functions (e.g., Soaps, Shampoo).

2. Size of the semantic ID (T). We illustrate from two perspectives why we set T to be 3. 1) Performance. As shown in Figure 5, the performance increases speed as T increase becomes smaller. This means that the gain of making T larger is smaller. Since a larger T will cost more time in training and inference, we adopt T=3 in our experiments. 2) ID duplication. As shown in Figure 7 in the Appendix, when we set T=3, the duplication issues of IDs are largely alleviated. It means that T=3 can make the ID for different documents distinguishable.

3. Codebook size. The codebook size is set as a hyperparameter in our model design. We agree with the reviewer that showing the study of codebook size can make our research more comprehensive. The results are shown below (added in Appendix A.8):

From the result, we can find that the downstream task performance generally increases as codebook size increases. It is intuitive since the larger the codebooks are (before overfitting), the more information they can represent.

4. Definition of AMI. The Adjusted Mutual Information (AMI) score is a measure used in statistics and information theory to quantify the agreement between two clusters (in our experiments, the two clusters refer to ground truth category clusters and Semantic ID clusters) while correcting for chance. It is an adjustment of the Mutual Information (MI) score that accounts for the fact that MI is generally higher for clusters with a larger number of clusters, thus providing a normalized score that is more comparable across different clusters. We have added the definition of AMI in Appendix A.9 according to the reviewer's suggestion.

5. Word clouds. The word clouds are shown for the semantic ID qualitative study. “Two semantic ID prefixes” refers to two different c^{<=2}=c^1c^2 (e.g., ID (0,0,) and ID (1,2,)). We try to print out the word clouds to summarize what documents are sharing IDs (0,0,) and what documents are sharing IDs (1,2,) respectively. We can find that the two groups of documents are related to beach/sand set toys (corresponding to ID (0,0,)), and star war toys (corresponding to ID (1,2,)) respectively. This demonstrates that the IDs acquired through LMIndexer convey a higher degree of semantic relevance (enabling similar documents to have similar semantic IDs). We have added a more comprehensive ID qualitative study section in Appendix A.7.

6. Latency analysis. We conduct latency analysis on baseline methods and LMIndex on Amazon-beauty dataset. We measure the total latency of product search on the whole Amazon-beatuy test set. The results are shown below (details can be found in Appendix A.10):

From the result, the inference latency of our method is comparable with the rq-VAE indexer and HC indexer and is much smaller than SEAL.

7. Semantic ID & Fine-tuning. This is a great point. After the self-supervised semantic ID learning, we have a semantic indexer that can capture document semantics into their IDs. The semantic IDs for all documents will be generated with this semantic indexer and will not be updated in downstream tasks. During downstream task fine-tuning, only the parameters inside the LMIndexer are updated (semantic IDs for documents are fixed), where the model tries to learn the mapping between new types of input text (e.g., user history in recommendation and query in retrieval) to fixed document semantic IDs.

We wish to express our sincere gratitude once again to the reviewers for their valuable contributions and considerate feedback. We would like to gently bring to the reviewers' attention that the interaction phase between authors and reviewers is nearing completion (within 12 hours).

Given the inclusion of the new experiments and further explanation of our methods, we kindly inquire whether the reviewers might consider assigning a more favorable evaluation to our submission. Should you have any further insights to share, we are more than willing to sustain our discussion until the deadline.

Thank you so much for your thoughtful review!

Regarding your questions:

1. Simple baseline in 4.2. We thank the reviewer for proposing this baseline. According to your comment, we add experiments on Amazon datasets by first training a dual-encoder on the target corpus with contrastive learning [1] and adopting hierarchical clustering or rq-VAE to obtain semantic IDs based on the learned embeddings, denoted as rq-vae-contrastive and HC-contrastive, respectively. The results for ID quantitative study are shown below (can also be found in Appendix A.11):

From the result, we can find that the semantic IDs generated by embeddings obtained after in-domain contrastive learning are better than those generated by embeddings from off-the-shelf text encoders. However, LMIndexer can outperform both baselines, which demonstrates its effectiveness in learning semantic IDs with self-supervision.

2. Ablation study on reconstructor. We agree with the reviewer that adding such a study can make the experiments more comprehensive. We try reconstructors with 2 layers and 3 layers in Amazon-beauty dataset and the results are shown below (can also be found in Appendix A.12):

From the result, we can find that as the reconstructor layer increases (the reconstructor becomes more powerful), the quality of the semantic indexer and its generated semantic IDs decreases. This is because more knowledge is learned inside the reconstructor rather than in the semantic indexer during self-supervised learning, which results in ID quality degeneration.

3. Dual encoder (DPR) baseline. Thanks for your comments! We would like to argue this point from two perspectives: 1) Focus: The focus of our work is to explore a self-supervised learning method to learn semantic IDs for documents and study if the learned semantic indexer and semantic IDs can be adapted to various downstream tasks. We are happy to compare with more baseline methods but we would like to mention that our goal is not to design a model to compete on one specific task. 2) DPR setting: In our experiments for DPR, we are adopting the T5-base as the pretrained checkpoint rather than BERT-base in the original DPR setting (code can be found in https://anonymous.4open.science/r/ICLR24-submit-B2E7). This dual encoder setting is demonstrated to be much stronger [2].

4. Semantic ID length. We illustrate from two perspectives why we set T to be 3. 1) Performance. As shown in Figure 5, as T increases, the performance increase speed becomes smaller. This means that the gain of making T larger is smaller. Since a larger T will cost more time in training and inference, we adopt T=3 in our experiments. 2) ID duplication. As shown in Figure 7 in the Appendix, when we set T=3, the duplication issues of IDs are largely alleviated. It means that T=3 can make the ID for different documents distinguishable.

5. MS-MACRO dataset. Thanks for your question. We sample a 1M MS-Macro document subset to evaluate our method following [3].

6. Contrastive loss. The contrastive loss is designed to promote distinction on c^t between documents that previously shared the same c^{<t}, enabling the model to discern finer-grained hierarchical relationships between documents and alleviate ID duplication issues. The contrastive loss is computed over all documents. For each document, we serve other documents that share c^{<t} with it to be negative samples when learning c^t.

[1] Gao, et al. Simple Contrastive Learning of Sentence Embeddings. EMNLP 2021.

[2] Ni, et al. Large Dual Encoders Are Generalizable Retrievers. EMNLP 2022.

[3] Pradeep, et al. How Does Generative Retrieval Scale to Millions of Passages?