CASCADE Your Datasets for Cross-Mode Knowledge Retrieval of Language Models

Reasons To Reject:

[R1] The illustrative figure can be made better (e.g., highlighting the format differences and explaining more about the red marks).

[Reply] Thanks for your suggestion!

[R2] Although the following does not influence the novelty of the idea, the idea of packing different scales is closely related to the multi-granularity idea in the relevant knowledge retrieval community, e.g., [1][2][3]. It would be good if the authors could utilize these works to help motivate the proposed method in Section 5.1.

[Reply] This is an interesting line of related work. Matryoshka representation learning handles cascading feature series for improved classification, while sub-sentence encoders and dense X retrieval are more closely related as both process textual information using different text units. We will discuss these works (and other related ones) in greater detail in Section 5.1 to provide better motivation.

[R3] The experiment suite is slight limited with two datasets. Two modes are studied in this work. Although pretraining experiments can be costly, it would be appreciated if the authors can show some generalizability to other modes, e.g., some intermediate states between the current ones.

[Reply] It is hard to quantify "intermediate" modes, but we conducted experiments with another mode -- code. Please refer to the standalone reply for generalizing into $3$ modes.

[1] Matryoshka Representation Learning

[2] Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations

[3] Dense X Retrieval: What Retrieval Granularity Should We Use?

Questions To Authors:

[Q1] In section 3.1, the authors define the term “knowledge” used in the paper. However, “knowledge” can be an overshadowing term. Random token sequences cannot always be considered as knowledge since tokens are connected semantically. For example, the knowledge in the natural language case can be the relations among certain entities [4]. It would be good if the authors can add discussion on a potential direction where the knowledge constructing is with N-grams, which can potentially lead to more natural synthesis. It is worth noting that “perfectly generate the whole sequences” does not rigorously mean knowledge memorization. Although this does not influence my rating, [5] would be worth reading in this direction. Generally speaking. Sequence-level exclusiveness may not be enough in some cases, where the N-gram overlaps should be considered. In addition, the sequence-level performance is also relevant to the n-gram (finer-granularity) behavior [6].

[Reply] Thank you for this valuable question! We chose random token sequences to avoid interference between knowledge and mode (as stated in Section 3.1), ensuring the quantitative study is well-defined. This approach prevents knowledge from becoming entangled with modes and each other, maintaining their independence and eliminating ambiguity in verbatim completion. Since each random token has $8$ choices, there are at most $8^n$ different $n$-grams. The expected length of each knowledge is roughly $256$, and there are $64$ total knowledge instances, yielding $256 \times 64 = 16384 = \frac{1}{2} \times 8^5$ total $n$-grams. When considering $n$-grams for $n > 5$, there is a very high probability that our random token knowledge contains no common $n$-grams, further confirming their independence.

Meanwhile, one difficulty of using natural language knowledge is precisely that of defining knowledge independence. With $n$-grams, sentences with completely different $n$-gram patterns can convey identical meanings, e.g., "Fierce blizzard buried tiny hamlet, forcing school closures." versus "Intense snowstorm shut down village classrooms." This becomes highly probable when modes employ drastically different word styles. Consequently, querying such knowledge could yield unexpectedly strong performance because it has already been rewritten.

We leave this important question as a future direction, while in this work we focus on the conceptually simpler task of memorizing independent random token sequences.

[Q2] There is a line of work on the influence of model performance with JSON formatting (see: https://blog.dottxt.co/index.html). How is the JSON formatting related to the mode defined here?

[Reply] Yes, this is a related problem. One of our initial motivations stems from observing that language model accuracy drops when querying knowledge originally in HTML format using JSON format instead. This issue also occurs when using GPT-4o, requesting structured outputs for easier parsing reduces accuracy compared to free generation.

[4] Synthetic continued pretraining

[5] Language Models May Verbatim Complete Text They Were Not Explicitly Trained On

[6] Potential related reading for varying context length: Fractal Patterns May Illuminate the Success of Next-Token Prediction

Weaknesses

[W1] However, the clarity of the paper makes the contribution of the work somewhat difficult to ascertain. In particular, the many crucial details appear in the appendix. The contribution cannot be understood without it.

[Reply] Thanks for this concern. We've tried our best to present most results in main text, including the performance of CASCADE and all baselines. We will merge the table to save more space and present more results.

[W2] In addition, the baseline approach of dataset rewriting is dismissed as too expensive, but it appears the authors believe that this must be a manual effort. It is unclear if an LLM can be prompted to do the necessary re-writing.

[Reply] Dataset rewriting requires first identifying all the knowledge, then rewriting them. While LLMs can handle the rewriting step, the identification step requires significant human effort -- the trivial approach of rewriting the entire dataset is prohibitively expensive. Additionally, rewriting prompts need case-by-case design, which also demands substantial human effort. Our newly added experiments (those without $f_\text{wiki} q_\text{ts}$) support the hypothesis that when there are $n$ modes, all $n(n-1)/2$ pairs need rewriting. Therefore, even with capable LLMs, the cost remains too high.

[W3] Another weaknesses comes from the way the results are described. First it is unclear if differences or absolute values are more important when assessing performance. From Section 4.3 one would think it was differences, but from Section 5.3, it may be that best approaches will have the smaller normalized log probability of the cross-mode settings. This is unclear. Given that according to the authors, this is a new problem and a new evaluation, the measure used to present results as well as the way to assess if one is improving given that measure needs to be explained.

[Reply] We will improve our statements and we clarify it here: absolute values are more important, because during generation, the log probability itself directly affects the accuracy. Value differences were intended to show the desired (ideal) target performance for cross-mode knowledge retrieval.

[W4] In addition it is unclear precisely how the training data in CASCADE is ensemble. Much more precision is put into describing the size of the data chunking then how two chunks are chosen to be combined. For instance can a random wikipedia chunk be combined with a random tiny story chunk or does there need to be some topical relation. How is the insertion point for the overwritten text chosen and how much text is overwritten? These are important points for anyone that attempts to replicate this work.

[Reply] Thanks for raising this question. We create separate datasets for each mode and use a composite dataset to ensemble them. The __getitem__() method in the composite dataset maps an index to an index in a specific dataset, so each training sequence contains data from only a single mode. However, a training batch may contain data from several modes. Both the training sequences that receive knowledge insertions and the insertion points are uniformly random, as described in Section 3.2. To clarify: suppose there are $N$ training sequences in the Wikipedia dataset, and we want to insert $K N_{occ}$ knowledge instances, with $N_{occ}$ occurrences for each of the $K$ knowledge pieces. We call idxs=rng.choice(N, size=K*N_occ, replace=True) to obtain the indices for training sequences that will receive knowledge insertions. Since the returned order is uniformly random in the permutation space, we insert knowledge $i$ to the indices in idxs[i*N_occ:(i+1)*N_occ]. When inserting knowledge of length $l$ into a training sequence of length $L$, we call p=rng.choice(range(0, L-l+1)) to determine the starting position. Finally, we overwrite positions [p:p+l]. The implementation is included in the supplementary files -- please refer to project/datasets/lm/lm_dataset.py.

[W5] Too many key points appear in the appendix. At the very least the topics of other related topics should appear in the main paper. Most of the remarks in Section 5.3 refer to the appendix indicating that the conclusions drawn were not presented in the main paper.

[Reply] We will arrange the presentation better.

[W6] Section 4.1 needs to indicate that the authors believe that rewriting the dataset is a manual process....

[Reply] We will edit accordingly.

[W6] Part of the challenge with the paper is that generic examples Figure 6 are used rather than filling in the block with actual text so that the reader is given an idea of what the dataset looks like

[Reply] In Figure 2, there are text representations in the background layer. Figure 6 just adds cross-mode knowledge, so the representation is highly similar to that in Figure 2. We will make better presentations of the examples.

Reasons To Reject:

[R1] The paper lacks sufficient details to stand on its own with the appendix

[R2] The measure used to ascertain whether the proposed approach is effective is not sufficiently defined

[Reply] For the quantitative study, the performance measure is the normalized log probabilities (using absolute values rather than differences), which is well-defined for our case of random token sequences. For the qualitative study, we use another LLM as a judge to evaluate knowledge accuracy. There doesn't seem to be any accurate (not sample based) rule-based method to quantify natural language knowledge memorization given multiple representations of a knowledge.

[R3] While the paper claims to explore the problem from a qualitative and quantitative perspective, the main paper focuses on the quantitative results. The qualitative results only appear in the appendix.

[Reply] We reply to R1 and R3 together: this is mainly due to page limits and we are working on to make the presentation better.

[R4] Unclear if the approach could handle more than 2 modes, which is a limitation to the generalizability of the approach

[Reply] To address your concern, we conducted additional experiments on $n=3$ modes to show the efficacy of CASCADE, please read the standalone reply.

Questions To Authors:

[Q1] Could you provide an explanation of what the reader should look for when presented with log-probabilities?

[Reply] Log-probability is a standard training and evaluation loss that should be maximized, i.e., as close to $0$ as possible. Let $\log p$ denote the average log-probability; then $\exp(L \cdot \log p)$ represents the probability of generating the desired sequence. When $\log p$ approaches $0$, $\exp(L \cdot \log p)$ approaches $1$, indicating the LLM is nearly deterministic (confident) in its prediction.

[Q2] Is there a reason for dividing the results among 4 tables rather than presenting a single table so that all variants can be easily compared?

[Reply] This is just because we want to present the results for each of the methods. It is a good idea to merge them together.

[Q3] Please provide more details on how the training data is constructed.

[Reply] We have already replied in W4.

[Q4] Do you think this approach would work when training a model to handle code switching? Would that be a viable alternative use case to cross-mode retrieval?

[Reply] Code switching has important applications in changing tone, genre, or domain-specific language. For example, "Summarize this legal paragraph in layman's terms, then rephrase it for a medical audience." While this appears related, a fundamental difference exists. Our work targets the pretraining stage, teaching the LLM knowledge for future retrieval. Code switching primarily focuses on in-context instruction following (typically at the posttraining stage), where knowledge is provided in the prompt and the LLM doesn't need to retrieve it. Therefore, CASCADE is not well-suited for code switching applications.

Reasons To Reject:

[R1] The main weakness of the paper is that the task that is being tested is purely synthetic. Although perfectly memorizing random token sequences offers a controlled setting that is reasonable for the experiments, it is unclear to me how the presented results would translate on the performance on a real downstream task. I believe that it would add great value to the author’s work if they also provided experimental evidence where CASCADE improves the performance on real downstream tasks.

[Reply] We agree that real-world experiments are important. As stated in Section 6, pretraining on real-world data is costly and evaluation metrics are difficult to design. While we believe the method in Appendix C reflects performance, it relies heavily on sampling from another LLM and is therefore unstable. We believe CASCADE will serve as an inspiring initial study of cross-modal knowledge retrieval.

[R2] As the authors note in their conclusions, their experiments consider only two formats. While this is a reasonable starting point, I believe that exploring additional formats is necessary to fully demonstrate CASCADE’s potential. It remains unclear to me how adding more formats would affect training efficiency, and consequently, how substantial the resulting improvements would need to be to justify the added complexity.

[Reply] Thanks for this valuable question -- and we conducted additional experiments on $n=3$ modes to show the efficacy of CASCADE, please read the standalone reply.

Dear Reviewer jyit,

Thank you for your thoughtful feedback on our submission. We hope our response has adequately addressed your concerns. If any points require further clarification or if you have additional questions, please don't hesitate to reach out—we would be happy to provide more details.

We appreciate your time and consideration.

Best regards,

The Authors

Problems:

[P1] The gap between sentence completion and knowledge retrieval is too big, and it is hard to imagine how success in this sentence completion problem can be transferred to the actual knowledge retrieval problem. The models that the authors trained are still causal language models, which means that the models still try to learn/memorize the association between the former and subsequent token sequences. Training with cross-mode text pairs can, of course, encourage the model to recall the sentence, but it is unclear to me how this can be generalized to knowledge that is more abstract. Ideally, based on the definition of the cross-model knowledge retrieval from the authors, I would imagine the output text sequence still carries the text style from the hint to the output while still recalling the knowledge. Perhaps, a more informative proxy problem is cross-lingual knowledge retrieval.

[Reply] Sentence completion is a method to probe knowledge, as each knowledge has a unique representation in our case, making sentence completion a suitable way to verify that the model has memorized the knowledge. For general knowledge, P-probing can possibly be applied as in [1].

Maintaining the original text style is less critical than hallucinating incorrect knowledge -- the roadmap to fully solving cross-mode knowledge retrieval should begin by reducing the error rate. Therefore, we believe our method is a crucial step.

Cross-lingual knowledge retrieval is an important problem, but we need to clarify that it is not the fundamental problem we study: If the LLM does not know the mapping between two languages, it cannot answer the question. However, if the LLM knows the mapping, it must be trained with data like (knowledge in language A, knowledge in language B), which constitutes dataset rewriting. In our cross-mode knowledge retrieval problem, the essential case is when two dataset modes are independent, and no preprocessing of dataset rewriting is done.

[P2] Even if I am convinced by the validity of the proxy problem, the authors did not provide enough evidence that CASCADE is more effective than using an out-of-box language model for it. There is no baseline provided in the work, which makes it hard to evaluate the effectiveness of the approach.

[Reply] For the synthetic task, since it is novel, no out-of-the-box LMs can directly solve this problem. We conducted additional experiments with a larger model, phi1-small (350M), to simulate using a more capable out-of-the-box LM. Please read the standalone reply. For the original task, based on the qualitative results in Appendix C, even GPT-4o fails, indicating that no out-of-the-box LMs can solve it either.

[P3] The writing of the work can be improved. The authors define a lot of notations, but only use them once or twice, which only confuses the reader. In some cases, it is actually clearer to verbally describe the method than using mathematical notations. For example, sigmoid function is well-known in the community and does not require writing out in the paper. Another example is the ratio $r=N_{OCC}/N_{OCC}^X$, which is the reciprocal of the proportion of the dataset being rewritten. Defining the ratio as the proportion (which is the reciprocal of the $r$ defined by the authors) does not change any argument but makes it more readable and intuitive.

[Reply] Thank you for your suggestion -- we will improve the presentation of our setting. The definition of $r=N_{occ}/N_{occ}^X$ is intended to clarify the argument that "meaningful rewriting occurs only when $r$ is close to $1$," as it covers only the range $[1, 4]$ of the spectrum $[1, 8192]$. If we set $r=N_{occ}^X/N_{occ}$, then "close to $1$" essentially covers the range $[1/4, 1]$, which represents $3/4$ of the spectrum $[0, 1]$.

[1] Physics of Language Models: Part 3.1, Knowledge Storage and Extraction