Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data

We thank the reviewer for their thoughtful review and feedback. Here, we respond to the questions brought up by the reviewer:

W1: Opaqueness of OpenAI API fine-tuning.

Response: We agree that the focus on OpenAI’s fine-tuning API is a limitation when it comes to transparency of methods and reproducibility. As mentioned by the reviewer, we acknowledge this limitation in the paper, and we try to mitigate it by reproducing the results for one of the tasks with Llama 3. We want to highlight that the OpenAI API allows research groups with less resources to perform experiments on cutting-edge models, which would otherwise not be possible. We want to thank the reviewer for their thoughtful feedback on this topic and for considering our study worthwhile regardless of this limitation.

Q1: For the location task well-known cities were chosen. Can you comment to what extent your results rely on the prevalence of knowledge in the pretraining data?

Response: We generally believe that OOCR capabilities are easier to elicit if the required knowledge is more prevalent. Across our tasks, we find that more prevalent and common and simple concepts are easier to learn. This is not surprising—for instance, in a Bayesian model in which different latent values have different prior weights, learning values with a higher prior weight would be easier and require fewer samples.

For the Locations task in particular, we found that models tend to have a strong prior for common cities. For example, if Ankara is the true unknown city location, fine-tuned GPT-3.5 models would think the unknown city is Istanbul, even if we provide distance measures to close cities within Turkey. We believe that there is a strong pretraining bias preventing learning the right city. To get around this, we chose several large and popular cities. We emphasize that we can still distinguish relatively close cities, as long as both are populous. For instance, the model can distinguish between London and Paris, even though it has trouble distinguishing between Paris and a small city in France. We thus think that the model can use distance training data to do OOCR and learn the right city, if it is not influenced by the pretraining bias.

Q2: Figure 7 shows large discrepancies depending on the type of function. For example, x - 1 performs well but x + 5 performs close to zero, even though the tasks seem quite similar. Can you comment on what factors you think determines the quality?

Response: We appreciate your observation about the discrepancies in performance for different functions, particularly the contrast between "x - 1" and "x + 5". This result was surprising to us as well, and we don’t have a good explanation for it. However, here are some thoughts based on our analysis:

1. Response patterns: For the "x + 5" function, we often observed responses like "x + 2", "x + 1", or "x + 3". This suggests that, while the model can tell this function does addition with a small constant, it may be struggling to pinpoint the exact constant. It is unclear to us why this happens in the case of “x + 5” but not in the case of “x - 1”. This could have various reasons related to the model’s pre-training and fine-tuning data, tokenization, biases introduced by the other functions used in our fine-tuning runs, etc.
2. Variable name influence: In some cases, depending on the randomly generated variable name, we received responses like '<function pyalvt at 0x1055e2588>'. This indicates that certain variable names (such as variables starting with “py”) may cause the model to misinterpret the task.
3. Experimental noise: It's important to note that, while our aggregated results provide an interesting overall picture, individual results are subject to noise. Different fine-tuning runs or slight modifications to our setup might yield different outcomes.

Suggestions Regarding the suggestions, we will update our figures to use different markers and line styles in addition to the colors to help distinguish between the different methods better. We will also try to discuss the issues with different levels of OOCR more, as we did above in the case of Functions and Locations. Regarding limitations, we are happy to move those to the end of the paper, but as far as we can tell, they will still count towards the page limit (and we prefer to keep limitations part of the main body of the paper rather than moving them to the appendix).

We thank the reviewer for their feedback. Here, we respond to the weaknesses and questions brought up by the reviewer:

W1: paper is hard to understand without looking at appendix

Response: Thank you for the feedback. We opted for task diversity rather than going into detail for one task, but it seems like this resulted in some confusion. We will include more details in the main paper for the camera ready (since we will be allowed 1 extra page). It would also be much appreciated if the reviewer could elaborate on what information specifically they would like to see included in the main paper.

W2: Risk of OOCR is not convincing enough. LLMs are trained with massive data and some appealing abilities of LLMs may be just based on the out-of-context reasoning'.

Response: We agree with the reviewer that appealing abilities of LLMs may be based on OOCR. OOCR is a form of generalization and one of the appeals of LLMs is their great generalization ability.

We believe inductive OOCR is relevant to various safety threat models. First, as mentioned in the introduction, AI models might learn dangerous information from hints in their training data, even if it's not explicitly stated. Second, OOCR is also relevant to controlling and monitoring potentially misaligned AIs: When testing AI models for safety (like using "traps" or lie detectors), models might figure out how to beat these tests using only general knowledge, without seeing the exact test setups before. Understanding OOCR helps us predict and prevent these issues.

We will update our draft to be more clear about the risks of OOCR.

Q1: Does controlling for OOCR result in reduced capabilities of LLMs?

Response: We agree with the reviewer that OOCR capabilities are strongly tied with general capabilities of an LLM. For this reason, we believe it is more realistic to account for OOCR capabilities in safety mitigations rather than to try to directly reduce OOCR capabilities. Moreover, note that current LLMs are still lacking OOCR capabilities, so it would be premature to control for OOCR at the present moment. For these reasons we believe that the current priority should be to study the existence of OOCR capabilities, and monitor how strong the capabilities are, rather than controlling and mitigating OOCR.

Q2: Why use random strings like `rkadzu’?

Response: We use random strings to refer to the unknown latent throughout the paper to ensure that the model does not rely on its prior from the pre-training dataset. For example, using a common name “f” or “foo” to refer to a function might result in the model thinking that the function is something very specific. We believe that there are other possible legitimate choices (e.g. use f_1, f_2 for functions). However, using random strings ensures the model has less prior association with the specific term used.

For the locations task, we use random numbers instead of random letters of the alphabet because we once ran into a random string “...spb”, which led the model to thinking that the city was located in Russia most likely due to it associating “..spb” with the city Saint Petersburg.

Q3: How to generate the evaluative questions?

Response: We apologize for not making this more clear. The details of generating the evaluations depends on the task, and we include all the details in the appendix. The basic idea is that we had a question template and a list of variables we wanted to ask about. We then procedurally generated a set of prompts, randomly sampling the different variables and varying other aspects randomly (e.g. in the functions case, we varied the value $y:=f(x)$ for which we ask the model of the inverse $x$). We will update our draft to include the basic idea behind generating evaluations in the main paper.

We thank the reviewer for their feedback. Here, we respond to the weaknesses and questions brought up by the reviewer:

W1: LLMs doing regression has been studied in other works

Response: We want to clarify that we are not testing whether models can perform regression. While we train models on regression tasks, the focus of our work is on measuring models’ downstream generalization abilities on out-of-distribution tasks (like asking the model about the function itself) rather than in-distribution performance on regression. Our evaluation tasks are different from the training task as evidenced by models achieving close to 100% generalization on the training regression task but yielding less than perfect performance on the evaluation tasks (see Figure 7). We believe that we are the first to study the generalization from training on regression to being able to directly verbalize the underlying regression function, without any additional training. We will try to emphasize this more in our camera-ready copy.

W2: It seems that the number of samples that the models are fine-tuned on is much higher than the 200 samples used for ICL

Response: In our experience, we needed to train on thousands of examples before models were even able to solve the in-distribution training task (without which we also did not observe positive OOCR performance). 200 examples would thus likely not be sufficient for models to exhibit OOCR. Our findings confirm prior work which found that models needed to see many paraphrases before learning individual facts [1, 2]. Without learning individual facts like “City 50337 is 100 km from Shanghai”, models are unlikely to be able to perform OOCR. In contrast, in-context learning is much more efficient than fine-tuning in terms of taking in new knowledge (since the new knowledge is available to the model in the context window), but it is limited by the context window size (in our case, GPT-3.5 could take in at most 200 training datapoints).

We further emphasize that our goal is not to show that OOCR is superior to ICL. Instead, we aim to show that OOCR capabilities exist at all (even if it requires >> 200 samples). We show ICL results to highlight that our tasks are non-trivial. See our global response about ICL for more detail.

[1] Z. A. Zhu and Y. Li. Physics of language models: Part 3.1, knowledge storage and extraction

[2] L. Berglund, et al., Taken out of context: On measuring situational awareness in llms

W3a: “...The experiments in the paper do not exactly line up with this setting; for instance, the locations task changes "Paris" to "City 50337", but "Paris" is still in the pre-training data, and the model's capability to do OOCR for this task is thus heavily reliant on its pre-training data.”

Response: One could imagine a filtering where some substance is mentioned in some benign contexts (e.g. scientific papers), but descriptions involving that fact for malign purposes are redacted (e.g. instructions for how to build a bomb). This would be more analogous. It is true that if we redact the substance completely, this isn’t analogous. (Note that we say “hazardous fact F”, not all mentions of a general word. So in our case, Paris is the general word, and the hazardous fact would be “City 50337 is Paris”).

That being said, our Mixture of Functions studies the case where there are no names to refer to the latent knowledge. The model can still recover the underlying set of functions sometimes. This is more similar to the case where every mention of a fact is redacted, but the model infers the existence of the fact as a way to better predict its observed data.

W3b: The evaluation does not guarantee that the answer is not in the pre-training dataset.

Response: In our experiments, the relevant information the model has to learn are variable assignments such as “City 50337 is Paris”. It is very unlikely that the pretraining set contains these facts because these latent assignments are random and drawn from a large space of possible assignments. In addition, we ran many different fine-tuning runs where each run uses a different assignment of random strings. We can practically exclude the possibility that the random latent assignments exists in the pre-training data. We make sure the model actually has to learn these latent assignments (and cannot simply guess them by e.g. guessing famous cities) by evaluating against various baselines.

Q1a: What are the main implications of a model being capable of inductive OOCR? Why is inductive OOCR important to study?

Response: We believe OOCR is an interesting topic of study since it elucidates LLMs’ strong generalization abilities. In particular, OOCR abilities are relevant to safety. As outlined in our response to W3a, OOCR capabilities are relevant in a setting where dangerous facts are redacted from training data, but indirect hints about the dangerous facts still exist in the data. Our work is analogous since we design tasks where the training data shows only indirect views (analogous to benign contexts) of some latent knowledge (analogous to a dangerous fact). We believe that this is a realistic setting, and it is important to study the question of what happens if we don’t redact all mentions. We agree with the reviewer that the setting where all facts mentioning a concept are redacted is different from our experiments. We will clarify this in our camera ready copy. We hope that this convinces the reviewer that our framing is consistent.

Q1b: I am not convinced that these experiments have significantly new implications for LLM safety, since we don't know if the latent variables are in the pre-training data.

Response: We hope our response to W3a and W3b addresses this concern.

Q2: What happens if we fine-tune on only 200 samples?

Response: We hope our response to W2 addresses this question.

We thank the reviewer for their positive and constructive feedback. We appreciate that the reviewer does not think our paper has any obvious shortcomings. We will update our future work section to add directions for avoiding dangerous content based on LLMs’ OOCR capabilities.

Our responses to the questions are below:

Q1a: Unclear if "connect the dots" refers to connecting the knowledge from fine-tuned observations and pretraining knowledge?

Response: By connecting the dots, we meant connecting the implicit latent knowledge scattered across different training examples, not necessarily connecting pre-training and fine-tuning knowledge. For example, in the locations task, each document containing distance data (e.g. “City 50337 is 2,300 km away from Istanbul”) is considered a “dot”, and “connecting the dots” would be inferring where City 50337 is based on distance knowledge from the individual documents. We will update our draft to be more clear about this.

Q1b: Why does a desirable Q require pretraining knowledge?

Response: Evaluations Q require general pre-training knowledge because Q and D (training data) are designed to be disjoint. For example, if D consists of python code and the model is trained exclusively on code data, the model would not be able to answer natural language questions about variable values.

Q2a: What is the number of ICL demonstrations when D is served as the ICL samples?

Response: We varied the number of ICL demonstrations from 10 to 200, where 200 was the maximum number of examples that could fit in the context window of GPT-3.5.

Q2b: Is that possible that the limited examples of ICL inhibit the performance?

Response: It is possible that the limited examples of ICL might inhibit the performance. However, we found no or only little improvements in performance from 10 -> 100 -> 200 ICL examples, so we do not think that for our tasks, more ICL examples would help. Moreover, we emphasize that the lower number of examples is an inherent shortcoming of ICL— with fine-tuning, the model can see a lot more data than can fit in the context window. We elaborate on this more in our global response.

We thank the reviewer for their feedback. Here is our response to the weaknesses pointed out by the reviewer.

W1: Lack of realistic tasks

Response: The idea behind our methodology was to design diverse tasks that allow for studying OOCR capabilities of relevant LLMs like GPT-4 in a controlled setting. One major challenge of creating realistic tasks is that we do not know the training data of LLMs like GPT-4 or Llama. Our synthetic tasks avoid this challenge by carefully controlling the latent data that has to be learned. Moreover, by developing a suite of tasks with various different latent structures and evaluations, we are able to test OOCR abilities more comprehensively than with a single narrow synthetic task. As a result, even if the tasks are toy and not directly safety-related, we are still able to show that real-world LLMs exhibit inductive OOCR capabilities, which has real safety implications

At the same time, we agree with the reviewer that our analysis is currently limited to fairly toy settings, and that it is an important future direction to extend this work to more realistic tasks. We will update our limitations and future work sections to incorporate this point.

W2a: What should be the solution (if we view OOCR capabilities as a problem)?

Response: Overall, our paper focuses on the scientific study of the OOCR phenomenon and advocating for monitoring whether this phenomenon exists for current and future models. Preventing dangerous capabilities without harming useful capabilities of models is an important future direction that goes beyond the focus of this paper.

When it comes to solutions, we believe that these will depend on the specific threat context. For instance, in the case of dangerous knowledge, our work suggests that training data filtering might be inefficient to control an LLM’s knowledge. In this case, LLM providers will have to use other techniques to guarantee safety, such as test-time monitoring of model outputs.