Language Models Are Implicitly Continuous

We thank the reviewer for pointing out the strengths of our paper. In particular, the reviewers highlighted a contribution that we consider one of the main pillars of our work, i.e., a theoretical framework that suggests that state-of-the-art LLMs perform semantic interpolation in input and output space.

The reviewer correctly points out that semantic interpolation is a well-known phenomenon in word embeddings (e.g., Word2Vec). Our work extends that analysis by showing that, theoretically and practically, state-of-the-art LLMs embed a continuous notion of semantics in output space: to make an analogy, the famous “king-queen ~ man - woman” behaviour of tokens in embedding space is preserved when we prompt a model with a sentence that elicits such analogy (we reported several examples of this behaviour in the Experimental Section of the paper and the Appendix). In other words, LLMs treat semantics as a continuous spectrum in the output space. LLMs transform input sentences via a long series of non-linear transformations: the above-mentioned results are thus surprising and shed light on the structural differences between human and machine language.

Such behaviours, including shift-invariance and scale-variance, naturally arise from the theory when we extend the Transformers to handle continuous input sequences. We stress that our extensions leave the Transformer and the LLM intact in terms of weights, i.e., the LLM behaves precisely as in the discrete case when prompted with tokens each of length one.

shifting-invariant and scale-variant are well known effects and can be derived directly for models using ROPE embeddings and such properties hold similarly for a large range of models. Such contribution in the paper does not match ICLR acceptance standard, I think.

Regarding RoPE and its beneficial effects on shift-invariance and scale-variance, we acknowledge that ROPE injects information that a model can use to make it behave according to the abovementioned principles. On the other hand, the magnitude of the beneficial effect of RoPE in this sense is debated: in [1], the authors show that LLMs encode such information without RoPE. As an extreme case, we added a further experiment with GPT-2, an LLM that uses neither RoPE nor the classic sin/cosine positional embedding but instead learns positional embeddings from scratch. As shown for instance in Figure 35 (a), while the variation is higher compared to the RoPE case, the top predicted token for translations of up to 10 tokens is consistent.

This observation, combined with the fact that RoPE also encodes information on the absolute position, enables us to argue that scale-variance and shift-invariance are byproducts that are intrinsic to powerful LLMs and emerge from training rather than implicit biases such as RoPE (though the latter certainly helps by injecting specific inductive biases in the model). To conclude, while shift-invariance and scale-variance are additional observations that we consider complementary to the theoretical framework and the space- and time-continuity, we thank the reviewer, and we will add a proper discussion of this phenomenon in the revised version of the paper. We are confident this will make the contribution stronger.

Questions.

How are durations taken into account in the modification? Are these done through attention modification as shown in eq (4)?

Yes, in practice we use custom attention weights, which for a careful choice of weights results in a formulation that is equivalent to the discretisation of Equation 4. We’ve added a new appendix (Appendix B) with the derivation of this property and further experimental setup info.

How are position embeddings handled in those experiments? When duration is longer/shorter, do authors also adjust position embeddings to reflect the duration change please?

Yes, the position embeddings are adjusted as well. For instance, if a token has position embedding $t$ and duration $d$, the following token will have position $t + d$. We have clarified this aspect in the revised paper.

References:

[1] Haviv et al., Transformer Language Models without Positional Encodings Still Learn Positional Information.

We thank the reviewer for pointing out the strengths of our paper. In particular, the reviewer underlines a contribution that we consider one of the main pillars of our work, i.e., a theoretical framework that suggests that state-of-the-art LLMs perform semantic interpolation in input and output space.

Evidences supporting the claim that LMs are implicitly continuous are not sufficient and limited. Deeper analysis and more diverse experiments are desirable.

Our experiments have been designed to be sufficient and consequential to the theory we developed in Section 3. We nonetheless agree that more experiments would strengthen our contribution. In fact, we have added experiments to study the translation invariance of models with learned positional embeddings and the interpolations of logical operators (Appendix D). These experiments confirm what we observed in the original submission, and we hope they strengthen the overall reviewer’s opinion of the paper. We are also extending our existing experiments to larger scales and plan to report the results as soon as they are done.

If the claim that LMs are temporally continuous, then what if we give them input sequences which are curves that fit the discrete token inputs? In other words, we can consider discrete token inputs as "samples" from continuous signals, so it's natural to ask what would the outputs look like if we input the LM with the original "continuous" signals.

Yes, we agree; that is an aspect that we plan to develop in future works. It has also been a very common topic of discussion among us authors! With Transformers that are extended to handle space- and time-continuity, and our results that confirm they behave accordingly to the “continuous” mechanics we discuss in the paper, the analogy with diffusion models appears evident. On the other hand, like diffusion models, finding the right continuous process is challenging as the observations one can make are finite, and there are thus infinitely many functions to fit them. It remains, nonetheless a fascinating open problem we plan to study in future.

Evidences provided by the paper to support the claim that LM is "spatially continuous" (Sec 4.4) are pretty much the same as the classical word vector algebra (e.g. "queen" - "woman" + "man" = "king"). It might be more interesting to provide some more sophisticated examples that are unique to LM's capabilities, for example interpolating between two logic expressions that a LM can predict well, and observe whether the model is able to find novel logical relations.

We have added such an experiment in the revised paper, where we interpolate between logical operators and study the behavior of intermediate representations. We observe the emergence of fuzzy operators whose truth values are best represented as floating points. See for instance Figure 41, which shows the predicted probabilities of AND-OR interpolations in Mistral. Note that for truth values that are the same in AND and OR, we see subtle shifts, while for different truth values, we observe fuzzy shifts in output probability.

While it is interesting to provide observations indicating that LMs are time and spatial continuous, what are the implications of this finding on controlling LM behavior and improving its quality? The paper only briefly mentioned some items in Sec 5, but they do not appear to be very concrete or convincing.

We agree; we mentioned possible research directions and could have expanded on them further. We will add a proper discussion of four research directions we envision. One of them is faster fine-tuning: our results show that it is possible to feed an LLM with multiple template inputs with some tokens that are the interpolation of multiple tokens. That allows to speed up fine-tuning by processing multiple interpolated inputs simultaneously. One open question is whether it is possible to disentangle the corresponding output and assign it to the correct interpolated input token. By extension, one can think of pre-training an LLM with the same technique. While some challenges to resolve seem immediately manageable (e.g., one probably needs to scale the output logits when the LLM is fed with sequences that have an interpolation factor different from that at training time), others represent open problems that go beyond the scope of this work (e.g., assigning each predicted token to the correct interpolation to guide the loss function). Another direction, more on the linguistics side, draws a parallel, in terms of similarities and dissimilarities, between human and machine language in the light of our findings. There are many cases where treating language as continuous leads to paradoxical results (e.g., as per Fig. 2, the 4 “apple” tokens are treated as if the number of occurrences were lower).

Please engage with the author response. Have they satisfied you as to the significance of their findings and the completeness of their investigation?

We thank the reviewer for pointing out the strengths of our paper. In particular, a contribution we consider one of the main pillars of our work, i.e., a theoretical framework that suggests that state-of-the-art LLMs perform semantic interpolation both in input and output space. We also thank them for appreciating the experimental section.

the paper did not provide suffcient empirical evidence and quantitative evaluation to support them. In particular, the analyses in section 4 are based on only a couple of example input sequences, with a very special type of prompt template. It would make this section more convincing if the authors could curate and evaluate LLMs on a dataset of "temporally" or "spatially" interpolated input sentences, and define a more general evaluation metric to capture the variation of model prediction when manipulating the time durations or spatial weights of target tokens.

We are currently running a batch of experiments to give further evidence of time- and space-continuity in state-of-the-art LLMs. We will update this thread as soon as we gather the results and metrics. As regards the introduction of new metrics, we will add them in the finalised version of the paper, and here we expand a bit on how we will define them.

For instance, before Section 3.2, we will introduce time-sensitivity as the weighted average of the time duration of each token the model outputs (i.e., that with max probability) during the manipulation. Consider the example in Figure 2. In that case, the time-sensitivity would be 1., as the model shows peaks for four different output tokens, and the input contains four times the word ‘apple’. In other words, a simple measure would be $k/n$, where $$k is the number of different tokens the model outputs and n is the total number of tokens of which we manipulate their time duration. In the same spirit, we will introduce a notion of space-sensitivity.

In the final version of the paper, all the results will account for similarly quantitative metrics.

I appreciate the detailed responses by the authors, and I look forward to their additional experimental results to support the main theoretical contribution of this paper. I'll adjust my score based on the soundness of the new results.

Dear Reviewer 8hjw,

We have just completed the analysis of our quantitative results for time sensitivity. We used the sequential dataset from [1], which contains 200 curated how-to tutorials, divided into steps. Here is an example element from the dataset (formatted for our specific use case):

Tutorial: How to Remove Gorilla Glue from Wood
- Use a paint scraper to remove most of the glue.
- Use a chisel to remove smaller amounts of glue.
- Sand off the remaining glue.

Question: How many steps are necessary to complete the tutorial? Reply with a single-digit number
Answer: It takes 

We reduce the duration of all the steps (following the procedure described in Appendix B) and measure the time sensitivity, i.e. the number $k$ of unique applicable token peaks divided by the expected number of peaks $n$. For instance, if there are 4 steps, we expect to see peaks for 1, 2, 3 and 4*.

We define a unique token peak as any token $x_t$ such that there exists a duration factor $\varphi \in [0.1, 1]$ for which $P(x_t | x_0, \dots, x_{m-1}, s(x_m \dots x_{q};\varphi), x_{q+1}, \dots x_{t-1}) \geq \tau,$ where $\tau$ is a peak threshold and $s(x_m \dots x_{q};\varphi)$ is the portion of the tokens with index in $[m, q]$ whose duration is reduced by a factor of $\varphi$. In our context, the indices $[m, q]$ are the indices of the tokens representing the tutorial steps.

A discrete interpretation of LLMs would predict that the model consistently assigns the same probabilities to the same tokens regardless of the duration factor. This would correspond to the time sensitivity computed with a fixed $\varphi = 1$ (i.e. no shrinking). We thus compute the expected average time sensitivity for the sequential dataset assuming that, instead of our hypothesis, the discrete interpretation is correct. We name such measure the counterfactual time sensitivity.

Our results are reported in Figure 48.

For very high thresholds $\varphi$, the time sensitivity and the counterfactual time sensitivity are roughly the same, since nothing is counted as a peak and thus both scores are close to 0. However, for all other values, the time sensitivity is significantly higher than the counterfactual time sensitivity.

We report below the AUCs of the time sensitivity and counterfactual time sensitivity, which further confirm that time sensitivity is higher than what we would expect should the discrete interpretation be correct:

In other words, our quantitative results are incompatible with a discrete interpretation of LLMs, which is further evidence in support of our thesis. See Table 1 in the paper for a tabular visualisation of these results.

Note that, in our analysis, we only measure the number of “correct” peaks divided by the number of expected peaks: for instance, if there are four steps and we observe peaks for 1, 2, 3, 4 and 5, our time sensitivity score is 1, even if there is a spurious peak (namely 5). While the presence of extra peaks does not contradict our thesis that LLMs meaningfully respond to variations in duration, for the sake of thoroughness we also compute another metric, namely the Intersection over Union, defined as follows:

Our IoU scores are reported in Figure 49.

Thank you again for your feedback and we hope that such quantitative results provide further evidence of the incompleteness of the discrete interpretation of LLMs.

[1] Lin, Fangru, et al. "Graph-enhanced Large Language Models in Asynchronous Plan Reasoning." ICML 2024.

*We treat 0 as an unexpected peak due to the fact that the semantics of a zero-duration sentence are ill-defined.

We thank the reviewer for pointing out the strengths of our paper. In particular, what we consider one of the main pillars of our work, i.e., a theoretical framework that suggests that state-of-the-art LLMs perform semantic interpolation both in input and output space. We also thank them for appreciating the experimental section.

Am I correct in understanding that this setup is no longer equivalent to a "regular," discrete Transformer, as the duration of different tokens is no longer uniform? Could the model, in principle, have learned a different behavior (i.e., the "human" interpretation of the sequence)?

No, our extensions leave the Transformer and the LLM intact in terms of weights and activations, i.e., the LLM behaves precisely as in the discrete case when prompted with tokens each of length one. In other words, our extension works precisely as the discrete case for uniform token durations and each of length one.

Specifically, is it correct that when using, for instance, a duration of 0.25 for each of the four occurrences of "apple," the attention dedicated to them would be identical to the attention given to a single occurrence of "apple" in a standard Transformer? No, that is incorrect: generating the output for each “apple” token after the first one involves attending the previous “apple” tokens, resulting in a different output sequence. Moreover, each apple token has a different positional embedding (e.g. $t$ for the first “apple” token, $t + 0.25$ for the second, $t + 0.5$ for the third, $t + 0.75$ for the fourth, and $t + 1$ for the following token).

In that case, the manipulation in that experiment is just a different way to acknowledge the fact that LMs are fed continuous vectors and not discrete symbols. We agree that our work theoretically frames the math to build continuous Transformers, i.e., Transformers that handle (continuous) functions. On the other hand, we observe that this extension, for which the Transformers behave for inputs of canonical length, precisely as in the discrete case, evidences how these models embed a continuous notion of semantics in input and output space. These observations force us to rethink how LLMs treat language, as it is hard to reconcile with the standard theories on language (i.e., phonemes in linguistics are inherently discrete, and there are nearly no attempts at modelling them as continuous). At the end of the paper, we provide some intriguing open problems to explore as future works, including cutting the cost of LLMs pre-training with input templates and interpolation.

However, these findings are not novel, and the paper lacks sufficient references to a substantial body of prior work. The continuous representation of distinct symbolic linguistic processes dates back to the creation of dense vectors via SVD of PMI matrices.

The reviewer correctly points out that semantic interpolation is a well-known phenomenon in word embeddings (e.g., connecting Word2Vec with PMI, SVD, etc.). Our work extends that corpus by showing that, theoretically and practically, state-of-the-art LLMs embed a continuous notion of semantics in output space. To make an analogy, the famous “king-queen ≈ man-woman” linear behaviour observed in embedding space is preserved in output space when we prompt a model with a sentence that elicits such an analogy. We reported several examples of this behaviour in the Experimental Section of the paper and the Appendix. In other words, LLMs treat semantics as a continuous spectrum in the output space. LLMs transform input sentences via a long series of non-linear transformations: the above-mentioned results are novel and shed light on the structural differences between human and machine language.

Additionally, in interpretability research, there has been extensive work on the continuous representation of symbolic properties, such as linear concept erasure, which identifies specific manipulations in the latent space to isolate particular concepts.

We thank the reviewer for pointing out works in interpretability that look at the problem of erasing concepts in embedding space. Our work is complementary in the sense that it can be used to theoretically and practically find regions in the embedding space associated with a particular concept and eventually erase them. Without acknowledging and formally proving that LLMs treat semantic concepts as a continuum, erasure cannot provide guarantees if applied to single vectors.

That is not true. Many works point out to the existence of interpretable continuous directions in the representation space of LMs, beyonf neurons. E.g. [1][2][3].

[1] Arditi, Andy, et al. "Refusal in language models is mediated by a single direction." arXiv preprint arXiv:2406.11717 (2024).

[2] Ravfogel, Shauli, et al. "Linear adversarial concept erasure." International Conference on Machine Learning. PMLR, 2022.

[3] Todd, Eric, et al. "Function Vectors in Large Language Models." International Conference on Learning Representations. ICLR, 2024.