Continuity and Isolation Lead to Doubts or Dilemmas in Large Language Models

We thank the reviewer for carefully reviewing our work and the comments raised. Below, we provide answers to the weaknesses and questions posed by the reviewer.

Weaknesses rebuttals:

• (Weakness 1) My issue with this theoretical idea is that it is too much of a corner case. The authors claim that the lack of these abilities “severely constrains their (even mild) intelligent behavior”, which is a bit unsubstantiated. It is also bizarre to assume intelligence depends on the learnability of infinite sequences. I think the evaluation examinations are too tangential to their actual use cases. Pragmatically, few give infinite sequences to transformers, but just missing this aspect of behaving properly with respect to infinite or very long sequences does not disqualify a transformer from being intelligent. The paper lacks an argument or a demonstration on why this can be a concerning characteristic of the transformer.

We note that Theorem 1 (our main result) is actually formulated for finite sequences. The conclusions we derive from it for infinite sequences are meant to illustrate a concrete limitation to perform inductive reasoning, understood as the ability to derive general rules and principles from the presented information (an ability usually considered as a key aspect of intelligent behavior). The limitation for infinite sequences means that the transformer will never be able to understand the rule that generates the sequence, regardless of how long the prompt is. To substantiate even more our claims, we have performed an additional experiment on a novel natural language task, analogous to the Code Syntax Verification from our paper.

The new dataset consists of 56 examples derived from the "babi-nli" dataset. Each instance includes a premise, a hypothesis, and a label (0 or 1). To meet theoretical requirements, we lengthen the premise using neutral text from Wikipedia, creating a "Long-Premise" that preserves the original label and constitutes the alpha-sequence. To create the corresponding "beta-sequence" we modified a single key word at the beginning making sure that it entails a change to the original label.

For example, Premise Alpha (Label=True): Yann is hungry. Jason is bored. Antoine is hungry. Yann went back to the kitchen. Yann picked up the apple there....

Premise Beta (Label=False): Yann is hungry. Jason is bored. Antoine is hungry. Yann went back to the kitchen. Yann picked up the pear there....

Hypothesis: Yann picked the apple because she was hungry.

Considering that Meta-Llama-3-8B-Instruct provided the best results in the Code Syntax Verification task, we tested this model on the new NLI task in two scenarios with different input lengths. Namely ~1000 tokens and ~5000 tokens. The results were as follows:

Sensitivity: 25% (about 1000 tokens), 14% (about 5000 tokens) Accuracy: 62% (about 1000 tokens), 57% (about 5000 tokens)

As we can see from the results, even in this simple natural language inference task, the sensitivity remains fairly low. We acknowledge that this is an important practical application of Theorem 1. Sensitivity to small changes in a long text is critical for some real-life applications like analyzing legal documents.

• (Weakness 2) The title is not appropriate to its content, as it is unclear what these doubts and dilemmas refer to, the wording deviates from the main findings of the experiments.

The term “doubts“ refers to the lack of certainty for the next token prediction, while “dilemmas” means that if two prompts are too similar then the transformer can make the right prediction with certainty for at most one of them (in a sense, “it has to choose”). As explained in the paper and tested in the experiments, these two features come from the continuity phenomenon that we study. This is discussed in Section 5 (Conclusion - Doubts and Dilemmas). The referee is correct in that we should explain this earlier in the paper, and we will do so in the CR-version.

• (Question 1) “Line 89, what is CPE transformer?”

CPE is first introduced in line 43 and stands for compact positional encoding. A formal definition of CPE is later given in line 174. The CR-version will formally define CPE when it first appears in the text.

• (Question 2) What is the accuracy of the model in the syntax prediction error detection task?

The accuracies for the different models are 52% (gemma-3-4b-it), 54% (gemma-3-12b-it), 49% (phi-4) and 60% (Meta-Llama-3-8B-Instruct). We will add this information to the CR-version.

• (Question 3) Is the model sensitivity dependent on sequence length?

Theorem 1 imposes a universal relationship between epsilon and delta that does not depend on the length of the sequence. However, if for instance we are perturbing a single token (as in the syntax experiment), then the Normalized Hamming distance between the two prompts is 1/n where n is their length. Thus, for larger prompts the resulting perturbation is smaller, and therefore more likely to show non-sensitivity. We have in fact tested this. In Appendix B.5, Figure 10, we present the results of the Code Syntax Verification task on a series of other models using prompts with around 300 tokens (as opposed to the results in the main body of the paper which stemmed from inputs of more than 1000 tokens) and observe improved sensitivity.

• (Question 4) I guess figure 2’s emphasis is on the left figure, do we know in closer detail, the message is if there is a small number/proportion of differing signals, then the prediction results are insensitive, and the right plot suggests that the prediction results are sensitive when there is a substantial differing portion of signals, but you cannot see very well from the left side of figure 2. (How come there are still some models that are insensitive to a big portion of differing signals?)

Observing insensitivity even for large perturbations is permitted by Theorem 1, although not guaranteed in all cases. Our results guarantee insensitivity for sufficiently small perturbations.

• (Question 5) “How does the sensitivity result depend on the sequence length? I know it is 190 in the experiment, but what happens for sequence length in the common token range?”

Our theoretical results predict that non-sensitivity is more likely to appear with longer sequences. Indeed, as illustrated in Appendix B.5 (Figure 10), on average we observe an improvement in sensitivity when the sequence length is shorter. Moreover, the novel natural language task described in weakness 1 also reveals this phenomenon.

We thank the reviewer for the time invested in carefully reviewing our work and the valuable comments raised during this process. Below, we provide answers to the questions posed by the reviewer.

• (Question 1) It has been shown that hybrid models between Mamba and Transformers work better than Transformers for long context (https://github.com/NVIDIA/RULER). Do you have any intuition of what could explain these observations? Would your work allow you to shed some lights on that matter? (Maybe Mamba models suffers a representation collapse at a lower speed).

As Merrill [1] has shown, simple state space models as RNNs can recognize any regular language. In particular, there exists a simple RNN which on any long sequence input like “0000…0” outputs 0 but on a sequence like “1000…0” outputs 1. At the same time, by Theorem 1, Transformers with compact positional encoding (such as RoPE) cannot do that. Theorem 1 can therefore be seen as a separation result between the capabilities of these two architectures. In turn, this suggests an explanation why incorporating SSM modules into a transformer architecture (as in Jamba) may in principle result in models overcoming representational collapse.

• (Question 2) Does your study make you bullish on approaches such as https://arxiv.org/pdf/2501.19399?

This is also an interesting point. We have in fact analysed it in Section B.4 of the Appendix where we tested the effect of replacing softmax in the attention layers by the so-called scalable softmax, which is a way to incorporate non-compact components. The resulting model therefore escapes from the framework of our theory, and is thus not subordinated to the constraints imposed by our results. And indeed, as can be seen from Figure 8, our experiment shows a significant improvement in sensitivity. We acknowledge that this observation is absent from the main body of the paper. We will correct this in the revised version.

We thank the reviewer for the careful read of our manuscript and the valuable comments raised. The reviewer is concerned about a better positioning of our work with respect to previous work, but also the scope of our findings. Below, we provide a point-by-point reply to the weaknesses and questions raised by the reviewer.

• (Weakness 1) The paper appears to be poorly positioned. While the title suggests a focus on large language models (LLMs), the main content primarily discusses Transformers. Although LLMs are typically built on Transformer architectures, the two are not synonymous, and this distinction is not clearly addressed.

We thank the reviewer for this remark. Our camera-ready version will position the scope of our paper taking into account this comment, and we will make sure to isolate our claims to transformer-based LLMs from the introduction onwards.

• (Weakness 2) Several prior works [1,2,3] have explored the fundamental limitations of the general Transformer architecture, but this paper does not sufficiently engage with or contextualize those contributions.

We thank the reviewer for pointing out these references. To improve the positioning of the paper, we will add the discussion of these references in the related work section.

• (Weakness 3) The findings of this paper are limited to pattern discovery in purely numerical sequences. It remains unclear whether these results can be generalized to natural language. In most cases, natural language understanding does not rely on the kinds of patterns examined in the paper. Moreover, natural language carries significantly richer semantic content than the simplified tasks considered here.

We note that Theorem 1 (our main result) is agnostic to the nature of the input and thus applies equally well to natural language or any other kind of input. The consequences we present for pattern recognition over numerical sequences are only meant as a concrete illustration of the limitations implied by Theorem 1 on inductive reasoning, understood as the ability to derive general rules and principles from the presented information (and usually considered as a key aspect for human intelligence). We also note that besides numerical sequences, section 3.2 of our paper presents experiments on a syntax verification task that requires both advanced semantic and complex syntactic structure understanding. Nonetheless, to address the concern raised by the reviewer, we have performed an additional experiment on a novel natural language task, analogous to the Code Syntax Verification task.

The new dataset consists of 56 examples derived from the "babi-nli" dataset. Each instance includes a premise, a hypothesis, and a label (0 or 1). To meet theoretical requirements, we lengthen the premise using neutral text from Wikipedia, creating a "Long-Premise" that preserves the original label and constitutes the alpha-sequence. To create the corresponding "beta-sequence" we modified a single key word at the beginning making sure that it entails a change to the original label.

For example, Premise Alpha (Label=True): Yann is hungry. Jason is bored. Antoine is hungry. Yann went back to the kitchen. Yann picked up the apple there....

Premise Beta (Label=False): Yann is hungry. Jason is bored. Antoine is hungry. Yann went back to the kitchen. Yann picked up the pear there....

Hypothesis: Yann picked the apple because she was hungry.

Considering that Meta-Llama-3-8B-Instruct provided the best results in the Code Syntax Verification task, we tested this model on the new NLI task in two scenarios with different input lengths. Namely ~1000 tokens and ~5000 tokens. The results were as follows:

Sensitivity: 25% (about 1000 tokens), 14% (about 5000 tokens)

Accuracy: 62% (about 1000 tokens), 57% (about 5000 tokens)

As we can see from the results, even in this simple natural language inference task, the sensitivity remains fairly low. We acknowledge that this is an important practical application of Theorem 1. Sensitivity to small changes in a long text is critical for some real-life applications like analyzing legal documents.

• (Question 1) Can similar findings be applied to RNNs? Additionally, could you suggest possible architectural designs that might address the limitations identified in this paper?

This is an interesting question. The Continuity phenomenon does not apply to RNNs. Indeed, Merrill [1] has shown that simple RNNs can recognize any regular language. In particular, there exists a simple RNN which on any long sequence input like “0000…0” outputs 0 but on a sequence like “1000…0” outputs 1. At the same time, by Theorem 1, Transformers with compact positional encoding (such as RoPE) cannot do that.

Theorem 1 can therefore be seen as a separation result between the capabilities of these two architectures. In turn, this suggests that suitably incorporating state space models modules into a transformer architecture (as in Jamba) may help to overcome the limitations imposed by continuity and isolation. Another possibility to overcome these limitations is to allow non-compact components. This is in fact illustrated in Appendix B.4 of our paper, where we tested the effect of replacing softmax in the attention layers by the so-called scalable softmax, which is a version of softmax that explicitly incorporates a term that grows logarithmically with the size of the input, therefore introducing non-compactness. As can be seen from Figure 8, our results unequivocally show that replacing standard softmax by scalable softmax (while leaving everything else unchanged) leads to a significant improvement in sensitivity. Finally, another possibility to overcome continuity limitations is to allow non-compact positional encodings.

[1] William Merrill. 2019. Sequential neural networks as automata.

We thank the reviewer for the positive appraisal of our work and greatly appreciate the time invested in reviewing it. Below, we reply to the three weaknesses raised by the reviewer, followed by replies to the questions raised.

• (Weakness 1) The theorems are all claims for infinite sequences. Hence, it is difficult to judge how applicable they are practically beyond the illustrated synthetic examples. Is it possible to show more convincing practical use cases such as the code-syntax related.

We note that Theorem 1 (our main result) is in fact formulated for finite sequences. The conclusions for infinite sequences are presented only to illustrate clear limitations on a task requiring inductive reasoning, namely, the ability to derive general rules and principles from the presented information (an ability considered as a key aspect of intelligence behavior). In addition to the code syntax verification task, and to address the reviewers comment, we have tested the corollary of Theorem 1 on a new natural language task.

We evaluate LLMs on a natural language inference (NLI) task analogous to the Code Syntax Verification task. The new dataset consists of 56 examples derived from the "babi-nli" dataset. Each instance includes a premise, a hypothesis, and a label (0 or 1). To meet theoretical requirements, we lengthen the premise using neutral text from Wikipedia, creating a "Long-Premise" that preserves the original label and constitutes the alpha-sequence. To create the corresponding "beta-sequence" we modified a single key word at the beginning making sure that it entails a change to the original label.

For example, Premise Alpha (Label=True): Yann is hungry. Jason is bored. Antoine is hungry. Yann went back to the kitchen. Yann picked up the apple there....

Premise Beta (Label=False): Yann is hungry. Jason is bored. Antoine is hungry. Yann went back to the kitchen. Yann picked up the pear there....

Hypothesis: Yann picked the apple because she was hungry.

Considering that Meta-Llama-3-8B-Instruct provided the best results in the Code Syntax Verification task, we tested this model on the new NLI task in two scenarios with different input lengths. Namely ~1000 tokens and ~5000 tokens. The results were as follows:

Sensitivity: 25% (about 1000 tokens), 14% (about 5000 tokens)

Accuracy: 62% (about 1000 tokens), 57% (about 5000 tokens)

As we can see from the results, even in this simple natural language inference task, the sensitivity remains fairly low. We acknowledge that this is an important practical application of Theorem 1. If the reviewer agrees we can add these findings in the revised version.

• (Weakness 2) For Fig. 3 the pattern does not seem to follow same as in Fig. 2 for continuity. Please explain if I am misunderstanding the figures.

Figures 2 and 3 are plots of different things, it is therefore normal that the patterns are different. In Fig. 2 the x-axis corresponds to the percentage of tokens that were randomly changed in order to produce the perturbed sequence, while in Figure 3 this perturbation size is fixed. The purpose of Fig. 2 is to show the number of times (out of 100) a given perturbation size produced a change in the output. In contrast, in Fig. 3 the plot is meant to compare the output probability of the most likely token given the original sequence as input (alpha) against the output probability of the same token, but given instead the perturbed sequence (beta).

• (Weakness 3) As also mentioned in the limitations, the analysis lacks consideration of chain-of-thought reasoning generated by recent reasoning models. Is it possible that these reasoning traces allow different output after reasoning, hence, not be limited by the results in the theorem. Can we check it empirically on some of the synthetic datasets using some reasoning models?

This is an interesting point, which we in fact discuss in Section B.6 of the Appendix. In particular, we tested Open AI’s o4-mini and o3-mini models on the syntax verification task. Strikingly, we observe that even on our very simple task, these models still display a level of error greater than 20%. This suggests that while CoT may slow down the effect of continuity, it may not completely eliminate it. We acknowledge that these results are omitted from the main body of the paper. We will make sure to correct this in the revised version.

• (Question 1) Line 223: "Middle positions result in lower sensitivity, and early tokens yield almost no sensitivity at all" Q: does the theory explain this observation?

No, our theoretical results do not account for differences in sensitivity associated with the positions of the perturbations (beginning, middle or end).

• (Question 2) Line 196: "same last token" Why is this condition in theorem 1 important, please explain intuitively.

Intuitively, because one can build a transformer that always outputs the last token (ignoring previous ones). For that transformer, changing the last token changes the output regardless of the input length. Hence, the condition about the last token is strictly necessary for the theorem 1.