Taming AI Bots: Controllability of Neural States in Large Language Models

Q: I am undecided about the significance of this work for one particular reason: Postulate 1. The authors are considering a very important problem that many people care about - if a paper can establish that real-world LLMs that we use are controllable, that would be an extremely important paper. The problem, in my mind, is that to reach this conclusion, the authors make a very big assumption about well-trained LLMs in Postulate 1. To their credit, the authors note this limitation, both in the main text and Appendix 1. Overall, I fear that the strength of the assumption in Postulate 1 could significantly limit the significance of this work.

A: Indeed, we acknowledge that Postulate 1 is a limitation of this paper. However, the assumption are actually not that strong: that current models are well-trained according to our definition can be tested by measuring perplexity, and qualitatively one would be hard-pressed to argue that at least some current LLMs generate sentences that are statistically distinguishable from human-generated training data, which presumably was written to convey some meaning. We could have made the claim seemingly rigorous by enumeration arguments in the style of old theory papers, so the conclusion would be formally proven. However, we are unsatisfied with this kind of argument which is why we state the proposition as a postulate. By doing so, we we hope to open a discussion, rather than close it, on this topic that we concur is an important one.

Q: I recognize that the primary contribution of this work is theoretical, rather than empirical, in nature. However, the authors do conduct experiments and present results in Appendix A. Even while keeping results in appendices, I would encourage the authors to at least provide a sentence or two in the main paper highlighting high-level trends of the experiment results.

A: Thank you, will do.

Q: I wish to emphasize again that I traditional study more empirical AI methods and analysis. Thus, if other reviewers find that this work would contribute to the more theory-focused community, I am happy to change my mind.

A: We view ours as more of a position paper, since it opens -- rather than closing -- a key question. We view its contribution as framing the problem in a specific technical language, that of control theory, which has a rich history that may yield insights for further analysis.

Q: In Section 3, the authors describe how, "For a sufficiently high temperature parameter ..." random sequences can be generated from any start token. This is true for any , right (barring numerical precision errors from very small numbers)? I do not see why one would need a larger value of for this theoretical claim (although I recognize that in practice, higher would "spread out" faster to cover more random sequences).

A: Yes, we could just say $T>0$. Thanks for pointing that out.

Q: As stated earlier, my main concern about this paper is Postulate 1. What evidence or arguments do the authors have supporting this assumption?

A: The fact that there exist LLMs that are well-trained according to our definition should not be very controversial, as one could be convinced by simply interacting with some of the best models, or measuring their perplexity on any benchmark validation set. There are also quantitative empirical studies establishing that the best models generate sentences that are indistinguishable from human-generated, with high-frequency/probability. As for the rest of the claim, that follows from the complexity of the trained map coupled with the finiteness of the context. Also empirical evidence is the practice of jailbreaking and adversarial prompting, as well as the massive effort that goes into preventing such in the area of Responsible AI. However, we recognize that even taking all this into account still does not make a proof. In our view, though, the framing of the problem in the language of controls still would be beneficial to the community.

Q: It's unclear what the contribution of the paper is. The related work section does not connect this paper to specific prior work (only citing two survey papers)

A: The contribution is to have framed the question of controllability of LLMs in the language of control systems: Controllability of dynamical models operating in the quotient space of trajectories determined by the model itself. As for prior work, we are unaware of any work that formalized the problem of controllability of LLMs. If the reviewer is, we would appreciate pointers so we can expand the discussion of relevant related work. We are also unaware of anyone having formalized meanings as pre-images of the output of trained LLMs, which is another contribution of our paper.

Q: Then, most of the paper is spent discussing preliminaries and introducing notation and definitions.

A: Indeed. We believe it is important to spend some effort on the definitions given the level of confusion and disagreement surrounding these concepts. For example, some in the community claim that LLMs are "uncontrollable", without giving any definition, and there is often quoted literature claiming that LLMS "cannot in principle represent meanings", based on a definition of meaning in terms of "intentionality" that is, however, left undefined, leading to tautological arguments. So, we think that within the plethora of empirical papers, a few papers focusing on definitions may not be such a bad thing.

Q: The first use of the proposed definitions to make a non-trivial claim is in Section 5 with Postulate 1, but this claim is not justified.

A: If the reviewer means it is not proven, that is indeed the case. We could have "proven" it by arguing that the context is finite so one can always find the prompt by enumeration, but this would not be practical, so we do not frame the claim as a theorem but rather a postulate. If by "justified" the reviewer means that it not a worthwhile effort to try to study the controllability of LLMs analytically, that would be a judgment call we would disagree with, and stress that of course our analysis is not an alternative but a complement to the necessary empirical assessments, which are currently dominating the literature.

Q: Theorem 1 in Section 5 seems to follow directly from the proposed definition of controllability. Theorem 2 is presented with no intuition and the proof in the appendix is only for a special case. It's also not explained how Theorem 2 justifies the main conclusion: that "a prompt engineer aided by enough time and memory can force an LLM to output an arbitrary sequence of ℓ tokens."

A: The notion of controllability requires verifying the existence of a sequence of inputs for the dynamics (11)-(13) which is a non-trivial endeavor. We further note that inputs appear in equation (13) whereas the assumptions in Theorem 1 relate to the function in equation (12).

In the revised version we will allocate more space in the appendix to provide an intuitive description of Theorems 1 and 2.

As we wrote in the proof of Theorem 2, a general proof would require heavy notation thus rendering it even less accessible to a general audience. Anyone versed in nonlinear controls can verify that the arguments applies in general, whereas readers not familiar with it can more easily follow the special case to gain insight.

Q: Last paragraph of introduction: “we hope to encourage the design of actual new control mechanism” --> It looks like the authors are not aware of the Rectification method, which is indeed a formal control paradigm for avoidance any definable “meaning,” as in the terminology of this paper. See the point bellow. Let me first emphasize that I am aware of the differences between these two works (no need to tell me they say X while we say Y). My main point is to help you broaden your research and make the current paper be more comprehensive and useful for the readers, also this is aligned with your last paragraph of the introduction…

A: Thanks for the useful reference and the thorough description of that work, which we were not aware of since it appeared on ArXiv after our work was completed. We will add discussion as it certainly sounds pertinent, thank you for the pointer and the explanation.

Q: Section Preliminaries, first part --> An LLM in its standard form provides a simplex over token space, not an embedding + a metric. How can an LLM be a discriminant?

A: An LLM is trained so that the vector of logits, or the corresponding soft-max, approximates the (log-)posterior of the next token, which is the optimal (Bayesian) discriminant. So, if you view the LLM as a map from an input sequence of discrete tokens to the softmax vector, it is indeed a discriminant, and since it is trained with the (exponential of the) inner product of the embedding with the one-hot encoding of the next token, the inner product (metric) is built into the embedding space by design of the training process.

Q: Section Preliminaries, second part --> The definition of equivalence is quite confusing. It looks like that equivalence requires both discriminant and a set of classes, no matter how they are defined (does it?). However, the presented definition only uses discriminant, which seems incomplete. In that case, x^1 phi ~ x^2 is ambiguous.

A: The definition requires either a discriminant or a set of classes, since one determines the other. If given a discriminant, one can define partitions using simple order relations or rules (e.g. thresholds), and given a set of classes one can define a discriminant as a (piecewise constant) function.

Q: Definition of equivalence seems to be insufficient and lacks important components. Definition of meaning seems to defeat the whole purpose of this paper, as it allows for any gibberish/random sequence of tokens to still induce a meaning and possibly a set of many other gibberish sequences to be in their equivalent class.

A: Any model trained on natural sentences determines a partition of the space of input sentences, including gibberish, indeed. This is why we restrict the model to operate in the quotient space (10), so input gibberish is excluded from the analysis. In theory, one can of course feed gibberish to a model, but that does not have any meaning in our definition since it is outside $\sigma({\cal I})$. In practice, one can easily exclude gibberish sequences by filtering the input, as done to avoid jailbreaking.

Q: How to find a discriminant for meaning is left out as the authors explicitly assume that “the mechanism is provided by human annotators and other providers of training data.”

A: It is described in in multiple passages (although we can make it clearer) that the discriminant is the trained LLM itself: That means that it is found by training, which is based on human annotations and human-generated utterances. For example, starting from the abstract: Page 1: "The LLM maps complete sequences to a vector space [...] that vector space can be coopted to represent meanings during fine-tuning" Page 2: "an LLM is a map [...] from complete sentences to meanings" Page 4: Note that the notation used for the discriminant, $\phi$ (Definition 1), is on purpose the same used for the (parametrized) logit vector $\phi_w$ of the trained model (1) Page 5: "by feeding back complete sentences to the input [of the model] and training [...] the same $\phi_w$ as a sentence-level discriminant to attribute meaning to synthesized sentences" Page 6: Claim 1 states that a well-trained model $\phi_{\hat w}$ generates meaningful sentences (and therefore meanings). Page 8: establishes that the domain of meaningful sentences is the range of the model $F_w$, again emphasizing that meanings are imposed by the trained model.

Having said that, we agree that this point could be expressed more clearly at the beginning: the trained model depends on human-generated content (during pre-training) and human annotation (during fine-tuning), so the origin of meanings is latent in the human content providers and annotators, and the mechanism by which they are transferred to the trained model is the training process. We will try to simplify this explanation and place it at the top of the introduction to avoid any confusion.

Q: While the authors emphasize in the introduction that such information can be used in the LLM training without external reward model: “This observation shows that sentence-level annotations can be incorporated directly into the trained model without the need for any external reward model nor external policy model, simply by sentence-level feedback,” I do not see the advantage of this approach over using the very same data to train a reward model and use that either during the training (as in RLHF) or as an augmentation (as in Rectification method), the latter indeed provides quite strong theoretical guarantees. If this secondary goal is valid, the paper requires sufficient reasoning for why the presented approach is superior. If not, the presentation requires to change and reflect only the controllability analysis.

A: We do not claim that forgoing an external reward model is superior, just that it is sufficient. Indeed the reviewer is correct that, if one can afford it, direct augmentation or RLHF with an externally-trained reward model (which then becomes the origin of meanings) is also possible, and possibly better. All we are saying is that an external reward model is not necessary, because one can use the model itself -- as has been established in various alternatives to RLHF, such as Direct Preference Optimization (DPO). We were not aware of the "rectification method", as discussed below, since that appeared after our paper was completed, but we will include it as an alternative way of endowing the trained model with meanings.

Q: The characterization of meanings seems quite subjective.

A: Indeed, our definition of meaning is specific to a discriminant function $\phi(\cdot)$, which in the case of an LLM is implemented by the trained model $\phi_w(\cdot)$. Different models partition the data differently, although models trained on the same or similar data will likely yield similar partitions. We do not believe in a universal notion of meaning, and in any case our scope is limited to how meanings are represented by LLMs.

Q: The authors mentioned that their characterization of meanings is compatible with the deflationary theories in epistemology. It seems that this explanation itself does not justify why such a characterization is meaningful for studying the controllability of LLMs in general.

A: It is not essential that our definition aligns with any particular epistemological theory, but we note that, once mapped to LLMs, some theories boil down to representing meanings as equivalence classes of expressions, and then differ on their genesis. In our case, such classes are determined by the state of the trained model, which is what we want to control.

Q: The authors claim that their conditions are largely met by today’s LLMs. More justifications are needed. This also seems a hand-waving statement. "Largely met" means "not always met"?

A: Met by some, but not all, trained LLMs. LLMs by design represent a joint distribution on a sequence of tokens, but how well that approximates the empirical distribution of natural sentences is a matter of approximation rather than constraint satisfaction. In this context, by "largely" we mean that perplexity of trained models is low enough that generated sentences are indistinguishable from human generated ones. This may not be true of all models, so we use "largely" to indicate that the conditions are valid only for some models.

Q: Does Definition 1 assume some sort of underlying classifier such that the equivalent classes can be defined? What is that specific classifier? The authors mentioned "the mechanism is provided by human annotators and other providers of training data." This is very confusing. I don't understand what exactly this phi is.

A: In the definition, $\phi$ can be any discriminant vector. In a trained model, $\phi = \phi_w$ is the soft-max vector after supervised fine-tuning (and optionally RLHF). Since that vector is a function of the training data, both in pre-training and fine-tuning, and the training data is provided by human annotators and authors, ultimately the source of meanings is latent in the expressions, ranking, or scoring humans have provided for training.

Q: Equation (6) is very confusing. I mean, in Equation (4), y is sampled from the softmax operation. Then all of a sudden, it becomes an additive noise? It seems that the noise $n_t$ depends on $x_{1:t}$?

A: Indeed the fact that the same token can be represented in different ways can be confusing at first: Tokens can be represented as elements of a discrete dictionary after sampling, as in (4), or as discriminant vectors before sampling, as in (5). In the latter case, the effect of different selections in the (previous round of) sampling results in a perturbation of the embedding vector of the next token, which is represented in (6) as an additive perturbation since embedding vectors (logits) are elements of a vector space. In general, it is true that the ``noise'' $n_t$ is dependent on the previous history, $x_{1:t-1}$, but in complex ways that can be modeled statistically.

Q: Regarding Definition 3, an LLM is well-trained if theta is any small positive number?

A: Yes, although different users, or use cases, may have different standards on what constitutes a well-trained model, reflected in the perplexity thresholds that are considered acceptable.