L38 states explicitly that observability is one of the pillars in the analysis of dynamical models. Since LLMs are a particular kind of dynamical model, we do not delve on the more general topic and focus on LLMs. We are cognizant that the ICLR community is vast and diverse, that most CS students do not have background in Systems and Controls, and that one can be an expert in the field and yet less experienced in a specific area. While our paper is only focused on LLMs and does not require extensive background in systems and controls, the interested reader who wishes to understand the broader concept can either consult the references cited (e.g. Herman-Krener), or simply query "observability" in Wikipedia, which provides more than enough background for our paper. None of these options require "investing many days", and we believe that a reviewer without expertise in the area should not render a strong rejection of our paper without at least familiarizing themselves with the basic prerequisites.

Furthermore, it should be clear that the concept of observability cannot be reduced to a simple definition (lest we would have provided it) since it depends on the class of models. Our entire paper is devoted to analyzing this property for LLMs.

As for the attempt to define notions commonly used in psychology and cognitive science, whether we like it or not these words are being used to describe LLMs, not just in the popular discourse, but even in technical and policy writings. So, we can pretend that the domain of psychology is disjoint from AI and leave it to others to use such terms without a formal definition, even if the concept is used to characterize (or regulate) the behavior of LLMs which are engineered artifacts, every aspect of which is determined and measurable. Or, we can try to define concepts familiar to most but often attributed to LLMs without definitions, and characterize them in terms of measurable properties of trained LLMs. It seems to us that an attempt to define these notions in relation to LLMs is not only useful, but actually necessary to sustain a scientific discourse in relation to artifacts that, unlike the human brain, we build and can directly observe. 

We appreciate the general suggestions on writing style and simplicity: We have made an attempt to follow them as much as possible in a revised version we uploaded.

We thank the reviewer for their feedback.

I do not understand the significance of some of the key contributions, such as measuring the cardinality of indistinguishable sets.

The cardinality of indistinguishable sets is a measure of observability of the model: If observable, each indistinguishable set is a singleton. Empirical measurement of the cardinality or volume of these sets provides a coarse empirical view that supports or invalidates the analysis, but is not a replacement of it.

In Section 4.4, the authors do not apply their method to established adversarial attack benchmarks (e.g. AdvBench), nor do they provide comparisons to relevant baselines (e.g. Sleeper Agents). This makes it difficult to judge how good their method is compared to other approaches and limits the significance from an empirical alignment perspective.

The unobservable part of the state space is — by definition — not reachable, whether by ordinary or adversarial inputs. So we already know that if a model is not observable, there is no input that can distinguish two states, and there is no need for costly experimental search over prompts. The vulnerability of models towards the Trojan Horse attacks we study in our work is simply an illustration of a possible consequence from having unobservable models.

We thank the reviewer for their feedback.

the authors formulate LLMs as a dynamical system in Sec2, however, they implicitly use a assumption of linear memory space (in x(t+1), the first token of the last state x_1(t) is not included) in the formulation. However, they neither provide references on this assumption nor giving any reasons for the formulation. Often, in theoretical analysis, for transformer models a log memory space [1] is often used, the authors should provide more explanations on their formulation.

Any LLM can only memorize a maximum number of tokens. In the second line of Section 2 we define that number as being c, the length of the context. This sequence is represented by x_i with i ranging from 1 to c. When a new token is generated, one token must be forgotten and equation (1) describes the update process that memorizes the new token by: 1) forgetting the token in location 1; 2) copying the token in location i+1 to location i; 3) copying the new token to location c. The model directly follows from an LLM being able to memorize at most c tokens and no assumptions of linear vs logarithmic memory were made.

They take y(t) which seems to be the hidden state before the last MLP layer (they haven't offer a strict claim, as seen in the next point) as the so-called trajectory in mental space without giving any reasons.

 $y(t)$ represents the output of the model before being projected into the space of tokens, and is formally defined in (2) by the equality $y(t)=\phi(\mathbf{x}(t))$ where $\phi$ is defined in the first line of the paragraph titled "Large Language Models" in Section 2. There is no claim to be made, since that is just a definition, and we do not assume a particular architectural configuration (like the last layer being an MLP) since our results are more general. The bulk of our introduction is also dedicated precisely to motivating this terminology, since this corresponds to the model's hidden state before projection to verbal space.

Following the point above, the definition of feelings provided in line 240 is not suitable, as y(t) is not rigorously claimed.

We are unsure what the reviewer means by "$y(t)$ is not rigorously claimed". Each trajectory of the state corresponds to a trajectory of the output, but conversely for each output trajectory there could be infinitely many states. Establishing whether this is the case is the purpose of observability analysis. 

We provide a formal definition within which all terms are well-defined. It is, however, a valid opinion of the reviewer if they disagree with the definition, for which we hope they can detail what aspects our definition fails to capture or suggest an alternative.

In L222, they claim that for each $y$, there are countably many expressions $x' \neq x$ that yield the same $y$. However, the citation they offer is [2] which is rejected by ICLR in 2022 and thus the statement is not confirmed.

The pre-image of a point under $\phi$ being a countably infinite set is used to motivate Eq. (2). However, Eq. (2) is well defined independently of the cardinality of the pre-image of $\phi$. Hence, this discussion seems irrelevant to the merits of the paper.

In line 351, they claim that they randomly sample p, therefore figure 1 doesn't plot Qτ(p), but EpQτ(p). Furthermore, the authors haven't offer references or reasons for why is the cardinality calculated as an expectation is suitable.

Almost all metrics in ML are computed via expectation over a dataset. $Q_\tau$  is of no exception, computed as an expectation over the SST-2 dataset.

Lack of definition for certain concepts in writing: (1) What is \phi, \pi in LLM/ transformers? Does \pi refers to the layers except for the last mlp? (2) The authors haven't provided the definition of g in Section 4 (3) The authors haven't provided a mathematical definition for reproducibility in theorem one

 All these concepts are duly defined at the beginning of Sec 2. 

 (1) $\phi$ is defined in L163-164, as a map from sequences of tokens to an LLM output, and $\pi$ is defined in L166-167 and further elaborated on in L168-171 is a projection from the LLM output to a single token (the greedy sampler $\pi_{max}$ defined in L169 is provided as an example).

 (2) $g$ is defined in Eqn (4), and on L279-280 as the feedback function.

 (3) We did not use the term "reproducibility" in any part of our paper. If the reviewer is referring to "reconstructibility" (of state trajectories), we note that its usage is standard in the analysis of dynamical systems, but even for the reader who is not versed with that literature, the literal definition (a state trajectory is “reconstructible” if it can be reconstructed) is sufficient.

We thank the reviewer for their detailed feedback and suggestions.

The notation in the paper is quite dense. The paper takes a humanistic presentation of LLMs; for example, the authors take the time to formalize mental state, visualization, verbalizations, control space, mental space, "feelings", and "sensation", but also specifies that it is not claiming to ascribe humanistic thought to LLMs. I’m concerned that these definitions do not contribute high relevance to the main contribution while requiring the reader to keep track of potentially unintuitive definitions.

Our paper is the first to study observability of LLMs interpreted as dynamical models. However, LLMs have peculiarities (e.g., tokenized input/output spaces) that make them unlike dynamical models that are typically analyzed in systems and controls. So, formalizing the notion of observability for LLMs is not just relevant but necessary for analysis. A "humanistic presentation of LLMs" is precisely what we stand to contrast: Unlike others who borrow suggestive terms from cognitive psychology to describe the behavior of LLMs informally, we map each term ('mental state,' 'verbalization,' 'thought,' and yes, 'feeling') to specific components of LLMs, so that each term is uniquely defined and corresponding to a measurable characteristic of a trained LLM. Since LLMs are engineered artifacts, each term used to describe them should map to a specific and measurable quantity, something that is not possible in biological systems.

Elaborate on the "why" for each of the parts of the evaluation: why the models selected are valid? what does each part of the eval seek to test? The theoretical results seem intuitive, but it would be helpful to add intuitive grounding examples. On the whole, do the theoretical results hold up in practice

This work has no precursors, there isn’t an established evaluation protocol. Therefore, our main goal is to validate the definitions (what it means for a model to be "observable", and what changes make a model flip from observable to unobservable). Once that is accepted, the claims follow analytically, so technically there is no need for experimental assessment. This may sound like anathema for an empirical discipline like deep learning, and it is easy to argue that ultimately empirical tests are the only convincing evidence. But here the point is to introduce ways to measure a concept (observability) in an LLM, and once the method is viable, there is no "better or worse" observability: It is a binary characteristic, which can be deduced from the structure of the model, or its specifications, rather than by testing on this or that dataset.

It would be helpful to expand on why the choice of the four models, and organize the empirical validation section by the specific research question each experiment sought to answer.

Type 1 and Type 2 models are actually used in modern LLMs, where system prompts are either "verbal" instructions or prefix vectors (Li & Liang, 2021) respectively. While, to the best of our knowledge, Type 3 and 4 models are not currently used in modern LLMs, such fading memory systems are motivated by those studied in systems and controls. We thank the reviewer for their suggestion and will elaborate on them in the paper.

Why was the Stanford Sentiment Treebank dataset chosen? Can you provide further detail in the experimental setup regarding what the trajectories, system prompts, and outputs were? Are 100 different choices for input sufficient coverage? Overall, I found the results section extremely hard to follow. It would be helpful to highlight takeaways and what each individual experiment sought to examine or test.

We wanted a dataset that would be recognized as containing "semantically meaningful sentences" by the community, of which we believe SST is an appropriate choice. We did not want to create our own dataset as it would be an additional confounder. Trajectories are simply sampled continuations from the prompt of length $\tau$. Appendix C.2. shows a qualitative example for the Type 3 model. The system prompt used is different for each model (Type 1/2/3/4), and are described in the beginning of Sec. 4. In general, they are either discrete tokens (Type 1) or vectors (Types 2/3/4) that may be updated as the model evolves (Types 3/4).