We thank the reviewer for their feedback—we want to clarify that several comments appear to stem from misunderstandings of the problem setup and our experiments, which we clarify below.

Sum1. This investigation... through the lens of a prediction task, rather than by directly addressing probabilities.

Our study does address probabilities. We evaluate Hellinger distance between the LLM‑predicted next‑emission distribution and the ground‑truth HMM (Main Fig. 5 right; App. D Fig. 3).

Sum2. The paper showcases that LLMs can be used as efficient and capable predictors of the next token for data coming from a Markov process.

Our experiments are on HMM emissions, not simple Markov chains (Sec. 2.1; App. A). The distinction between next‑token prediction on sequences from a Markov chain versus on HMM emissions is non‑trivial. There is no existing theory that provides finite‑sample complexity or error bounds for next‑token prediction given a single finite trajectory of emissions from an HMM.

Sum3. Moreover…interesting real-world data, which is considered to come from a Markovian process.

Our real‑world animal data are commonly modeled with HMMs [1,2], but are not assumed to be Markovian.

W1: It's unclear... complex temporal data.

Unlike Baum‑Welch (BW), which uses non‑convex EM to infer HMM parameters, LLM ICL achieves near‑optimal prediction (Sec. 2.3). This suggests LLMs act like spectral methods—directly mapping sequences to next‑emission distributions. Empirically, LLMs behave similarly to spectral learning (reviewer oWyV W1 & W2). We provide theoretical guarantees for spectral learning (Sec. 3.3), explaining LLM performance trends (Sec. 3.1–3.2).

Evaluating LLMs on theoretically grounded tasks like HMMs is non‑trivial and novel. Even in the simpler case of fully observed Markov models, the performance evaluation of LLM’s or transformers has received significant attention from works published at top conferences [3-5]. Our results show LLMs outperform non‑convex BW and align with direct methods like spectral learning.

Finally, although LLMs are known to handle complex temporal sequences, the community remains interested in when and how ICL succeeds. This is why benchmarking LLMs on tasks with clear theoretical baselines—even relatively simple tasks like linear regression [6,7]—is considered valuable. Our work follows this philosophy.

W1.1: This appears... low-entropy transition matrices (where the most likely token has a high probability).

It is expected that low‑entropy transition matrices naturally produce high‑probability next emissions. However, our evaluation is not based on raw accuracy approaching 1, but rather on the gap to the theoretical optimum for each HMM (see Main Paper Fig. 3).

W1.2: LLM's memory bandwidth... ample "signal from the past" in finite sequences.

We acknowledge the reviewer’s observation that LLMs can leverage long‑range context through attention. For a fully observed Markov sequence, this indeed provides an advantage because induction heads could perform $n$-gram‑style prediction.

However, in our setting with hidden Markov models, the emissions themselves are not Markovian. Memorizing recent emissions alone is insufficient for optimal prediction. Leveraging longer context is statistically valid and necessary, as classical methods such as Baum‑Welch also utilize the entire observed sequence via the forward‑backward algorithm.

W2: The paper aims to address the “hidden” aspect of HMMs, but this is largely unaddressed in both the text and the simulations.

We would like to clarify that the “hidden” aspect of HMMs is central to our study. All of our numerical experiments are conducted on synthetic data generated from true HMMs, and real‑world datasets that can reasonably be modeled as HMMs [1,2].

W2.1: There's a lack of analysis on how emission probabilities or the loss from misclassifying observations to hidden states affect predictor results.

We do not assume LLMs explicitly classify observations into hidden states. In fact, our results suggest the opposite—LLMs ICL is qualitatively different from the Baum‑Welch algorithm (Sec. 3.2).

W2.2: Also it's unclear whether sequential information is truly critical, or if the LLM is simply better at picking states (which is often more important in practice).

Our experimental design evaluates performances across different mixing rates (Sec. 3.1). The fact that LLM performance varies with mixing rate (e.g., performing better on fast‑mixing HMMs) demonstrates that it is leveraging sequential structure. If sequential information were irrelevant, the mixing rate would have no impact on results.

W2.3. The benchmarked graphs ignored the more interesting Markov processes like jumping processes or cyclic processes which have unique properties.

We appreciate the reviewer’s observation that hidden Markov jump processes (HMJPs) and cyclic (periodic) chains exhibit dynamics that differ from the ergodic HMMs we focus on.

• HMJPs introduce an additional holding‑time random variable, which is application‑specific (e.g., queuing systems). Modeling HMJPs requires predictors to capture additional sojourn‑time distribution. Classical HMM baselines like Baum‑Welch and Viterbi do not directly apply to HMJPs without significant modifications.
• Cyclic chains' dynamics become more structured once the chain enters a cycle and the prediction space narrows. However, such processes are typically non‑stationary, which requires careful treatment for finite‑trajectory evaluation.

Our benchmark focuses on ergodic HMMs, which allows us to study sequential prediction in a theoretically tractable setting [8,9]. Extending our evaluation to HMJPs or cyclic processes is an exciting future direction. However, we expect that the relative performance trends among pre‑trained LLMs, Baum‑Welch, and spectral methods would remain consistent, as all predictors would be equally affected by the increased complexity.

W3: It would be more robust to learn a Markov model from the real data and then sample from that as a benchmark.

We respectfully disagree with the suggestion to first learn a (hidden) Markov model from the real‑world dataset and then benchmark on samples from that learned model.

Our study is specifically motivated by the real‑world scientific setting, where researchers observe a single finite trajectory and must make predictions or analyze patterns without knowing the true latent model. Learning a model from the data and then resampling would (1) create a simulation‑to‑real gap, (2) reduce the problem to a fully synthetic benchmark—already covered in the first part of our experiments, and (3) fail to test the LLM’s ability to handle the messiness and variability of real emission sequences directly.

Instead, we justify in the paper that the chosen dataset can reasonably be modeled as an HMM, and we evaluate LLMs directly using real-world data.

W4: The main paper doesn't contain enough details about the experimental setup, specifically regarding the emissions and their prior distributions.

The experimental details of initial state distributions $\pi$ are in Section 2.2 (uniform or deterministic). We did suggest generating the emission matrix is relatively straight-forward (Sec. 2.2), but we would love to explain how we construct $\textbf B$ here.

The code (hmm.py, 311–326) builds a sequence of stochastic transition matrices with increasing entropy. It starts with a deterministic matrix (zero entropy), then, for each target entropy up to the maximum, computes a row distribution with one dominant entry and the rest uniform, repeating it across rows to form a homogeneous matrix.

W5: Theorem 1 appears to be quite obvious for any learner, and it's unclear what makes an LLM particularly sensitive to these parameters.

Deriving sample‑complexity and error bounds for spectral learning on a single trajectory is non‑trivial due to output correlations. Standard mixing arguments are challenging because we observe only one realization rather than i.i.d. samples. We address this by proving that the single‑trajectory estimates of $\hat P_1, \hat P_2, \hat P_{3,n}$ in Algorithm 6 (Appendix F) converge to the corresponding multi‑trajectory (i.i.d.) estimates.

Regarding the LLM’s sensitivity to Theorem 1 parameters, we refer the reviewer to our responses to Q4 (reviewer 1stE) and W1&W2 (reviewer oWyV).

W6: In the context of Markov models... rather than always providing the full context.

For HMMs, evaluating on a single new observation is different from a sequence of observations. It’s necessary to infer the hidden state sequence using the entire observation sequence.

[1] Ashwood Z.C. et al. Mice alternate between discrete strategies during perceptual decision‑making. Nat. Neurosci. 25(2), 201‑212 (2022).

[2] McClintock B.T. et al. Uncovering ecological state dynamics with hidden Markov models. Ecol. Lett. 23(12), 1878‑1903 (2020).

[3] Edelman E. et al. The evolution of statistical induction heads: In‑context learning Markov chains. NeurIPS 37, 64273‑64311 (2024).

[4] Rajaraman N. et al. Transformers on Markov data: Constant depth suffices. NeurIPS 37, 137521‑137556 (2024).

[5] Rajaraman N., Jiao J., & Ramchandran K. An analysis of tokenization: Transformers under Markov data. NeurIPS 37, 62503‑62556 (2024).

[6] Li Y. et al. Transformers as algorithms: Generalization and stability in in‑context learning. ICML (2023).

[7] Garg S. et al. What can transformers learn in‑context? A case study of simple function classes. NeurIPS 35, 30583‑30598 (2022).

[8] Gassiat E., Cleynen A., & Robin S. Inference in finite‑state non‑parametric hidden Markov models. Stat. Comput. 26, 61‑71 (2016).

[9] Abraham K., Gassiat E., & Naulet Z. Fundamental limits for learning hidden Markov model parameters. IEEE Trans. Inf. Theory 69(3), 1777‑1794 (2022).

We will continue to address the specific RCs.

RC to Sum1: Sorry if I wasn't clear, but I believe this answer is not fair and escapes the need to address the point. My question was about the ICL “addressing probabilities,” while your reference is about the Viterbi algorithm. I hope this is an honest mistake.

From the context in Sum1, “...through the lens of a prediction task, rather than by directly addressing probabilities,” we initially interpreted the distinction between “prediction” and “probability” as argmax prediction versus predicting the full next‑emission probability distribution.

If this is not the intended meaning, we apologize for the misunderstanding. Based on your follow‑up, we now believe that “addressing probabilities” refers to recovering the underlying transition and emission matrices—that is, classical parameter recovery.

If our interpretation is now correct, we would like to clarify:

Our synthetic experiments show that LLM ICL approaches the theoretical optimal predictive distribution. To do so, the model must implicitly approximate $P(o_{t+1}|o_{1:t})$, whose optimal form is given in Eq. 1 (main paper line 183). From this perspective, pre‑trained LLMs act as effective input‑output predictors of structured sequences, similar to spectral learning algorithms. This differs from Baum–Welch, which tackles a non‑convex parameter‑recovery problem and may fail to converge to the theoretical optimum.

RC to W1.2: The point is not about the emissions. Looking at the transition alone, the ICL is not limited by the need to optimize Σ₁ᴺ p(xₜ|xₜ₋₁) given that it can decode p(xₜ|xₜ₋₁, xₜ₋₂...).

We would like to point the reviewer to the body of classical methods such as Viterbi and Baum-Welch (Appendix C, Algorithm 1 and 3). Note that they both utilize the entire observed sequence for decoding via the forward-backward algorithm (Appendix C line 82-83 and 98-104).

RC to W2.1: I believe the comment comes from a misunderstanding. As addressed above, the emission probabilities that are missing are those that created the data, and which define to the greatest extent the "misclassification" of states to any model (even when the prior is known, hence the classification is optimal from a Bayesian perspective).

As we explained in main point 1, the emission matrix $\textbf B$ is an HMM parameter that we explicitly model during the synthetic experiments.

RC to W3: I respectfully disagree as well. If the goal of the paper is to learn the ICL ability as a predictor on HMM data, what is the point of using "almost-Markovian" data for evaluation? Does the fact that ICL is a better predictor than any other model on data which is assumed to be somewhat Markovian have meaning in the context of this paper?

First, we would like to clarify that we never suggest the real‑world data is “almost‑Markovian.” Instead, the real‑world datasets we chose can be interpreted as observation sequences from HMMs (main paper, lines 257‑259).

We originally interpreted W3 as a suggestion to change our real‑world experiments into a two‑step procedure: learn $\textbf A$, $\textbf B$, $\boldsymbol\pi$ from real‑world data, then evaluate LLM ICL on sequences sampled from these learned parameters.

If this is not the intended meaning, we can reinterpret W3 as: the reviewer would like us to change the synthetic experiments to learn $\textbf A$, $\textbf B$, $\boldsymbol\pi$ from data rather than procedurally generate them. Under this interpretation, our response is:

• Data scarcity and uniqueness. In each scientific setting, the latent dynamics are unique, and typically only a small set of finite trajectories is available. Learning HMMs from such limited data would likely produce a small and fragmented set of parameters. Moreover, parameter recovery is a non‑convex optimization problem, and Baum‑Welch often fails to converge to the true parameters. As a result, the learned HMM parameters would likely be inaccurate and unreliable.
• Need for simulation coverage. Simulation allows us to systematically explore a wide range of HMM properties (e.g., entropy, mixing rate, emission uncertainty) that are interesting to capture the complexity in latent dynamics and emission randomness. This broader coverage is important for evaluating pre‑trained LLM ICL performance.

[1] Y. Ephraim and N. Merhav. Hidden markov processes. IEEE Transactions on Information Theory, 48(6):1518–1569, 2002. doi: 10.1109/TIT.2002.1003838.

[2] Max Simchowitz, Karan Singh, and Elad Hazan. Improper learning for non-stochastic control. In Conference on Learning Theory, pages 3320–3436. PMLR, 2020.

We thank the reviewer for updating their score and acknowledging the emission matrix $\mathbf{B}$ ablation in our experiments. We will discuss the properties of $\mathbf{B}$ in more detail and bring additional experimental setup details into the new revision to avoid such confusions in the future.

Regarding the comment on “simpler influence” and the “memoryless invariant channel” framing, our intention was not to downplay the role of $\mathbf{B}$ in shaping the observation process. Rather, this description reflects the standard factorization in HMM theory: under the Markovian assumption on the hidden states, temporal dependencies in the observations are mediated entirely by the latent chain. The complexity introduced by $\mathbf{B}$ lies in how it transforms hidden dynamics into observed dynamics, which can indeed be substantial in practical modeling. We agree that emphasizing this transformation’s role—especially in the context of the scientific modeling gap we address—would better align with the stated contributions.

We also wish to clarify a potential misunderstanding: while the latent state process in an HMM is first-order Markov, the observation process is generally not first-order Markov unless emissions are deterministic (0 entropy). In general,

$p(o_t \mid o_{1:t-1}) = \sum_{x_t} p(o_t \mid x_t) p(x_t \mid o_{1:t-1}),$

where $\mathcal{X}$ is the latent state space and $\mathcal{O}$ is the observation space. This conditional probability depends on the entire observation history via the posterior over latent states.

This differs from the reviewer’s stated factorization for a first-order Markov time series model:

$p(o_{1:T}) = \prod_{t=2}^T p(o_t \mid o_{t-1}),$

which holds only if the observations themselves are Markov (i.e., $o_t \perp o_{1:t-2} \mid o_{t-1}$). In an HMM, the emission mapping $\mathbf{B}$ breaks this property: two observation sequences with the same last symbol $o_{t-1}$ can have different predictive distributions for $o_t$ depending on the earlier history $o_{1:t-2}$, because that history changes the posterior belief over hidden states.

In other words, $\mathbf{B}$ is not just a static “channel” but the mechanism that converts hidden-state dynamics into history-dependent observation dynamics, creating long-range dependencies that fully observed Markov models cannot capture.

Thank you for your valuable review of our paper. We address each of your points below.

Q1: How are the initial HMM parameters (initial state, transition, emission probabilities) initialized for Baum-Welch training?

We initialize the Baum-Welch (BW) HMM parameters with random seeds using the actual number of states. This already gives BW more information than the LLM, which does not know the number of states.

Each setting is averaged over 4,096 sample sequences to reduce variance in performance (line 106). We do not initialize BW near the ground‑truth parameters, as this would artificially favor BW in low‑sample regimes by effectively giving it prior knowledge of the HMM model.

Q2: What stopping criterion or tolerance is used for Baum-Welch training, and how many iterations does it typically require?

The Baum-Welch algorithm in this implementation uses the change in log-likelihood between iterations as its stopping criterion, halting when the absolute difference falls below a specified tolerance of 1e-6. This ensures the algorithm stops once the parameter updates no longer yield meaningful improvement in fitting the observed sequence. The number of iterations required for convergence typically ranges from 10 to 100, depending on factors such as the number of hidden states, the length and complexity of the observation sequence, and the random initialization of parameters. A maximum iteration cap of 100 is also set to prevent excessive computation in cases where convergence is slow or unattainable.

Q3: How are HMM-generated sequences presented to the LLMs? Are there any special formatting or tokenization strategies used to maximize in-context learning?

All results in the main paper are using upper case letters like ‘ABC’ to represent emissions (Appendix E.2). For each unique emission, we represent it by an upper unique letter and a space. We ensure that each emission would only get tokenized into one token. We performed an ablation on different tokenization strategies for LLMs (Appendix E.2), and found no significant differences on the prediction task performance.

Q4: What decoding strategy and parameters (e.g., temperature, top-k, greedy) are used for LLM next-token prediction during evaluation?

Decoding strategy does not apply in our setting since we directly use the normalized logits for prediction. For example, given a sequence “A B A” with 2 possible emissions (A/B) the LLM would predict 0.3 probability for A, 0.4 probability for B, and 0.3 probability for the rest of the tokens in the vocabulary. We normalize the probability for the possible emissions such that they add up to 1 (3/7 for A and 4/7 for B). For accuracy, we directly take the argmax of the probability and compare it with the ground-truth emission. For Hellinger distance, we compare the probability of all possible emissions (in this case, [3/7, 4/7]) with the ground-truth emission probabilities based on the current hidden state (Appendix B).

W1 Quality: ICL does not generalize HMMs for usual applications for speech and handwriting recognition where HMMs are trained as generative models with multiple sequences at a time.

First, we clarify that our goal is not to replace HMMs as a model architecture in classical applications like speech or handwriting recognition. Instead, this work explores pre-trained LLMs ICL as a complementary statistical tool for analyzing discrete sequential data, particularly in scientific settings where sequences are often unlabeled.

Second, we acknowledge an inherent limitation: pre-trained LLMs operate on discrete tokenizations. As discussed in Section 6 (“Existing Gaps”), extending our approach from discrete observations with underlying discrete hidden dynamics to continuous, real-valued, or high-dimensional observations remains an open and promising research direction.

Finally, our contributions are orthogonal to traditional HMM applications—we focus on demonstrating that there exists a lightweight, forward-pass-based tool for exploratory sequence analysis rather than a replacement for established generative modeling pipelines.

W2 Clarity: Some important implementation details, such as prompt formatting and LLM configuration, are only available in the appendix, making it difficult to understand the methodology without reading all the material.

Due to the main paper space limits, we had to push the implementation details to Appendices for finishing our story. We will work on a better content organization in the revised version. Specifically, we plan to add more details of the experimental setup (Appendix B), a detailed discussion about when LLMs fail to converge (Appendix D.4), and results of ablations on LLMs sizes and tokenizations (Appendix E) to the main paper contents.

W3 Significance: LLMs are generally more complex than usual architectures for HMMs, which can make this an utterly more complex approach to solve problems that could be easily solved with some tweaking in HMM topology.

We appreciate the reviewer’s concern and address it in three parts:

First, beyond using LLMs as a predictive tool on HMM sequences, a key contribution of our paper is demonstrating that LLMs with ICL can achieve near-optimal prediction on HMM sequences. As the community continues to explore the mechanism of ICL, we believe this new empirical observation provides a valuable step toward understanding how LLMs acquire and leverage latent structure without explicit training.

Second, the strength of LLM ICL is not solely due to the transformer architecture but also to large-scale pretraining. In contrast, “tweaking HMM topology” is not straightforward in practice. Our synthetic experiments show that Baum-Welch often fails to match LLM ICL’s robust predictive performance without significant parameter fitting or prior knowledge of the true dynamics.

Third, pre-trained LLMs require no additional training and perform ICL via simple forward passes. In contrast, classical methods require either strong initialization for Baum-Welch or a large dataset to train RNNs effectively. In our experiments, using LLM ICL on average takes 0.14 seconds for a sequence with 4096 emissions on a H100 while it takes 10.15 seconds for a Baum-Welch algorithm to converge a single time on an Intel(R) Xeon(R) Gold 6426Y CPU, and 13.21 seconds for an LSTM model to converge on a H100. This efficiency gap comes from the ability of pre-trained LLMs to perform parallel inference over previous tokens without training, while Baum-Welch and RNNs depend on iterative expectation-maximization or sequence-level training pipelines. From a practitioner’s perspective, this makes LLM ICL a fast, convenient, and low-overhead tool for exploratory sequence analysis.

W4 Originality: The paper does not introduce new learning algorithms or theoretical frameworks, and the most original contributions are in empirical observation and framing rather than methodological invention.

Our work is motivated by the needs of scientists analyzing real‑world dynamical systems. By assessing pre‑trained LLMs as statistical tools, we demonstrate that LLMs performing in‑context learning can achieve near‑optimal performance on HMM prediction, a highly challenging task. These findings provide a foundation for studying how LLMs internally approximate latent probabilistic structures, potentially shedding light on the mechanisms behind in‑context learning. Similar contributions have appeared at NeurIPS, including works such as [1, 2].

[1] Garg, Shivam, et al. "What can transformers learn in-context? a case study of simple function classes." Advances in neural information processing systems 35 (2022): 30583-30598.

[2] Gruver, Nate, et al. "Large language models are zero-shot time series forecasters." Advances in Neural Information Processing Systems 36 (2023): 19622-19635.

I acknowledge that I have read the rebuttal and reviews and will raise my final score to "accept". I think this a good, interesting paper

Thank you for your insightful and helpful feedback. We respond to your individual questions below.

Q1: In Fig 4 caption, isn't smaller lambda 2 fast mixing and large lambda 2 slow mixing?

Yes. Thanks for catching this, we will update in the next version.

Q2: There must be limits, right? Are there extreme cases where LLMs cannot figure out the hidden state well?

Yes. During the synthetic experiments, we did observe cases (e.g., Appendix D, Fig. 2: 8 states, 4 observations, A entropy ≈ 2.5) where LLMs ICL failed to converge to the Viterbi algorithm’s performance. The detailed discussions are in Appendix D.4.

Q3: Have you observed any sudden transitions? i.e. the LLM "grokking" that there exists a hidden state?

This is an insightful question. Indeed, in-context learning is an emergent property, which appears during later stages of pretraining. A related theoretical work [1] studies how ICL emerges during pretraining. However, a key difference between this and our experiments is training versus inference.

From our experiments, we do not observe any sudden transitions. Instead, the performance improvements of LLMs ICL are gradual and steady (as shown in Main paper Figure 3, Appendix D multiple Figures). Such behavior is one key reason that we hypothesize LLMs ICL operate as input–output predictors, effectively performing a form of smart filtering over observation sequences.

W1 & W2: While the paper demonstrates some connection to spectral learning theory, the predicted learning curves are not explicitly fitted/compared to the empirical in-context learning curves.

Following the reviewer’s suggestion, we added an additional experiment by implementing Algorithm 6 (Appendix F), a variant of spectral learning with theoretical performance guarantees in Theorem 1, and evaluating it on the same synthetic HMM setting used in Figure 5 (8 states, 8 observations). The comparative results are shown in the table below:

When the sequence length is short (≤ 64), the hidden‑state belief update b_\hat​ becomes numerically unstable because the sample complexity of spectral learning scales as $\mathcal O (M^2 L)$, which is 512 in our setting. Once the context length reaches ≥ 512, spectral‑learning predictions converge distributionally to the ground truth, and their performance aligns with that of the LLMs, unlike the BW‑based predictions. At very large context lengths, spectral learning yields smaller Hellinger distances to the true output distribution, whereas the LLM achieves slightly higher accuracy. This occurs because LLMs tend to over‑concentrate probability mass on the top choice—boosting accuracy but increasing divergence for the rest of the distribution.

Q4: What if you gave the context then natural language prompted the model (+CoT) to predict the next state?

We appreciate the thinking engagements of the reviewer! Although we didn’t include any experiments using natural language and reasoning models in this paper, we did experiment with this setup directly on neuroscience real-world data. What we found was that the LLMs struggle to do reasoning in the meaningful way that we hope. More importantly, when natural language prompts (e.g., the mice's experimental contexts) are mixed into the dynamical sequences, the prediction accuracy actually has decreased significantly. It is an interesting future work to understand what happens to the HMM sequential prediction task when natural language prompts are introduced.

[1] Edelman, Ezra, et al. "The evolution of statistical induction heads: In-context learning markov chains." Advances in neural information processing systems 37 (2024): 64273-64311.

Thank you for your thorough review and thoughtful suggestions. We address your questions and concerns below.

Q1: Are LLMs... constrained to prediction?

Our findings show that LLMs ICL performance extends beyond simple “argmax” prediction. Using distributional metrics such as Hellinger distance (Main Paper Fig. 5, right; App. D Fig. 3), we observe that LLM‑predicted distributions converge toward the ground‑truth next‑emission distributions of the underlying HMMs.

Motivated by the needs of scientists analyzing real‑world dynamical systems, our goal is to assess pre‑trained LLMs as statistical tools for structured sequential data rather than to explain their internal mechanisms. Future work will investigate how LLMs internally approximate latent probabilistic structures, which may illuminate the mechanisms behind in‑context learning.

Q2: How can we guarantee... learnability?

We interpret this question as aligned with the reviewer’s concern in W5 regarding whether the guidelines in Section 4 generalize beyond the specific findings in this paper.

Our decision to use synthetic experiments on HMMs is directly motivated by the scientific use case we stated above. We deliberately chose HMMs for two reasons:

1. HMMs are an expressive class of generative models that are broadly used in scientific modeling—many real-world discrete-time dynamical systems [1-3] can be represented as or reduced to HMMs.
2. HMMs capture how real-world scientific data is often structured: scientists often start with hypotheses about latent dynamics and choose observables accordingly [4,5]. This results in the fact that we often only can obtain partially observable data—a setting for which HMMs are a natural formalism.

We also want to emphasize that the guidelines in Section 4 are intentionally cautious and observational, not prescriptive. For example, we state: “If you observe that an LLM’s ICL prediction accuracy on your data sequence steadily improves and saturates with increased context length (like in Figure 3), this strongly indicates a learnable, non-random underlying structure.” We do not claim this is a necessary and sufficient condition for learnability. Rather, the goal is to provide a useful heuristic for scientists who may not be experts in LLMs but are trying to assess whether their sequential data contains structure that can be exploited.

Lastly, regarding the possibility of guaranteeing the utility of LLMs for real data: in scientific domains like neuroscience, practitioners commonly use models such as LSTMs [6,7] to fit behavioral or neural sequences—despite limited theoretical guarantees. LLMs ICL plays a similar practical role, offering the additional benefit of being usable out-of-the-box without task-specific training.

Q3: How do we expect transformer... prediction task?

This is an excellent question that will be helpful for us to understand the internal mechanisms of LLMs ICL’s strong performance on HMMs. We have included an additional experiment here as a first step in response to this suggestion, and will add the full set of results to the final version of the paper. Specifically, we train a 2-layer transformer model using the same training setup as in our LSTM experiments (App. D) with the same hidden/feed forward dimension and the same embedding dimension. For each evaluation, we average performance over 16 training samples, as we do with LSTMs. We provide a comparative table below under the same setting as Figure 5 (8 states, 8 observations):

As shown, the trained small transformer model performs more similar to the pre-trained LLM (Qwen2.5 7B) than the LSTM baseline does. While the trained models exhibit higher variance due to limited sample sizes, these findings suggest that the transformer architecture itself plays a meaningful role in performance. In parallel, large-scale pretraining likely enables LLMs to leverage in-context learning to approximate the benefits of task-specific training.

We appreciate the review’s suggestion and will incorporate the trained transformer baseline into the synthetic experiments section of the paper.

Relatedly, in W3, the reviewer requested an ablation study on LLM size. We have conducted ablations on both model size and tokenization (App. E) from the perspective of tool usability. However, we did not emphasize these results in the main paper, as they did not reveal useful scaling trends.

Q4: What is the relationship... theory in Section 3.3?

We interpret this question associated with the reviewer’s concern in W2—that the paper does not clearly explain why the theoretical characterization of spectral learning for HMMs may provide insight into the ICL results.

Our primary motivation for introducing spectral learning is the hypothesis that LLM ICL behaves as an input–output predictor, effectively performing a form of smart filtering over observation sequences. This contrasts with explicit parameter‑estimation methods like the non‑convex Baum–Welch (BW) algorithm, which may fail to converge to the global optimum. This perspective is supported by the distinct behavioral patterns we observed between LLMs and BW across experiments (Sec. 3.2, Fig. 5; App. D).

The second motivation is that we desire a notion of sample complexity relating to accurate prediction and complexity of the underlying dynamics. Spectral methods provide sample‑complexity guarantees for HMM prediction tasks in terms of the structural properties of the underlying process (e.g., mixing rate and emission rank). This gives a principled way to interpret the LLM ICL results (Sec. 3.1, Fig. 4), and motivates our derivation of a sample complexity bound for spectral learning on HMMs.

To further support this connection, we added an additional experiment by implementing Algorithm 6 (App. F), a variant of spectral learning with theoretical performance guarantees in Theorem 1, and evaluating it on the same synthetic HMM setting used in Figure 5 (8 states, 8 observations). The results are discussed in our response to reviewer oWyV (W1 & W2) and are consistent with our hypothesis: LLMs behave more like spectral learners than EM‑based methods like BW.

We acknowledge that this conceptual link between ICL and spectral learning could be emphasized more clearly in the main paper and plan to highlight it in a revision.

W4: The findings from the reward-learning rats...

This is a very thoughtful assessment.

The CogFunSearch baseline (App. H.2, “Rat FullModel” in [8]) represents a cognitive model with latent states reflecting mice’s internal strategies and emission probabilities corresponding to their decisions. This naturally maps to an HMM framework.

Importantly, CogFunSearch achieves its strong performance through extremely computationally expensive search over model structures, fitting each mouse individually—taking weeks to complete—so its results are close to an “optimal algorithm” benchmark.

We want to further address the reviewer’s concern about the numerical performance gap:

1. Neuroscientists typically evaluate models by normalized likelihood over full sequences, aggregated per animal. By this metric (used in [8]), LLM ICL achieves competitive performance for 3 out of the 20 animals (CogFunSearch 0.672±0.012). In our paper, we chose to visualize prediction accuracy versus sequence length to highlight that LLMs ICL curve is flat, which mirrors our synthetic experiments when the normalized entropy of transitions and emissions is high (e.g., App. D, Fig. 2: 8 states, 4 observations, A entropy ≈ 2.5).

2. The dataset has only two possible observations, so even modest numerical gaps can appear visually large. The Figure 7 gap corresponds to a (0.864 - 0.758)/0.5 ≈ 0.21 over chance, which is comparable to our synthetic results comparable to our synthetic high‑entropy HMM results (e.g., above setting 0.05/0.25=0.2).

W1 and Comments

Thanks for the suggestions on style improvements and catching the typo. We will surely update those, incorporating the additional experiments.

[1] Mor B et al. (2021). A systematic review of hidden Markov models and their applications. Arch Comput Methods Eng 28:1429–1448.

[2] McClintock BT et al. (2020). Uncovering ecological state dynamics with hidden Markov models. Ecol Lett 23:1878–1903.

[3] Noé F et al. (2013). Projected and hidden Markov models for calculating kinetics and metastable states of complex molecules. J Chem Phys 139:18.

[4] Gershman SJ & Niv Y (2010). Learning latent structure: carving nature at its joints. Curr Opin Neurobiol 20:251–256.

[5] Gershman SJ et al. (2010). Context, learning, and extinction. Psychol Rev 117:197.

[6] Jordan MI. (1986). Attractor dynamics and parallelism in a connectionist sequential machine. Proc CogSci 8.

[7] Wills TJ et al. (2005). Attractor dynamics in the hippocampal representation of the local environment. Science 308:873–876.

[8] Castro PS et al. (2025). Discovering symbolic cognitive models from human and animal behavior. bioRxiv.

We sincerely thank the reviewer for the thoughtful follow‑up and for acknowledging that our initial rebuttal clarified several points, including how LLM ICL performance on real‑world HMM data informs our understanding of the underlying dynamics. We also appreciate the opportunity to address the remaining concerns.

1. On LLMs “learning” HMMs vs. next‑token prediction

We agree that for an LLM to fully “learn” an HMM in the classical statistical sense, it would need to internally recover the hidden structure—such as the transition and emission matrices. Explicit parameter recovery, as in Baum–Welch, additionally requires assumptions on the model structure (e.g., that the hidden‑state transition matrix is $M\times M$; Table 1 in the main paper). Our claim is not that LLMs perform such recovery. Rather, their in‑context learning behavior is consistent with implicit statistical learning of latent dynamics, specifically in the sense of improper learning.

As we note in Section 3.3 (line 188-191):

“…one can predict the next observation directly using these parameters along with hidden‑state belief updates, without explicitly learning the matrices $\textbf A$ and $\textbf B$. This is therefore an example of improper learning, which has been extensively studied in related areas like linear dynamical systems [1].”

To achieve the observed performance—approaching the theoretical optimal predictive distribution—a pre‑trained LLM must internally approximate $P(o_{t+1}|o_{1:t})$, whose optimal form is given in Eq. 1 (main paper, line 183-186). From this perspective, pre‑trained LLMs act as an effective input‑output predictors of structured sequences, much like spectral learning algorithms, and unlike BW which solves a non-convex optimization problem and does not always converge to the theoretical optimum.

To avoid confusion, we are open to renaming our contribution as “learn to predict” rather than “learn” in the title, to emphasize that LLM ICL focuses on effective in‑context prediction rather than classical parameter recovery. We also invite the reviewer to clarify whether their concern is primarily terminological—i.e., about our use of “learn”—or whether they believe contributions based on implicit predictive learning are insufficient without explicit parameter recovery. We believe the former is a naming issue, whereas the latter would discount a large body of influential work on predictive modeling (e.g., spectral and meta‑learning approaches) despite their strong empirical utility.

2. On the role of transformer architecture vs. pre‑trained LLMs

We appreciate the reviewer’s observation that the transformer architecture is a key factor in the performance trends we report. As shown in the table from our previous response, a 2‑layer transformer trained from scratch on synthetic HMM sequences achieves comparable performance to a 7B pre‑trained LLM in ICL, whereas LSTMs perform substantially worse. This validates that transformer architecture provides an inductive bias for learning latent dynamics.

However, we want to continue discuss several points where our perspective differs:

LLMs themselves are not the solution to this task… Larger transformers that are not the size of LLMs might even outperform these pre-trained LLMs…

First, in many of our HMM settings, pre‑trained LLMs already converge to the theoretical optimal predictive distribution, leaving little headroom for other models to “outperform.”

Our goal in this work is not to find the single highest‑performing HMM predictor, but rather to examine whether in-context learning (without any gradient updates) can do well on this hard statistical task. And based on our current findings in the experiments, the Hellinger distance of ICL shows very stable distributional convergence, which is impressive, whereas the trained small transformers show instability.

If the finding is simply that transformers can do well on HMM tasks, then this limits the novelty of the findings/results as it is already well-studied that LLMs are very good at pattern matching.

Pre‑trained LLMs achieve this near-optimal performance on HMMs without any gradient updates. In contrast, small transformers require direct supervised training on the task data, for which having a good performance is more expected.

The fact that ICL alone can solve HMM prediction is non‑trivial and not directly implied by prior ICL work. Prior work on ICL spans a spectrum:

1. Theoretically grounded explanations for simple tasks such as fully observed Markov models [2–4], and
2. Empirical demonstrations of pattern matching on complex sequences without theoretical justification [5].

Our work benchmarks LLMs on HMMs with clear theoretical baselines, including Viterbi, Baum–Welch, and spectral learning, and shows that LLMs can match or surpass these classical methods. This goes beyond rote pattern‑matching and aligns ICL behavior with classical statistical learning in a principled way. Benchmarking LLMs on tasks with clear theoretical baselines—even relatively simple ones like linear regression [6,7]—is considered valuable for understanding ICL.

Moreover, we demonstrate the practical usage of this capability by applying LLM ICL to real animal behavioral data, which to our knowledge is the first demonstration of LLM ICL performing HMM‑style inference in a real‑world setting. For these reasons, we believe our study makes a novel and meaningful contribution.

We thank the reviewer for their time in delving deeper with us and for their openness to discussion.

We respectfully disagree with the characterization of our contribution as “limited in the context of existing work mentioned in those fields.” Our core contribution is the quantitative demonstration that LLM ICL achieves near-optimal prediction for HMMs—a complex yet expressive class of generative models widely used in scientific modeling. This is significant for two reasons:

• For Science: In many scientific applications, obtaining an optimal predictive model without complex, non-convex parameter fitting or additional training is highly valuable. Our synthetic experiments reveal ICL scaling trends linked to key HMM properties like mixing rate and emission entropy (Section 3). We complement this with practical guidelines for scientists who may not be LLM experts, and a real-data demonstration of the tool’s effectiveness (Section 4). This combination of empirical characterization and real-world validation makes this “off-the-shelf statistical tool” truly usable for scientists. We believe that works demonstrating how to apply ICL to a specific field or use case [20] offer significant value.
• For ICL: Prior work does not imply that pre-trained LLMs can steadily converge to optimal HMM prediction. Existing works either focus on how ICL emerged [1-3,14,19], demonstrate ICL on simpler or non-sequential tasks (e.g., fully observed Markov or i.i.d. samples) [4-12,15], or examine correlations between ICL performance and pre-training data [16–19]. A detailed discussion is provided in our follow-up comment, including analysis of the “ICL as implicit Bayesian inference” line of work [14,15], which covers the paper cited by the reviewer. Achieving near-optimal predictive distributions here requires implicitly tracking latent states and performing belief-state updates over long contexts—capabilities beyond rote pattern-matching. Benchmarking against Viterbi, Baum–Welch, and spectral learning shows that LLM ICL matches or surpasses classical methods on a classically hard sequential problem without gradient updates, pushing the boundaries of current ICL understanding.

For point 2, we thank the reviewer for noting our results with the transformer. We would like to emphasize the following (independent of model size):

• The small, trained transformer is a specialist. Its strong performance is expected, as it was trained explicitly and exclusively on HMM data. Regarding the reviewer’s suggestion to explore larger transformers, we agree this could yield useful insights and will conduct necessary ablations in future work to better understand the internal mechanisms of ICL on HMMs. Our current experimental design follows prior ICL studies, which typically use 1–3-layer transformers to obtain theoretically tractable results [1–12].
• The pre-trained LLM is a generalist. It has never been fine-tuned on HMMs, yet with zero gradient updates it can infer the task from context and perform on par with a specialist—approaching the theoretical optimum. This ability is both surprising and, in our view, novel. Moreover, our ablations on LLM size and tokenization (Appendix E) show that this level of performance emerges once the model exceeds a certain size threshold, and that it remains robust across different tokenization schemes.

On point 3, we apologize for the lack of clarity here. The connection to spectral learning is not intended to provide a complete theory of how pre-trained LLMs work. Rather, it serves two specific, pragmatic purposes in our paper:

• Spectral learning theory offers a principled framework for interpreting our findings. Given an observation sequence, the optimal form of $P(o_{t+1}|o_{1:t})$ (Eq. 1, main paper line 183) can be directly derived from the ground-truth HMM parameters. Spectral learning is designed to optimize this exact objective, making its sample-complexity bounds true lower limits for any learner without prior knowledge of the sequence’s latent structure. These bounds—explicitly dependent on HMM properties such as mixing rate and emission entropy—provide a plausible explanation for the shapes of the ICL performance curves we observe. They clarify how data properties influence model performance, and in practice, they can guide the choice of context length needed for reliable next-emission prediction.
• Spectral learning theory helps situate the ICL relative to classical methods. Unlike EM-based algorithms (e.g., Baum-Welch), which can become trapped in local optima, both spectral learning and LLM ICL behave as input-output predictors that can bypass this non-convexity. The similar performance curves suggest they might operate in a similar paradigm, distinct from explicit parameter fitting.