One Filters All: A Generalist Filter For State Estimation

Thank you very much for your comments once again. I hope this clarifies your concerns.

[Question 1]: Have you evaluated it in a synthetic setting where the Kalman filter is known to be optimal? If so, what is the performance gap between your method and this optimal benchmark? This would be very useful to understand the limits of your approach.

Thank you for the insightful suggestion. We have added an experiment in a synthetic linear Gaussian system, the Tracking system, where the classical Kalman filter is known to be optimal. This system is part of the cross-system evaluation in Section 5.2, with detailed system dynamics provided in Appendix B. All training and data configurations follow those in the main paper. The results are summarized below:

As expected, the Kalman filter achieves the best RMSE. LLM-Filter performs competitively, slightly outperforming KalmanNet and LLM-Filter-O. The improvement over LLM-Filter-O highlights the benefit of incorporating external prior knowledge via prompts.

[Question 2]: The paper would benefit from a discussion on sample efficiency. Could you clarify the data regime of your experiments (e.g., low-sample vs. high-sample) and analyze how the relative performance of your method against baselines changes with the number of available samples? Such an analysis could be particularly insightful, as it might also explain why fine-tuning with LoRA leads to performance degradation.

That's a good suggestion. In our main experiments, all methods are trained using 20,000 samples per system, as detailed in Appendix B. This data volume was selected because performance across methods, including ours, saturates beyond this point. To better assess sample efficiency, we conducted additional experiments on the Selkov system by subsampling the training data to 10%, 40%, 70%, 90%, and 100% of the original size. The results are summarized in the table below:

As shown, performance degrades rapidly with reduced training data, and most methods fail to converge at the 10% level. However, LLM-Filter consistently outperforms all baselines across all data regimes.

Regarding the observed performance drop when fine-tuning with LoRA, we attribute this to the limited expressiveness of low-rank parameter updates in capturing the precision required for accurate state estimation.

[Question 3]: I am surprised that frozen LLMs lead to this level of performance. Could you explain in more detail why you think LLMs can produce a representation that is only a simple MLP away from predicting the system's hidden state? In particular, would it be possible to run experiments to gain some understanding of what kind of representations the LLM produces?

Thank you for the thoughtful question. First, we observed that in the time-series forecasting domain, some studies [1, 2] have leveraged frozen LLMs. These works suggest that the pretrained knowledge embedded in LLMs can be effectively applied to forecasting tasks. This inspired us to ask: could pretrained LLMs also contain knowledge that is helpful for filtering tasks?

To investigate this, we attempted to isolate the contribution of pretrained knowledge. As suggested by Reviewer #HiuW in Question 3, we compared a pretrained LLM with a randomly initialized LLM of the same size. Training the randomly initialized model from scratch led to divergence, whereas the pretrained LLM successfully performed the filtering task and demonstrated generalization in the cross-system task. This suggests that pretrained LLMs encode useful knowledge and priors that benefit the filtering task.

Additionally, as shown in Figure 6, removing the SaP results in significant performance degradation. This suggests that LLMs are able to extract and utilize structured information about the system from the prompt, further supporting their utility in state estimation.

Despite these observations, we acknowledge the difficulty in providing a system-level explanation of the internal representations LLMs produce. We tried various visualization techniques, such as attention heatmaps, hidden state visualizations, and adversarial perturbations. Unfortunately, none of these methods provided a clear explanation of how LLMs work in estimation tasks.

We have reviewed a broad body of related literature and found that even successful applications of LLMs to specialized domains (e.g., [1–6]) typically do not offer clear explanations of why LLMs perform so well. We would greatly appreciate any insights or suggestions you might have for deeper interpretability or representation analysis in this context.

[Weakness 1]: It is generally unclear what the technical originality of this paper is. The proposed ObsEmbedding and StateProjection are standard for time series, as the authors mention. The prompting is also straightforward. Apart from the experimental evidence provided, I do not see any additional contributions. There is no in-depth analysis, explanation, or intuition regarding why the proposed approach works better than conventional learning-based filters or online Bayes filters.

Thank you for the opportunity to clarify our contributions. The primary contribution of this work lies not in proposing novel architectural modules or theoretical frameworks, but in demonstrating a new paradigm for filtering: leveraging the in-context learning capabilities of LLM to build a generalizable filter. This approach enables adaptation to a wide range of dynamical systems without requiring system-specific retraining, thereby addressing a major limitation of conventional learning-based filters that rely heavily on system-specific data and training procedures.

To be honest, our motivation originated from practical challenges in robotic state estimation, particularly in quadruped robots. We initially employed KalmanNet [7], a representative learning-based filter. However, we observed that KalmanNet's performance degrades significantly when the environment changes (e.g., from indoor flat surfaces to outdoor sandy terrain), and adapting it to new settings required additional data collection and retraining, highlighting the limitations in generalization.

Simultaneously, we observed that the control and time-series domains have explored the use of pretrained LLM knowledge to improve generalization. Inspired by OpenVLA [2], we initially attempted to represent continuous system states as discrete tokens within the LLM’s vocabulary space. However, we found that this discretization introduced significant precision loss, making it unsuitable for high-accuracy tasks like continuous-state estimation in filtering.

Following insights from time-series forecasting works [4, 5], we then designed a modality alignment architecture to connect observations and states to the LLM. However, early versions of this setup struggled to generalize across systems, largely because the model lacked explicit knowledge of the underlying system dynamics. This led us to introduce the SaP, which encodes system-specific information in a format interpretable to the LLM. This component was crucial for enabling the LLM to effectively reason over both the observation history and the system characteristics.

We believe our work contributes both a practical solution and a new perspective on using pretrained LLMs for general filtering, which has not been previously explored in this form.

Reference

[1] Gruver, N., Finzi, M., Qiu, S., & Wilson, A. G. (2023). Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 19622-19635.

[2] Liu, Y., Qin, G., Huang, X., Wang, J., & Long, M. (2024). Autotimes: Autoregressive time series forecasters via large language models. Advances in Neural Information Processing Systems, 37, 122154-122184.

[3] Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., ... & Finn, C. (2024). Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246.

[4] Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., ... & Han, K. (2023, December). Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning (pp. 2165-2183). PMLR.

[5] Liu, X., Hu, J., Li, Y., Diao, S., Liang, Y., Hooi, B., & Zimmermann, R. (2024, May). Unitime: A language-empowered unified model for cross-domain time series forecasting. In Proceedings of the ACM Web Conference 2024 (pp. 4095-4106).

[6] Shojaee, P., Meidani, K., Gupta, S., Farimani, A. B., & Reddy, C. K. (2024). Llm-sr: Scientific equation discovery via programming with large language models. arXiv preprint arXiv:2404.18400.

[7] Revach, G., et al. KalmanNet: Neural network aided Kalman filtering for partially known dynamics. IEEE Transactions on Signal Processing, 70, 1532-1547.

Thank you very much for your insightful question. We appreciate the opportunity to clarify and hope the following explanation addresses your concerns.

[Question 1]: Baseline parity and training budget.

Thank you for your question. We provide a detailed breakdown of the training protocols and implementation details of each baseline below to clarify training budget, early stopping, and tuning fairness.

• For KalmanNet, we directly used the official implementation. This version does not include built-in early stopping, and saves the best-performing checkpoint on the validation set and reports its test results.
• For RStateNet, MEstimator, and ProTran, we re-implemented the architectures and applied the same early stopping strategy used for LLM-Filter (patience=3, based on validation loss).
• For all methods, we conducted modest grid searches on core hyperparameters (e.g., learning rate, hidden size), using the same training and validation data. Test data was strictly held out until final evaluation.

Regarding the divergence of KalmanNet on high-dimensional systems, we emphasize that this issue is not due to insufficient training time or suboptimal hyperparameter tuning. Instead, we believe it arises from KalmanNet’s architectural reliance on Kalman structure and a hidden state representation whose dimensionality scales quadratically with system size. These structural choices, while suitable for low-dimensional linear systems, significantly limit scalability and representational capacity for high-dimensional nonlinear systems.

We hope this addresses your concern and assures a fair comparison was made across all baselines.

[Question 2]: Observation-to-token embedding fidelity.

Thank you for your question. While visual inspection might suggest instability, the actual numerical results show that performance variance decreases as the hidden dimension shrinks. The complete RMSE values across different segment lengths and hidden dimensions are provided in the table below:

Overall, the performance variance remains within a small range (generally under 0.1), indicating that our method is stable across window lengths. Additionally, increasing the hidden dimension generally leads to improved performance, but also results in slightly higher variance across different window lengths. This is because larger hidden dimensions give the MLP more capacity to capture complex patterns from LLM embeddings, leading to better accuracy. However, the increased capacity can also make the model more sensitive to training data and hyperparameters, resulting in slightly higher variance.

We apologize if we misunderstood your request. We assumed that "reconstruction error visualization" refers to visualizing the residuals between the estimated states and ground-truth values. We have plotted the estimation vs. ground truth trajectories for the Selkov system to analyze the residuals. Due to rebuttal constraints, we are unable to include figures here, but we will add them in the final version of the paper. We apologize for this limitation.

We would also like to clarify that our work focuses on the discrete filtering problem [2], where both the system and observations are discretized. For each dynamical system, we apply discretization beforehand, as described in Appendix B, which introduces unavoidable information loss equally to all methods. To ensure fairness, all baseline methods we compare against are also discrete filtering approaches and are trained and evaluated on data collected from the same discretized system.

[Question 3]: Scaling-law interpretation vs. parameter count confound.

This is a great question, which also helps explain why large models are effective. We initially considered conducting an ablation study by randomly initializing the 7B-parameter LLM backbone to disentangle the effects of model capacity from pretraining knowledge. However, in practice, we found that training such a large model from scratch using our task-specific dataset leads to divergence, after trying all possible skills, such as warmup schedules, gradient clipping, and regularization techniques. This instability may partly be due to our current hardware limitations, which make it infeasible to fully train a 7B-parameter model from scratch. More fundamentally, it aligns with prior observations that training large models such as LLaMA-7B typically requires extensive data, multi-stage curricula, and significant computational resources to achieve stable convergence [1].

A promising direction for future work is to train a foundation filtering model from scratch by first collecting a comprehensive and unified dataset encompassing multiple dynamical systems. Following this, a multi-stage training procedure could be employed, beginning with large-scale pretraining on the diverse dataset to learn general filtering capabilities, and then fine-tuning on task-specific datasets to adapt to particular system dynamics.

[Question 4]: Runtime and memory claims.

Thank you for your question. All methods were evaluated on the same hardware: a single NVIDIA H800-80G GPU. For LLM-Filter, we used bf16 precision, while for the smaller baseline models, we followed PyTorch's default fp32 precision.

We would also like to clarify that the runtimes reported in Table 6 reflect per-step inference latency with batch size = 1, which is typical for real-time filtering tasks. As such, batch size does not influence the latency values presented in the table.

It is important to note that inference speed is not solely determined by model size. Differences in network structure are also crucial. For example, LLMs often benefit from optimized inference kernels, fused attention mechanisms, and high degrees of parallelism, which can result in faster execution even with larger parameter counts. In contrast, small custom models such as RStateNet may not take advantage of such low-level optimizations, potentially resulting in slower inference despite their compact size.

[Weakness 1]: Theoretical Justification.

Thank you for your insightful comment. We acknowledge that establishing formal connections or deriving estimation error bounds would certainly strengthen the theoretical foundation of this work, and we view this as a promising direction for future research.

We also wish to clarify that the primary contribution of this work lies not in proposing novel theoretical analysis, but in demonstrating a new paradigm for filtering: leveraging the in-context learning capabilities of LLM to build a generalizable filter. This approach enables adaptation to a wide range of dynamical systems without requiring system-specific retraining, thereby addressing a major limitation of conventional learning-based filters that rely heavily on system-specific data and training procedures.

[Weakness 2]: Limited Noise Models.

Thank you for your comment. We would like to clarify that a classical observation-disturbance scenario, featuring heavy-tailed noise, has been included in Appendix E.2.

As shown in Table 8, although LLM-Filter did not achieve the best performance in this setting, it performed comparably to specialized robust filtering algorithms. This result highlights LLM-Filter’s ability to generalize well even in the presence of significant outliers or system disturbances, making it a promising solution for diverse and challenging scenarios.

Reference

[1] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M. A., Lacroix, T., ... & Lample, G. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

[2] Särkkä, S., et al. Bayesian filtering and smoothing (Vol. 17). Cambridge university press.

Thank you for your valuable feedback on our manuscript "One Filters All: A Generalist Filter For State Estimation".

In our response, we supplemented experiments and addressed your concerns regarding baseline parity, embedding fidelity, and so on. As the review period is concluding shortly, we wish to confirm whether our clarifications adequately resolved your questions? We understand your schedule may be demanding, and we are happy to provide additional information promptly if needed.

Thank you again for your essential contribution to improving this work.

Thank you very much for your insightful question. We appreciate the opportunity to clarify and hope the following explanation addresses your concerns.

[Question 1]: For very long sequences it may not be feasible to feed the whole embedding sequence into the Transformer. Did the authors consider what are good ways to compress the context in this case? Also does the quality drop significantly?

Thank you for your question. Indeed, we have adopted a sliding window mechanism, which corresponds to the commonly used Moving Horizon Estimation (MHE) framework [1] with theoretical guarantees, as described in Section 3.1 of the paper. This approach ensures that only a fixed number of recent observations are included in the input to the LLM, keeping the embedding sequence length bounded and computationally tractable.

Additionally, as discussed in Appendix E.4, we observe that increasing the window length beyond 20 provides diminishing returns in performance. With a window length of 20, the resulting embedding sequence typically contains fewer than 200 tokens, which is well within the effective context length of modern large language models [2, 3]. Based on this observation, we use a window length of 20 in all experiments.

[Weakness 1]: The paper assumes that we have access to the true states in Eq 4. But I wonder if this is a very strong assumption.

In practice, this assumption is not as strong as it may initially seem. Supervised learning-based filters typically require access to ground-truth states during training [4, 5, 6], which are often obtained using high-precision sensors or external tracking systems. This setup is standard in many real-world applications.

For example, in quadruped robot state estimation [7], ground-truth data is commonly collected using an indoor motion capture system to supervise the training process. Once trained, the filter can operate using only onboard measurements such as IMUs and leg odometry, while still producing accurate state estimates.

[Weakness 2]: Moreover, there have been prior papers on neural approximation of hidden state models and Lin and Eisner, which may be discussed in the paper. In particular, Lin and Eisner considered a smoothing scenario where the entire sequence of is given before hand, which can likely further improve accuracy in this paper's setting as well.

That's a good question. We would like to clarify that smoothing and filtering correspond to different problem formulations [8]. Smoothing estimates the current state based on both past and future observations, whereas filtering relies solely on observations available up to the current time to infer the current state. While smoothing can improve accuracy by leveraging future information, it is generally infeasible in real-time applications, Such as robotics control. For example, a leg-robot must make decisions (e.g., control actions, trajectory planning, collision avoidance) based on the current belief of its state, which must be inferred from past and immediately available sensor data. Future sensor readings are inherently unavailable during execution, making smoothing inapplicable in such real-time systems. Therefore, this is a typical filtering problem, and our paper is designed with this class of applications in mind

We also appreciate your pointing us to the work by Lin and Eisner. We acknowledge that our paper currently lacks a discussion of smoothing-based methods, and we will include a more complete review of related smoothing literature in the appendix to better situate our work. Thank you again for your thoughtful feedback.

Reference

[1] Alessandri, A., Baglietto, M., & Battistelli, G. (2008). Moving-horizon state estimation for nonlinear discrete-time systems: New stability results and approximation schemes. Automatica, 44(7), 1753-1765.

[2] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., ... & McGrew, B. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

[3] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M. A., Lacroix, T., ... & Lample, G. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

[4] Revach, G., et al. KalmanNet: Neural network aided Kalman filtering for partially known dynamics. IEEE Transactions on Signal Processing, 70, 1532-1547.

[5] Dahal, P., Mentasti, S., Paparusso, L., Arrigoni, S., & Braghin, F. (2024). RobustStateNet: Robust ego vehicle state estimation for Autonomous Driving. Robotics and Autonomous Systems, 172, 104585.

[6] Ji, G., Mun, J., Kim, H., & Hwangbo, J. (2022). Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion. IEEE Robotics and Automation Letters, 7(2), 4630-4637.

[7] Buchanan, R., et al. Learning inertial odometry for dynamic legged robot state estimation. In Conference on robot learning (pp. 1575-1584). PMLR.

[8] Särkkä, S., et al. Bayesian filtering and smoothing (Vol. 17). Cambridge university press.

Thank you very much for your insightful question. We appreciate the opportunity to clarify and hope the following explanation addresses your concerns.

[Question 1]: Did you do any experiments playing with the levels of SaP? Such as omitting taks examples, varying the amount of examples provided. I know you included LLM-filter-O, but I would like to see a bit more experimentation with how SaP added to the efficacy of the filter.

Thank you for the insightful suggestion. To better understand the role of SaP, we conduct an ablation study by varying the number of task examples in the prompt, while keeping the task instruction fixed. The results are summarized in the table below.

We observe that adding a few task examples significantly improves performance, confirming their value as in-context cues. However, the marginal gain diminishes as the number increases. This is likely due to attention dilution, where too many examples reduce focus on the actual input segment. Moreover, the result for 0 example (task instruction only) outperforms LLM-Filter-O (no instruction nor examples), highlighting that task instruction alone provides a meaningful signal for the model.

These findings suggest that both task instruction and examples are beneficial, but the number of examples should be balanced to avoid overwhelming the attention capacity of the model.

[Question 2]: LLM-Filter optimizes MSE but does not output filter covariance: Can the authors: quantify residual error distributions for the Hopf system, and identify conditions where the filter diverges or large errors occur?

Thank you for your question. As far as we know, existing learning-based filters [1-4] generally do not provide an explicit quantification of residual error distributions either. They are usually trained to directly predict outcomes (e.g., filtering decisions), but they don't come with a built-in probabilistic model that would allow for estimating uncertainty in the residual error.

One possible direction to address this limitation is to adapt uncertainty estimation techniques developed for LLMs. For example, some recent work has proposed using Laplace approximation [5] to estimate the variance of the model's output, which provides a principled way to quantify uncertainty. Incorporating such techniques into LLM-Filter could enable future extensions that include uncertainty estimates alongside state predictions.

[Question 3]: Could you clarify a bit more about where the latency/memory overhead is coming from in the LLM filter?

Thank you for the question. Below, we provide a detailed breakdown of the training time and GPU memory usage for each component of the LLM-Filter in the Selkov system. The 'Ops' primarily consist of reshape, normalization operations. The pre-allocated memory includes storage for input data, model weights, and key-value (KV) caches, which remain mostly static during inference and training.

The latency and memory overhead primarily stem from the LLM core, while the remaining components and operations add only minor costs. This is consistent with the expected resource demands of LLMs during filtering tasks.

We hope this clarifies where the latency and memory overheads arise.

Reference

[1] Revach, G., et al. KalmanNet: Neural network aided Kalman filtering for partially known dynamics. IEEE Transactions on Signal Processing, 70, 1532-1547.

[2] Dahal, P., Mentasti, S., Paparusso, L., Arrigoni, S., & Braghin, F. (2024). RobustStateNet: Robust ego vehicle state estimation for Autonomous Driving. Robotics and Autonomous Systems, 172, 104585.

[3] Ji, G., Mun, J., Kim, H., & Hwangbo, J. (2022). Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion. IEEE Robotics and Automation Letters, 7(2), 4630-4637.

[4] Buchanan, R., et al. Learning inertial odometry for dynamic legged robot state estimation. In Conference on robot learning (pp. 1575-1584). PMLR.

[5] Yang, A. X., Robeyns, M., Wang, X., & Aitchison, L. Bayesian Low-rank Adaptation for Large Language Models. In The Twelfth International Conference on Learning Representations.