VideoTitans: Scalable Video Prediction with Integrated Short- and Long-term Memory

We thank the reviewer for careful evaluation and helpful suggestions. All of your feedback has been thoroughly reviewed.

[Weakness1] A major limitation is that the paper directly ports the Titans memory module from NLP to video prediction with minimal adaptation. All key components—adaptive long-term memory, sliding window attention, and persistent memory—are used as-is. There is no domain-specific modification to account for the spatio-temporal nature of video data. This raises concerns: (1)How effectively can Titans model spatial locality or hierarchical motion patterns? and (2) Why does this direct transplant work for videos without domain-specific tuning?

[Answer for Weakness1] We would like to respectfully clarify that the adaptation of Titans memory to the video domain involved substantial domain-specific modifications beyond its original NLP formulation. In particular, several changes were essential to successfully tailoring the memory architecture for video prediction:

• Complex patch embedding and reshaping:
  Unlike NLP’s straightforward 1D tokens, video data required converting high-dimensional tensors (frames × channels × height × width) into spatial patches, followed by meticulous reshaping into meaningful 1D embeddings suitable for attention modules. Without these structural modifications, applying Titans directly would lead to prohibitive memory usage and unstable convergence.

• Patch-level gradient-based memory commitment:
  The gradient-based surprise mechanism, originally designed for NLP’s token-level representation, was structurally modified to operate at the video-specific patch level. Without this adaptation, the method would fail to capture the spatial-temporal redundancy and complexity inherent in video data, causing inefficient memory usage and suboptimal predictive performance

These domain-specific structural adaptations were key reasons VideoTitans successfully modeled spatial locality and hierarchical motion patterns, clearly going beyond a direct transplant of Titans memory. We will explicitly emphasize these structural distinctions in our revised manuscript.

[Weakness2] Although the authors highlight linear scaling and computational savings, there is no systematic study of resolution sensitivity.

[Answer for Weakness2] Thank you for this valuable suggestion. To address your concern, we conducted an additional resolution sensitivity analysis, systematically varying input resolutions and measuring computational complexity. Our experiments clearly show that VideoTitans consistently achieves lower FLOPs and better scalability compared to SimVP across various input resolutions:

These results confirm the linear computational scaling and significant efficiency advantage of VideoTitans over SimVP. Importantly, achieving substantially lower computational complexity compared to SimVP—despite SimVP’s inherently simpler CNN-based architecture—highlights the exceptional efficiency and scalability of our proposed memory integration strategy. We will include this systematic comparison and discussion in the revised manuscript.

[Weakness3] Beyond FLOPs, the paper does not report other important efficiency metrics such as model parameters, training time, or inference speed (e.g., FPS). These are necessary for a more complete assessment of efficiency.

[Answer for Weakness3] We appreciate this suggestion. To address your concern, we expanded our efficiency analysis by reporting inference speed (FPS), the number of parameters, training time, as well as FLOPs. As shown clearly in the table below, VideoTitans achieves strong efficiency metrics compared to baseline methods, providing significantly faster inference speed, lower parameters, and fewer FLOPs, while maintaining better predictive accuracy. These results clearly demonstrate the efficiency advantages of our proposed model on the MovingMNIST dataset.

[Weakness4] There are a lot of minor errors or inconsistencies. For example, line-150: Neural Long-term Memory Module or long-term neural memory module? Line-171: five widely-used datasets but only four in Table 2. In Table A (Suppl.), the second best result of LPIPS should be 0.0551 not 0.0577.

[Answer for Weakness4] We thank the reviewer for pointing out the minor inconsistencies.

• Line 150: The correct term is "Neural Long-term Memory Module", and we will ensure consistent usage throughout the paper.
• Line 171: The mention of "five widely-used datasets" was incorrect — the correct number is four, as reflected in Table 2.
• Table A (Supplementary): We confirm that the second-best LPIPS score should be 0.0551, not 0.0577. This will be corrected in the revised version.

We will carefully revise the manuscript and supplementary material to address these issues and ensure consistency.

We sincerely appreciate the reviewer's insightful feedback. We have carefully addressed the points raised.

[Weakness1] The proposed memory integration methods are based on rather standard algorithms, and the paper does not present a substantial methodological advance or novel architecture over prior work.

[Answer for Weakness1] Prior work in video prediction has often leveraged Transformer-based architectures to leverage their expressive power, but the quadratic cost of attention with sequence length makes them inefficient for long videos. CNN- and ConvLSTM-based models, while efficient, suffer from limited receptive fields and fail to capture long-range temporal structure, leading to compounding errors—especially in scenarios with complex motion or long-term dynamics. To overcome these issues, we draw inspiration from the Titans framework, originally developed for language modeling.

Titans was designed to preserve long-range dependencies through dynamic memory access, demonstrating strong performance in tasks like long-form text generation. However, video data poses distinct challenges due to its spatio-temporal nature and high dimensionality. VideoTitans addresses this by replacing discrete tokens with spatio-temporal patches—compact units encoding spatial and temporal structure—enabling richer context and broader receptive fields. To ensure scalability over long sequences, we pair localized attention with a memory mechanism adapted from Titans. While Titans uses a gating mechanism to combine sliding-window attention and long-term memory, we adapt this structure through patch-aware gating and attention mechanisms tailored to video dynamics. Unlike prior patch-based video models that depend on full space-time attention, our architecture maintains scalability and temporal coherence without processing the full sequence. These adaptations yield a video prediction framework that directly addresses the limitations of prior methods in receptive field coverage and long-horizon modeling.

[Weakness2] The range of practical applications and depth of experimental analysis remain limited; results would be more convincing with comparison to the latest, most advanced video prediction models.

[Answer for Weakness2.] We appreciate the reviewer’s suggestion to benchmark against more recent video prediction models. To that end, we conducted experiments on MovingMNIST, and trained VideoTitans for 2,000 epochs to follow the implementation and protocols of the respective baseline papers. Below, we provide results along with a brief description of each compared method.

Additional Comparison Methods:

• VPTR [1]: An early transformer-based model for video prediction using a spatio-temporal transformer with attention.
• MIMO-VP [2]: A video prediction model that adopts a Multi-In-Multi-Out (MIMO) architecture. It is based on a Transformer backbone augmented with convolutional modules such as 3D spatio-temporal blocks and a multi-output decoder.
• PredFormer [3]: A recent state-of-the-art method using a pure transformer-based architecture, without any recurrence or convolution. It leverages global attention and achieves strong results across multiple datasets.

We note that the above results of comparison methods are directly referred from their original papers. These results demonstrate that VideoTitans achieves the lowest error (MSE/MAE) and the highest perceptual quality (SSIM), even with the lowest computational cost (FLOPs) among all compared models. Despite the strong performance of PredFormer, our method yields better accuracy and efficiency. This confirms that VideoTitans is highly competitive not only with classical architectures but also with the latest transformer-based video prediction models, addressing concerns regarding relevance and comparison to state-of-the-art methods.

[1] Ye, Xi, and Guillaume-Alexandre Bilodeau. "Vptr: Efficient transformers for video prediction." 2022 26th International conference on pattern recognition (ICPR). IEEE, 2022.

[2] Ning, Shuliang, et al. "MIMO is all you need: A strong multi-in-multi-out baseline for video prediction." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 2. 2023.

[3] Tang, Yujin, et al. "Video Prediction Transformers without Recurrence or Convolution." arXiv preprint arXiv:2410.04733 (2024).

[Weakness3] The exploration of alternative or more sophisticated memory mechanisms, or deployment with more complex encoders, is mentioned only as future work and not demonstrated.

[Question1] Can the authors clarify whether VideoTitans can be easily extended to incorporate more advanced memory architectures or different encoder types?

[Answer for Weakness3 & Question1] We acknowledge the reviewer’s valuable suggestions regarding the exploration of alternative memory mechanisms and encoder architectures. To address these concerns, we conducted additional experiments on MovingMNIST to demonstrate that VideoTitans is not only modular and extensible in principle but also in practice.

(1) Memory architectures: We compared our NeuralMemory module with a Transformer-based memory module.

The Transformer memory showed comparable performance to NeuralMemory (slight improvement in MSE and MAE, negligible SSIM change), but required significantly higher computational resources (~49% increase in FLOPs). Thus, while VideoTitans supports advanced memory structures, our choice balances accuracy and computational efficiency.

(2) Encoder architectures:

Transformer encoder improved performance but nearly doubled computational cost (~93% increase in FLOPs). ConvNeXt encoder also significantly improved performance across all metrics, with a modest increase in computational cost (~42% higher FLOPs). These results demonstrate VideoTitans’ flexibility, allowing easy integration of diverse encoders to balance performance and computational complexity.

[Weakness4] The analysis of failure cases, or the framework's ability to generalize to more challenging scenarios, is not sufficiently addressed.

[Answer for Weakness4] We appreciate the reviewer’s comment regarding the analysis of failure cases. As with many video prediction models, a common failure case in the video prediction task is the gradual blurring of frames over time due to accumulated prediction errors, especially in the latter part of the sequence. To address this, we provide evaluation results for two examples from the KTH dataset in the supplementary material. Despite this known limitation, VideoTitans consistently outperforms SimVP-gSTA, a strong baseline.

In particular, in the last predicted frame:

• kth_sample1:
  • SimVP-gSTA — PSNR: 27.12, SSIM: 0.634, MSE: 126.10
  • VideoTitans — PSNR: 29.29, SSIM: 0.653, MSE: 76.56
• kth_sample2:
  • SimVP-gSTA — PSNR: 23.59, SSIM: 0.641, MSE: 284.45
  • VideoTitans — PSNR: 28.72, SSIM: 0.712, MSE: 87.23

These results show that even in the most challenging failure cases, such as the last frame, VideoTitans delivers superior reconstruction quality, demonstrating better robustness over long prediction horizons.

[Question2] Are there scenarios where the proposed approach underperforms compared to recent transformer-based methods?

[Answer for Question2] We did not observe any clear scenarios in our experiments where VideoTitans significantly underperformed compared to recent transformer-based methods. However, we observed that both VideoTitans and the ViT-based baseline achieved identical MSE errors of 8.8 at the initial prediction frame (t=1) on the Moving-MNIST benchmark. This implies that for very short-term predictions, transformer-based methods—which utilize global self-attention—can match our performance, as global attention can effectively capture immediate spatial relationships in early frames. Nonetheless, VideoTitans demonstrates clear advantages at longer prediction horizons due to its specialized memory integration and selective storage mechanisms.

[Question3]. How sensitive is the performance to the specific design of the surprise signal and memory modules?

[Answer for Question3] Performance is indeed quite sensitive to the specific design choices of the surprise signal and memory modules. As shown in Figure 5(b) in the main paper, varying the number of memory layers significantly impacts convergence, with deeper memory configurations (e.g., >2 layers) often failing to converge. Additionally, we observed that carefully tuning hyperparameters—such as the gradient clipping threshold, learning rates, and memory dimensions—was essential for achieving stable training and optimal performance. This sensitivity highlights the importance of careful architectural and hyperparameter selection when using Titans memory in video prediction tasks. We will mention it as our limitation and future work.

We thank the reviewer for the helpful comments on long-range prediction and scalability. Based on comments, we address each of your questions with corresponding answers.

[Weakness1] The manuscript repeatedly emphasizes VideoTitans’ strength in modeling long-range dependencies over long sequences, yet all evaluation datasets use very short sequence lengths (T < 10), and the chosen frame-prediction tasks do not adequately demonstrate its long-range prediction capabilities.

[Question1] Could the authors provide results on tasks or datasets with longer sequences to better demonstrate the claimed long-range modeling capabilities?

[Answer for Weakness1 & Question1] We agree and added a Moving‑MNIST‑100 benchmark (10 input + 100 prediction steps, image size 64 × 64), where longer-term predictions are obtained by sequentially applying our 10-step predictor iteratively 10 times. Results averaged over 3 random seeds are shown below (@ frame 10 and @ frame 100 denote the 10th and 100th predicted frames, respectively):

VideoTitans maintains an error rate consistently less than 60% of SimVP‑Swin's error even after 100 steps, confirming stable long-range forecasting. The results of the benchmark are also used to address Question2 below.

[Question2] How does VideoTitans perform when T > 1000 ? Any insights on memory usage in such settings?

[Answer for Question2] Running video prediction models with sequence lengths T > 1000 poses significant challenges due to rapidly increasing computational complexity. For example, processing 1,000-frame Moving-MNIST videos with VideoTitans (shape 1000×1×64×64) quickly exceeds manageable FLOPs limits even before accounting for gradients. The computational cost in a standard non-autoregressive setup rises from 173.553 GFLOPs at T = 100 to 1,490.638 GFLOPs at T = 1000.

The effective solution is to perform autoregressive rollouts using shorter predictions (e.g., 10-step segments). By doing so, long-range video predictions beyond conventional benchmarks become feasible, with each 10-frame rollout requiring only about 13 GFLOPs. VideoTitans’ neural long-term memory module retains only a small set of memory tokens between steps, preventing linear growth in computational cost and enabling stable, coherent predictions over extended horizons. Although autoregressive rollouts can suffer from error accumulation over longer prediction horizons, our results on the Moving‑MNIST‑100 benchmark (autoregressive) show improved accuracy and scalability compared to baseline methods, highlighting the architecture’s potential for long-term video forecasting.

[Discussion2]

The manuscript repeatedly emphasizes VideoTitans’ strength in modeling long-range dependencies and scalability, claims that are also prominently featured in the title. However, the authors did not provide any experiments demonstrating scalability in terms of data complexity, sequence length, or model size. The current experiments fail to support these claims.

[Answer for Discussion2]

We appreciate the reviewer’s concern and agree that clear scalability evidence is essential.

Our experiments cover datasets significantly varying in complexity and time scales—from Moving-MNIST (simple grayscale digits at ~10fps) and TrafficBJ (urban flow data at 30-minute intervals) to Human3.6M (RGB articulated motion at 25fps) and WeatherBench (global meteorology at hourly intervals). Across all benchmarks, VideoTitans consistently achieves strong accuracy while simultaneously providing the lowest computational cost compared to existing methods.

Specifically, our model demonstrates:

• High accuracy and low computational cost across all evaluated datasets.
• Strong performance in extended sequence-length predictions (up to 100 frames), significantly outperforming existing methods.

Moreover, we quantify scalability along three axes: data complexity, sequence length, and model size.

1. For data complexity, we evaluate on a structured ladder of four datasets—Moving-MNIST, TrafficBJ, Human3.6M, and WeatherBench—which progressively increase in spatial resolution, temporal dynamics, modality, and real-world variability. Across all of them, VideoTitans delivers state-of-the-art accuracy while reducing FLOPs by 8–93% compared to strong published baselines.

2. For sequence length, in response to a reviewer’s question about long-horizon behavior, we supplemented the standard 10→10 protocol with a 10→100 test on Moving-MNIST. VideoTitans sustained strong prediction quality throughout the extended rollout, demonstrating robustness in long-range temporal modeling.

3. For model size, following Reviewer akfP’s suggestion, we performed a controlled comparison under uniform training conditions. VideoTitans achieves the lowest MSE, fastest inference speed (367 FPS), few parameters (6.0M), and lowest computational cost (13.33 GFLOPs) among high-performing models—proving it scales down efficiently without sacrificing accuracy.

We will explicitly highlight these findings, including the time scales, in the revised manuscript to avoid misunderstandings.